Implementation tracking for 7.0 types deprecation. #35190

jtibshirani · 2018-11-02T01:13:39Z

Tracks the details of the 'In 7.0' part of this comment: #15613 (comment)

Plan for 7.0

For requests with a type in the URL or as a leaf field, we will accept both typed + typeless versions of the API. We’ll emit a deprecation warning to tell users they need to move to the typeless endpoints before 8.0. Responses will still contain a _type field, but we will return the dummy name _doc regardless of the underlying type name.
For APIs whose request/ response structure changes with the deprecation (create index, get mapping, etc.), we’ll have a request parameter include_type_name that should be set to false to omit types in requests + responses. It will default to true in 6.7 with a warning that it needs to be explicitly specified (to either true or false), default to false in 7.0 with a warning to stop specifying it, and finally be removed in 8.0.

More information can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

To-do List

Phase One: Add Typeless APIs. These items are critical for 6.7/ 7.0, and should be in before feature freeze.

Phase Two: Important Clean-up. These tasks should be in by 6.7/ 7.0, but can go in after feature freeze.

Update removal_of_types.asciidoc to reflect the new plan. @jtibshirani Update 'removal of types' docs to reflect the new plan for 7.0. #38003
Switch to typeless index creation in REST tests. @colings86 Removes typed calls from YAML REST tests #37611 Default include_type_name to false in the yml test harness. #38058
Switch to typeless index creation (remove include_type_name=true) in documentation. @cbuescher Remove include_type_name in asciidoc where possible #37568 Remove more include_type_name and types from docs #37601 Remove remaining occurances of "include_type_name=true" in docs #37646
Solidify the approach to allowing typeless requests on an index with a custom type (Solidify the approach to allowing typeless requests on an index with a custom type. #37450).

Phase Three: Additional Deprecations. These are good to have by 6.7/ 7.0, but could be pushed into 7.1 if strictly necessary.

Deprecate references to _type in the search request body.
- uses of _type as a field name (in match queries, etc.) @mayya-sharipova Deprecate uses of _type as a field name in queries #36503 Avoid duplicate types deprecation messages in search-related APIs. #36802
- aggregations @jtibshirani Deprecate use of the _type field in aggregations. #37131
- ~~retrieving fields~~ Loading _type as a field doesn't actually work, and doesn't make much sense because each search hit already contains a _type component.
~~Emit a deprecation warning when a search template containing a type is triggered. Note this is different from emitting a warning when a typed template is originally added.~~ Already covered by other search-related PRs.
~~Emit a deprecation warning when an index template containing a type is triggered. Note this is different from emitting a warning when a typed template is originally added.~~ We support making typeless calls against an index with a custom type, so it is not really harmful to have a template with a custom type (and in fact several internal templates around monitoring use typed templates).
Deprecate references to _type in scripts. @jdconrad Deprecate _type from LeafDocLookup #37491 Add types deprecation to script contexts #37554
Deprecate types in simulate pipeline requests (Deprecate types in simulate pipeline requests. #37731). @gwbrown Deprecate _type in simulate pipeline requests #37949
Deprecate types in graph explore requests. (Deprecate types in _graph/explore calls. #40466)

Items we’re still following-up on

For responses that contain the type as a leaf field, should we always return _doc regardless of the underlying type even when the old typed APIs are used?
Types may be present in saved search requests, including search templates and watches. We should think through the upgrade plan here. Implementation tracking for 7.0 types deprecation. #35190 (comment)
When an index template is stored, the mappings are nested under the type name. We also need to consider how these will be accessed and upgraded.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2018-11-02T01:13:40Z

Pinging @elastic/es-search-aggs

jtibshirani · 2018-11-06T19:37:37Z

CC @elastic/es-clients

Mpdreamz · 2018-11-12T11:07:20Z

One last thing I want to raise which might be super contentious:

Right now the default for include_type_name when not specified is true in 7.0. There are two downsides with this:

New installations will get deprecation warnings OOTB unless they specify include_type_name=false which they need to remove in 8.0.
Someone who wants to go through all the type related changes in one major version can not do so.

Can we remove some of the confusion of include_type_name defaulting to true by simply having it
default to false in 7.0 already?

New installations will return the desired responses by default.
Someone who wants to go through all the changes in one major version can do so.

Upgrades will have to account for the type removal whether the include_type_name defaults to true or false.

Are we actually doing our users a disservice by holding on to a deprecation period?

markharwood · 2018-11-15T17:25:49Z

Types may be present in saved search requests, including search templates and watches. We should think through the upgrade plan here.

The proposal is in 7.0 we will introduce deprecation warnings when saved queries with types are executed, but not do anything to proactively detect and upgrade them automatically. Some of the easier type removals would be in the structured metadata held around a saved search whereas other changes may be buried in clauses embedded in the saved search's choice of query DSL.

jpountz · 2018-11-15T17:38:35Z

@Mpdreamz I'm thinking that this could be an issue for mixed version clusters that have 6.latest and 7.0 nodes as the response format would depend on the version of the node that you are querying?

bleskes · 2018-11-19T15:46:26Z

New installations will get deprecation warnings OOTB unless
@Mpdreamz I'm thinking that this could be an issue for mixed version clusters that have 6.latest and 7.0 nodes as the response format would depend on the version of the node that you are querying?

This is typically dealt with in the following fashion:

Add a flag to enable new functionality in the old version, while defaulting to disable. Requests without that flag will emit deprecation warnings.
New version only supports the flag in "enabled mode" and will omit deprecation logs when flag is used.

The suggestion above follows this pattern, which is good. The only difference with what we do normally is the duration of deprecation. Normally it will be scoped down to something like OLD_MAJOR.latest. With the above suggestion it's NEW_MAJOR.x , which is a long time. I wonder if we should look at the costs of backporting the changes made to 6.x so we can follow our standard path.

jpountz · 2018-11-20T10:24:28Z

In the beginning this flag was supposed to exist on most APIs including some of the most used like the document APIs. This made me view the long deprecation period as a feature. Now that it's only about the mappings APIs, I could change my mind. One subtlety with backporting to 6.x is that it might still have 5.x indices that have multiple types, so include_type_name=false would not make sense on such indices and we'd need to respond appropriately (error?).

…rom `_doc`. This commit makes `document`, `update`, `explain`, `termvectors` and `mapping` typeless APIs work on indices that have a type whose name is not `_doc`. Unfortunately, this needs to be a bit of a hack since I didn't want calls with random type names to see documents with the type name that the user had chosen upon type creation. The `explain` and `termvectors` do not support being called without a type for now so the test is just using `_doc` as a type for now, we will need to fix tests later but this shouldn't require further changes server-side since passing `_doc` as a type name is what typeless APIs do internally anyway. Relates elastic#35190

markharwood · 2018-11-21T18:11:29Z

I looked into auto-detecting PUTs of typeless mappings/templates to see if we can avoid the need for passing an include_type_name flag in the URL. It looks like this will be possible as we can check if the root object has a value called "properties". Originally I thought that real estate agents with a doc type called "properties" might be an edge case which would cause ambiguity. Fortunately 6.5 has a parsing bug that prevents you from creating doc types called "properties" (try it and see).

markharwood · 2018-11-22T18:01:54Z

Fortunately 6.5 has a parsing bug that prevents you from creating doc types called "properties"

Turns out there's more to auto-detecting no-types than checking for the presence of a top-level properties field. Legal mappings can have no properties but include other top-level attributes e.g. _source:{enabled:false} or dynamic_templates:{...}.

While it would have been nice to auto-detect typeless mappings and provide the 7.0 examples in reference docs without any types this would not be consistent with the results of GET _mapping calls which would return types by default (unless the include_type_name=false param is passed).

Does auto-detection still seem useful or shall we insist on documenting APIs with include_type_name=false params?

…rom `_doc` (#35790) This commit makes `document`, `update`, `explain`, `termvectors` and `mapping` typeless APIs work on indices that have a type whose name is not `_doc`. Unfortunately, this needs to be a bit of a hack since I didn't want calls with random type names to see documents with the type name that the user had chosen upon type creation. The `explain` and `termvectors` do not support being called without a type for now so the test is just using `_doc` as a type for now, we will need to fix tests later but this shouldn't require further changes server-side since passing `_doc` as a type name is what typeless APIs do internally anyway. Relates #35190

Relates to elastic#35190

Relates to #35190

…stic#37484) Added deprecation warnings for use of include_type_name in put/get index templates. HLRC changes: GetIndexTemplateRequest has a new client-side class which is a copy of server's GetIndexTemplateResponse but modified to be typeless. PutIndexTemplateRequest has a new client-side counterpart which doesn't use types in the mappings Relates to elastic#35190

Added deprecation warnings for use of include_type_name in put/get index templates. HLRC changes: * GetIndexTemplateRequest has a new client-side class which is a copy of server's GetIndexTemplateResponse but modified to be typeless. * PutIndexTemplateRequest has a new client-side counterpart which doesn't use types in the mappings Relates to #35190

Backport for elastic#38039 Relates to elastic#35190

Backport for #38039 Relates to #35190

Backport for elastic#38039 Relates to elastic#35190

) The HLRC client currently uses `org.elasticsearch.action.admin.indices.get.GetIndexRequest` and `org.elasticsearch.action.admin.indices.get.GetIndexResponse` in its get index calls. Both request and response are designed for the typed APIs, including some return types e.g. for `getMappings()` which in the maps it returns still use a level including the type name. In order to change this without breaking existing users of the HLRC API, this PR introduces two new request and response objects in the `org.elasticsearch.client.indices` client package. These are used by the IndicesClient#get and IndicesClient#exists calls now by default and support the type-less API. The old request and response objects are still kept for use in similarly named, but deprecated methods. The newly introduced client side classes are simplified versions of the server side request/response classes since they don't need to support wire serialization, and only the response needs fromXContent parsing (but no xContent-serialization, since this is the responsibility of the server-side class). Also changing the return type of `GetIndexResponse#getMapping` to `Map<String, MappingMetaData> getMappings()`, while it previously was returning another map keyed by the type-name. Similar getters return simple Maps instead of the ImmutableOpenMaps that the server side response objects return. Backport for #37778 Relates to #35190

Backport for #38039 Relates to #35190

…stic#37484) Added deprecation warnings for use of include_type_name in put/get index templates. HLRC changes: GetIndexTemplateRequest has a new client-side class which is a copy of server's GetIndexTemplateResponse but modified to be typeless. PutIndexTemplateRequest has a new client-side counterpart which doesn't use types in the mappings Relates to elastic#35190

Added deprecation warnings for use of include_type_name in put/get index templates. HLRC changes: * GetIndexTemplateRequest has a new client-side class which is a copy of server's GetIndexTemplateResponse but modified to be typeless. * PutIndexTemplateRequest has a new client-side counterpart which doesn't use types in the mappings Relates to #35190

jakelandis · 2019-02-08T16:54:25Z

When an index template is stored, the mappings are nested under the type name. We also need to consider how these will be accessed and upgraded.

I logged #38637 to further address this point.

…8825) We expect many users to have a custom document type in 6.x, for example they might have used `doc` when creating indices. In this case, users cannot easily switch over to typeless index creations in 6.7 using `include_type_name=false`, because the rest of their document CRUD calls will refer to the custom type. Instead, we are recommending that users take the following steps: set `include_type_name=true` in 6.7 for all relevant calls, upgrade to 7.0, then switch over completely to the typeless APIs and stop using `include_type_name`. This means that it will be very common to set `include_type_name=true` in 6.7, so it is misleading to emit a deprecation warning to tell the user to switch to using `include_type_name=false`. This PR switches to emitting a deprecation warning only if `include_type_name` is not set at all. The warning serves as an important note to users that the request and response format of these APIs will change in a breaking way in 7.0. Relates to #35190.

jpountz · 2019-04-02T09:27:45Z

All items have been completed.

jtibshirani added :Search Foundations/Mapping Index mappings, including merging and defining field types Meta >deprecation labels Nov 2, 2018

jtibshirani mentioned this issue Nov 6, 2018

Remove include_type_name from the relevant APIs. #35192

Merged

jtibshirani self-assigned this Nov 8, 2018

This was referenced Nov 13, 2018

Remove support for types? #15613

Closed

Deprecate types in count and msearch. #35421

Merged

jtibshirani assigned jpountz and markharwood and unassigned jpountz Nov 14, 2018

jpountz mentioned this issue Nov 21, 2018

Make typeless APIs usable with indices whose type name is different from _doc #35790

Merged

gwbrown mentioned this issue Nov 29, 2018

Update Deprecation Info API for 7.0 breaking changes #36024

Closed

26 tasks

cbuescher self-assigned this Dec 6, 2018

mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue Dec 7, 2018

Deprecate types in update_by_query and delete_by_query

c323868

Relates to elastic#35190

mayya-sharipova mentioned this issue Dec 7, 2018

Deprecate types in update_by_query and delete_by_query #36365

Merged

mayya-sharipova mentioned this issue Dec 11, 2018

Deprecate uses of _type as a field name in queries #36503

Merged

mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue Dec 11, 2018

Deprecate uses of _type as a field name

0947bad

Relates to elastic#35190

mayya-sharipova added a commit that referenced this issue Dec 11, 2018

Deprecate types in update_by_query and delete_by_query (#36365)

2f18325

Relates to #35190

mayya-sharipova added a commit that referenced this issue Feb 4, 2019

Deprecate types in rollover index API (#38039)

6417044

Relates to #35190

mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue Feb 5, 2019

Deprecate types in rollover index API

d1f2cb1

Backport for elastic#38039 Relates to elastic#35190

mayya-sharipova mentioned this issue Feb 5, 2019

Deprecate types in rollover index API #38389

Merged

mayya-sharipova added a commit that referenced this issue Feb 5, 2019

Deprecate types in rollover index API (#38389)

2e7b7a8

Backport for #38039 Relates to #35190

This was referenced Feb 5, 2019

Backport: Add typless client side GetIndexRequest calls and response class #38420

Closed

Backport: Add typless client side GetIndexRequest calls and response class #38422

Merged

mayya-sharipova mentioned this issue Feb 5, 2019

Deprecate types in rollover index API (#38389) #38458

Merged

mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue Feb 5, 2019

Deprecate types in rollover index API (elastic#38389)

34e81e3

Backport for elastic#38039 Relates to elastic#35190

mayya-sharipova added a commit that referenced this issue Feb 5, 2019

Deprecate types in rollover index API (#38389) (#38458)

aa9816c

Backport for #38039 Relates to #35190

This was referenced Feb 6, 2019

Make the 'get templates' types deprecation message consistent. #38532

Merged

Make the 'get templates' types deprecation message consistent. #38533

Merged

Make the 'get templates' types deprecation message consistent. #38549

Merged

jtibshirani mentioned this issue Feb 12, 2019

Only issue a deprecation warning if include_type_name is not set. #38825

Merged

jtibshirani assigned jpountz and unassigned markharwood, mayya-sharipova and cbuescher Mar 4, 2019

jpountz closed this as completed Apr 2, 2019

nknize mentioned this issue Apr 21, 2022

[BUG] Breaking changes with opensearch.http.override_main_response_version=true opensearch-project/OpenSearch#3023

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation tracking for 7.0 types deprecation. #35190

Implementation tracking for 7.0 types deprecation. #35190

jtibshirani commented Nov 2, 2018 •

edited by jpountz

Loading

elasticmachine commented Nov 2, 2018

jtibshirani commented Nov 6, 2018

Mpdreamz commented Nov 12, 2018 •

edited

Loading

markharwood commented Nov 15, 2018

jpountz commented Nov 15, 2018

bleskes commented Nov 19, 2018

jpountz commented Nov 20, 2018

markharwood commented Nov 21, 2018

markharwood commented Nov 22, 2018

jakelandis commented Feb 8, 2019

jpountz commented Apr 2, 2019

Implementation tracking for 7.0 types deprecation. #35190

Implementation tracking for 7.0 types deprecation. #35190

Comments

jtibshirani commented Nov 2, 2018 • edited by jpountz Loading

Plan for 7.0

To-do List

Items we’re still following-up on

elasticmachine commented Nov 2, 2018

jtibshirani commented Nov 6, 2018

Mpdreamz commented Nov 12, 2018 • edited Loading

markharwood commented Nov 15, 2018

jpountz commented Nov 15, 2018

bleskes commented Nov 19, 2018

jpountz commented Nov 20, 2018

markharwood commented Nov 21, 2018

markharwood commented Nov 22, 2018

jakelandis commented Feb 8, 2019

jpountz commented Apr 2, 2019

jtibshirani commented Nov 2, 2018 •

edited by jpountz

Loading

Mpdreamz commented Nov 12, 2018 •

edited

Loading