Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Elasticsearch creates index templates with deprecated types #37773

Closed
15 tasks
chrisdavies opened this issue Jan 23, 2019 · 10 comments
Closed
15 tasks

Elasticsearch creates index templates with deprecated types #37773

chrisdavies opened this issue Jan 23, 2019 · 10 comments
Labels
Meta :Search Foundations/Mapping Index mappings, including merging and defining field types :Search/Search Search-related issues that do not fall into other categories

Comments

@chrisdavies
Copy link

chrisdavies commented Jan 23, 2019 •
edited by gwbrown
Loading

We're trying to get Kibana to work with ES7, but are getting errors like this:

Cannot put multiple mappings: [_doc, doc]

The reason is that we're modifying Kibana to no longer specify the root type for a document, but there are existing index templates which are interfering with us.

Here's a list of template names and their root doc types:

  • .ml-meta [ 'doc' ]
  • .ml-notifications [ 'audit_message' ]
  • .ml-anomalies- [ 'doc' ]
  • .ml-config [ 'doc' ]
  • .ml-state [ 'doc' ]
  • .logstash-management [ 'doc' ]
  • .watches [ 'doc' ]
  • .triggered_watches [ 'doc' ]
  • .watch-history-9 [ 'doc' ]
  • .monitoring-alerts [ 'doc' ]
  • .monitoring-es [ 'doc' ]
  • .monitoring-kibana [ 'doc' ]
  • .monitoring-logstash [ 'doc' ]
  • .monitoring-beats [ 'doc' ]
  • security_audit_log [ 'doc' ]

I saw that there are some individual issues to track this (#37442) but I thought it might be good to have a more comprehensive issue.

Feel free to close if this is redundant!
@chrisdavies chrisdavies mentioned this issue Jan 23, 2019
Closed
@romseygeek romseygeek added the :Search/Search Search-related issues that do not fall into other categories label Jan 23, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search

@chrisronline chrisronline mentioned this issue Jan 23, 2019
Merged
@markharwood markharwood added the :Search Foundations/Mapping Index mappings, including merging and defining field types label Jan 23, 2019
@gwbrown gwbrown added the Meta label Jan 23, 2019
@gwbrown
Copy link
Contributor

gwbrown commented Jan 23, 2019 •
edited
Loading

Per @chrisdavies on Slack, clarifying the issue:

What happens is that Kibana's test suite loads in archives which create indices + mappings and import a bunch of docs, then the tests run against that data.

We've upgraded our archives to match ES7 expectations (e.g. no root type specified). But this causes a conflict with the existing index templates most of which have a root type of 'doc'.

There may be another way around this for the Kibana test suite, but we should still clear all of these up for the reasons detailed in #37442.

/cc @jtibshirani for awareness

@jtibshirani
Copy link
Contributor

Thank you for flagging this @chrisdavies, and sorry that you ran into these issues!

We had actually planned to keep the type name doc for these internal templates for the 7.0 release. There are a few points behind this reasoning:

  • In almost all cases we don’t expect deprecation warnings from interacting with these templates, because warnings are only issued at the REST layer and these internal products use the transport client.
  • Typeless APIs like 'index' and 'bulk' work against indices with a custom type name, so there is not a strong reason to migrate these index templates right now to be typeless.
  • Moving off the type name doc would be a breaking change for those internal products, and also potentially Kibana, as Kibana is still in the process of moving off of the typed APIs.

We will certainly plan to move to typeless templates after the 7.0 release, when we begin to update more server-side code to be typeless, and work towards the goal of removing types completely in elasticsearch.

While I don’t think there’s immediate action in terms of upgrading templates, the error you ran into highlights a bug in our current approach: it’s not possible to issue a typeless 'create index' request when it triggers an index template that contains a custom type. I will look into fixing this bug, as it’s something I think we should be able to support.

Lastly, @jpountz or @bleskes would you be able to confirm if the reasoning above around templates above makes sense to you as well?

@jpountz
Copy link
Contributor

jpountz commented Jan 25, 2019

@jtibshirani Agreed.

@bleskes
Copy link
Contributor

bleskes commented Jan 25, 2019

@jtibshirani I'm ok with the approach. I'm not 100% sure I understand why we can't update all the templates (it's just json, right?) to remove the types in 7.0.0 but I trust your judgement.

@chrisdavies
Copy link
Author

Yeah. This actually makes Kibana's process trickier, I think, because this will mean we have "doc" in some places and "_doc" / default in some places.

That said, what I'm doing right now is just removing the mappings for these indices from our test archives, and that seems to be working out so far. It may be that we don't have any production code in Kibana that explicitly specifies the root type for these indices. I'll know more in a few days once the Eleasticsearch 7 update is complete on our end.

jpountz added a commit to jpountz/elasticsearch that referenced this issue Jan 25, 2019
@jpountz
Give precedence to index creation when mixing typed templates with ty…
69f0b1e 
…peless index creation and vice-versa.

Currently if you mix typed templates and typeless index creation or typeless
templates and typed index creation then you will end up with an error because
Elasticsearch tries to create an index that has multiple types: `_doc` and
the explicit type name that you used.

This commit proposes to give precedence to the index creation call so that
the type from the template will be ignored if the index creation call is
typeless while the template is typed, and the type from the index creation
call will be used if there is a typeless template.

This is consistent with the fact that index creation already "wins" if a field
is defined differently in the index creation call and in a template: the
definition from the index creation call is used in such cases.

Closes elastic#37773
@jpountz jpountz mentioned this issue Jan 25, 2019
Merged
@jtibshirani
Copy link
Contributor

jtibshirani commented Jan 25, 2019 •
edited
Loading

@bleskes The main issue is all the indices generated from the templates would switch from having type doc to being typeless. We would then need to update all the code in internal products to use the type name _doc/ the typeless APIs for 7.0, and also reindex any existing internal indices that were created using the old type doc.

@chrisdavies After we fix the bug mentioned above, I wouldn't think that there'd be a situation where you still need to refer to the old doc type name. If you run into a situation like this or continue to have any issues with the templates, please let us know.

@jpountz
Copy link
Contributor

jpountz commented Jan 25, 2019

@chrisdavies Can you tell us what API call you did exactly to run into this error?

@chrisdavies
Copy link
Author

It was es_archiver in Kibana, which basically is doing a create index with mappings.

@jpountz
Copy link
Contributor

jpountz commented Jan 28, 2019

Ok, so it is creating an index explicitly that matches a template, rather than letting the index be created automatically, this is what I needed to know. Thank you.

jpountz added a commit that referenced this issue Jan 30, 2019
@jpountz
Give precedence to index creation when mixing typed templates with ty…
b63b50b 
…peless index creation and vice-versa. (#37871)

Currently if you mix typed templates and typeless index creation or typeless
templates and typed index creation then you will end up with an error because
Elasticsearch tries to create an index that has multiple types: `_doc` and
the explicit type name that you used.

This commit proposes to give precedence to the index creation call so that
the type from the template will be ignored if the index creation call is
typeless while the template is typed, and the type from the index creation
call will be used if there is a typeless template.

This is consistent with the fact that index creation already "wins" if a field
is defined differently in the index creation call and in a template: the
definition from the index creation call is used in such cases.

Closes #37773
@chrisronline chrisronline mentioned this issue Feb 5, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Meta :Search Foundations/Mapping Index mappings, including merging and defining field types :Search/Search Search-related issues that do not fall into other categories
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@markharwood @jpountz @chrisdavies @bleskes @romseygeek @gwbrown @jtibshirani @elasticmachine