Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. #37871

Merged

Conversation

Projects
None yet
5 participants
@jpountz
Copy link
Contributor

commented Jan 25, 2019

Currently if you mix typed templates and typeless index creation or typeless
templates and typed index creation then you will end up with an error because
Elasticsearch tries to create an index that has multiple types: _doc and
the explicit type name that you used.

This commit proposes to give precedence to the index creation call so that
the type from the template will be ignored if the index creation call is
typeless while the template is typed, and the type from the index creation
call will be used if there is a typeless template.

This is consistent with the fact that index creation already "wins" if a field
is defined differently in the index creation call and in a template: the
definition from the index creation call is used in such cases.

Closes #37773

Give precedence to index creation when mixing typed templates with ty…
…peless index creation and vice-versa.

Currently if you mix typed templates and typeless index creation or typeless
templates and typed index creation then you will end up with an error because
Elasticsearch tries to create an index that has multiple types: `_doc` and
the explicit type name that you used.

This commit proposes to give precedence to the index creation call so that
the type from the template will be ignored if the index creation call is
typeless while the template is typed, and the type from the index creation
call will be used if there is a typeless template.

This is consistent with the fact that index creation already "wins" if a field
is defined differently in the index creation call and in a template: the
definition from the index creation call is used in such cases.

Closes #37773
@elasticmachine

This comment has been minimized.

Copy link
Collaborator

commented Jan 25, 2019

@jpountz jpountz changed the title Give precedence to index creation when mixing typed templates with ty… Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. Jan 25, 2019

@jpountz jpountz requested a review from jtibshirani Jan 25, 2019

@jpountz

This comment has been minimized.

Copy link
Contributor Author

commented Jan 28, 2019

@elasticmachine run elasticsearch-ci/2, run elasticsearch-ci/oss-distro-docs

@jtibshirani
Copy link
Member

left a comment

Thanks a lot @jpountz for this PR!

I originally had a different intuition for the behavior: I expected that given a template with a custom type, and a typeless create index request, the final index would contain the custom type. This is because I think of create index requests with no mappings as being 'typeless', and these requests won't influence the type of the final index. The overall principle is something like 'a custom type always wins out over having no type (including the dummy _doc type)', because it indicates a desire to break with the default behavior.

However, I can see the benefits of this approach -- it fits with the overall idea of the index request 'winning out', and users have more flexibility to convert each index creation to a typeless one. In any case, since the right behavior is not obvious, I think it would be worth updating the templates documentation with an explanation.

@@ -0,0 +1,77 @@
---
"Create a typeless index while there is a typed template":

This comment has been minimized.

Copy link
@jtibshirani

jtibshirani Jan 29, 2019

Member

Since the change is a bit subtle, I wonder if it'd be worth adding some unit tests in IndexCreationTaskTests? In particular, reading the code I had to think a bit about what would happen if no mappings were provided in the create index requests, or if the template or create request had more than one type.


In case of implicit index creation, because of documents that get indexed in
an index that doesn't exist yet, the template is always honored. This is
usually not a problem due to the fact that typless index calls work on typed

This comment has been minimized.

Copy link
@jtibshirani

jtibshirani Jan 29, 2019

Member

Small typo (typless) but it's so hard to get a green build, I can just fix this when we do another pass over this documentation.

This comment has been minimized.

Copy link
@jpountz

jpountz Jan 30, 2019

Author Contributor

I will open a follow-up PR.

@jpountz jpountz merged commit b63b50b into elastic:master Jan 30, 2019

8 checks passed

CLA Commit author is a member of Elasticsearch
Details
elasticsearch-ci/1 Build finished.
Details
elasticsearch-ci/2 Build finished.
Details
elasticsearch-ci/default-distro Build finished.
Details
elasticsearch-ci/docbldesx Build finished.
Details
elasticsearch-ci/docs-check Build finished.
Details
elasticsearch-ci/oss-distro-docs Build finished.
Details
elasticsearch-ci/packaging-sample Build finished.
Details

@jpountz jpountz deleted the jpountz:mix_typed_typeless_template_index_creation branch Jan 30, 2019

jpountz added a commit to jpountz/elasticsearch that referenced this pull request Jan 30, 2019

Fix typo in docs.
This has been introduced in elastic#37871.

@jpountz jpountz referenced this pull request Jan 30, 2019

Merged

Fix typo in docs. #38018

jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Jan 30, 2019

Merge branch 'master' into retention-lease-stats
* master: (29 commits)
  Fix limit on retaining sequence number (elastic#37992)
  Docs test fix, wait for shards active.
  Revert "Revert "Documented default values for index follow request parameters. (elastic#37917)""
  Revert "Documented default values for index follow request parameters. (elastic#37917)"
  Ensure date parsing BWC compatibility (elastic#37929)
  SQL: Skip the nested and object field types in case of an ODBC request (elastic#37948)
  Use mappings to format doc-value fields by default. (elastic#30831)
  Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. (elastic#37871)
  Add classifier to tar.gz in docker compose (elastic#38011)
  Documented default values for index follow request parameters. (elastic#37917)
  Fix fetch source option in expand search phase (elastic#37908)
  Restore a noop _all metadata field for 6x indices (elastic#37808)
  Added ccr to xpack usage infrastructure (elastic#37256)
  Fix exit code for Security CLI tools  (elastic#37956)
  Streamline S3 Repository- and Client-Settings (elastic#37393)
  Add version 6.6.1 (elastic#37975)
  Ensure task metadata not null in follow test (elastic#37993)
  Docs fix - missing callout
  Types removal - deprecate include_type_name with index templates (elastic#37484)
  Handle completion suggestion without contexts
  ...

henningandersen added a commit to henningandersen/elasticsearch that referenced this pull request Jan 30, 2019

Give precedence to index creation when mixing typed templates with ty…
…peless index creation and vice-versa. (elastic#37871)

Currently if you mix typed templates and typeless index creation or typeless
templates and typed index creation then you will end up with an error because
Elasticsearch tries to create an index that has multiple types: `_doc` and
the explicit type name that you used.

This commit proposes to give precedence to the index creation call so that
the type from the template will be ignored if the index creation call is
typeless while the template is typed, and the type from the index creation
call will be used if there is a typeless template.

This is consistent with the fact that index creation already "wins" if a field
is defined differently in the index creation call and in a template: the
definition from the index creation call is used in such cases.

Closes elastic#37773

jpountz added a commit to jpountz/elasticsearch that referenced this pull request Jan 30, 2019

Give precedence to index creation when mixing typed templates with ty…
…peless index creation and vice-versa.

This is a backport of elastic#37871 and elastic#38032.

jpountz added a commit that referenced this pull request Jan 31, 2019

Fix typo in docs. (#38018)
This has been introduced in #37871.

jpountz added a commit that referenced this pull request Feb 1, 2019

Give precedence to index creation when mixing typed templates with ty…
…peless index creation and vice-versa. (#38055)

This is a backport of #37871 and #38032.

jpountz added a commit to jpountz/elasticsearch that referenced this pull request Feb 1, 2019

Enable bw tests for elastic#37871 and elastic#38032.
Mixed-version clusters tests had been disabled initially since they wouldn't
work until the functionality would be backported.

jpountz added a commit that referenced this pull request Feb 1, 2019

Enable bw tests for #37871 and #38032. (#38167)
Mixed-version clusters tests had been disabled initially since they wouldn't
work until the functionality would be backported.

jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Feb 1, 2019

Merge branch 'master' into retention-leases-version
* master: (36 commits)
  Ensure joda compatibility in custom date formats (elastic#38171)
  Do not compute cardinality if the `terms` execution mode does not use `global_ordinals` (elastic#38169)
  Do not set timeout for IndexRequests in GatewayIndexStateIT (elastic#38147)
  Zen2ify testMasterFailoverDuringIndexingWithMappingChanges (elastic#38178)
  SQL: [Docs] Add limitation for aggregate functions on scalars (elastic#38186)
  Add elasticsearch-node detach-cluster command (elastic#37979)
  Add tests for fractional epoch parsing (elastic#38162)
  Enable bw tests for elastic#37871 and elastic#38032. (elastic#38167)
  Clear send behavior rule in CloseWhileRelocatingShardsIT (elastic#38159)
  Fix testCorruptedIndex (elastic#38161)
  Add finalReduce flag to SearchRequest (elastic#38104)
  Forbid negative field boosts in analyzed queries (elastic#37930)
  Remove AtomiFieldData#getLegacyFieldValues (elastic#38087)
  Universal cluster bootstrap method for tests with autoMinMasterNodes=false (elastic#38038)
  Fix FullClusterRestartIT.testHistoryUUIDIsAdded (elastic#38098)
  Replace joda time in ingest-common module (elastic#38088)
  Fix eclipse config for ssl-config (elastic#38096)
  Don't load global ordinals with the `map` execution_hint (elastic#37833)
  Relax fault detector in some disruption tests (elastic#38101)
  Fix java time epoch date formatters (elastic#37829)
  ...

matriv added a commit to matriv/elasticsearch that referenced this pull request Feb 2, 2019

Enable bw tests for elastic#37871 and elastic#38032. (elastic#38167)
Mixed-version clusters tests had been disabled initially since they wouldn't
work until the functionality would be backported.

@colings86 colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.