Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping potentially lost with `"dynamic" : "strict"`, `_default_` mapping and failed document index #8650

Closed
brwe opened this issue Nov 25, 2014 · 5 comments · Fixed by #10634

Comments

@brwe
Copy link
Contributor

commented Nov 25, 2014

The effect is very similar to #5623
When a document is indexed that does have a dynamic field then the indexing fails as expected.
However, the type is created locally in the mapper service of the node but never updated on master, see brwe@340f5c5#diff-defbaaff93b959a2f9a93e7167f6f345R165

This can cause several problems:

  1. _default_ mappings are applied locally and can potentially later not be updated anymore, see brwe@340f5c5#diff-ed65252ffbbf8656bf257a8cd6251420R68 (thanks @pkoenig10 for the test, #8423 (comment))
  2. Mappings that were created via _default_ mappings when indexing a document can be lost, see brwe@340f5c5#diff-defbaaff93b959a2f9a93e7167f6f345R187
@miccon

This comment has been minimized.

Copy link

commented Nov 27, 2014

Will this also be fixed on the 1.4 branch?

@brwe

This comment has been minimized.

Copy link
Contributor Author

commented Nov 27, 2014

Yes.

To fix this, there is two options:

  1. make sure the type is not created if indexing fails
  2. update the mapping on master even if indexing of doc failed

Option 1 is rather tricky to implement and I do not see why the type should not be created in the mapping so I'll make a pr for option 2 shortly.

@miccon

This comment has been minimized.

Copy link

commented Nov 28, 2014

I just checked 0992e7f, but https://gist.github.com/miccon/a4869fe04f9010015861 still fails.

I suppose that when the mapping is created after the failed indexing request, the _all mapping is set to true (by default) and then the _all cannot by set to false anymore.

IMHO not creating the type at all would be the cleaner solution, because even when the default mapping is set to strict creating the type can prevent you from updating the mapping later on (as the _all mapping is created automatically).

@clintongormley clintongormley added v1.4.3 and removed v1.4.2 labels Dec 16, 2014

@clintongormley clintongormley added v1.4.4 and removed v1.4.3 labels Feb 10, 2015

@spinscale spinscale added v1.4.5 and removed v1.4.4 labels Feb 19, 2015

brwe added a commit to brwe/elasticsearch that referenced this issue Feb 23, 2015
mappings: update cluster state with type mapping also for failed inde…
…xing request

When indexing of a document with a type that is not in the mappings fails,
for example because "dynamic": "strict" but doc contains a new field,
then the type is still created on the node that executed the indexing request.
However, the change was never added to the cluster state.
This commit makes sure mapping updates are always added to the cluster state
even if indexing of a document fails.

closes elastic#8650
brwe added a commit that referenced this issue Feb 24, 2015
mappings: update cluster state with type mapping also for failed inde…
…xing request

When indexing of a document with a type that is not in the mappings fails,
for example because "dynamic": "strict" but doc contains a new field,
then the type is still created on the node that executed the indexing request.
However, the change was never added to the cluster state.
This commit makes sure mapping updates are always added to the cluster state
even if indexing of a document fails.

closes #8692
relates to #8650
brwe added a commit that referenced this issue Feb 24, 2015
mappings: update cluster state with type mapping also for failed inde…
…xing request

When indexing of a document with a type that is not in the mappings fails,
for example because "dynamic": "strict" but doc contains a new field,
then the type is still created on the node that executed the indexing request.
However, the change was never added to the cluster state.
This commit makes sure mapping updates are always added to the cluster state
even if indexing of a document fails.

closes #8692
relates to #8650
brwe added a commit that referenced this issue Feb 24, 2015
mappings: update cluster state with type mapping also for failed inde…
…xing request

When indexing of a document with a type that is not in the mappings fails,
for example because "dynamic": "strict" but doc contains a new field,
then the type is still created on the node that executed the indexing request.
However, the change was never added to the cluster state.
This commit makes sure mapping updates are always added to the cluster state
even if indexing of a document fails.

closes #8692
relates to #8650
@brwe

This comment has been minimized.

Copy link
Contributor Author

commented Feb 24, 2015

I pushed the fix for the lost mappings but as @rjernst pointed out, not updating the mapping can only be done once #9365 is done so I'll leave this issue open.

@s1monw s1monw added v1.6.0 and removed v1.5.0 labels Mar 17, 2015

@clintongormley clintongormley added v1.5.3 and removed v1.4.5 labels Apr 26, 2015

@brwe

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2015

fixed by #10634

@brwe brwe closed this May 22, 2015

@clintongormley clintongormley changed the title mapping potentially lost with `"dynamic" : "strict"`, `_default_` mapping and failed document index Mapping potentially lost with `"dynamic" : "strict"`, `_default_` mapping and failed document index May 29, 2015

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
mappings: update cluster state with type mapping also for failed inde…
…xing request

When indexing of a document with a type that is not in the mappings fails,
for example because "dynamic": "strict" but doc contains a new field,
then the type is still created on the node that executed the indexing request.
However, the change was never added to the cluster state.
This commit makes sure mapping updates are always added to the cluster state
even if indexing of a document fails.

closes elastic#8692
relates to elastic#8650
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.