Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"no index mapper found for field" bulk indexing in 0.90 #3088

Closed
philayres opened this Issue May 24, 2013 · 7 comments

Comments

Projects
None yet
3 participants
@philayres
Copy link

philayres commented May 24, 2013

Today I've been getting horrible performance on 0.90 server I upgraded last week. Running on Ubuntu, Ruby 1.9.3, Rails 3.2.3, with the Stretcher gem.

The issue is in bulk indexing Twitter data, with JSON very similar to the response shown at: https://dev.twitter.com/docs/api/1.1/get/statuses/user_timeline, except the user node of each item is removed and is instead a parent document (although I don't think that is the issue).

My mappings are intentionally kept simple, using dynamic mappings to add fields. Everything has been working fine to this point. Now I'm getting the following error, bringing the server to its knees.

The error is:
[2013-05-23 22:13:01,792][WARN ][index.merge.scheduler ] [Rocket Racer] [twitter_history_76816072][1] failed to merge
org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [entities.description.urls.display_url]
at org.elasticsearch.index.codec.PerFieldMappingPostingFormatCodec.getPostingsFormatForField(PerFieldMappingPostingFormatCodec.java:52)
at org.apache.lucene.codecs.lucene42.Lucene42Codec$1.getPostingsFormatForField(Lucene42Codec.java:59)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:102)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:71)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
[2013-05-23 22:13:01,939][WARN ][index.engine.robin ] [Rocket Racer] [twitter_history_76816072][1] failed engine
org.apache.lucene.index.MergePolicy$MergeException: org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [entities.description.urls.display_url]
at org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:100)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
Caused by: org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [entities.description.urls.display_url]
at org.elasticsearch.index.codec.PerFieldMappingPostingFormatCodec.getPostingsFormatForField(PerFieldMappingPostingFormatCodec.java:52)
at org.apache.lucene.codecs.lucene42.Lucene42Codec$1.getPostingsFormatForField(Lucene42Codec.java:59)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:102)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:71)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
[2013-05-23 22:13:04,187][WARN ][cluster.action.shard ] [Rocket Racer] sending failed shard for [twitter_history_76816072][1], node[_1cPivHASUOzVcgQ2fK0_A], [P], s[INITIALIZING], reason [engine failure, message [MergeException[org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [entities.description.urls.display_url]]; nested: ElasticSearchIllegalStateException[no index mapper found for field: [entities.description.urls.display_url]]; ]]
[2013-05-23 22:13:04,234][WARN ][cluster.action.shard ] [Rocket Racer] received shard failed for [twitter_history_76816072][1], node[_1cPivHASUOzVcgQ2fK0_A], [P], s[INITIALIZING], reason [engine failure, message [MergeException[org.elasticsearch.ElasticSearchIllegalStateException: no index mapper found for field: [entities.description.urls.display_url]]; nested: ElasticSearchIllegalStateException[no index mapper found for field: [entities.description.urls.display_url]]; ]]
[2013-05-23 22:13:07,433][WARN ][indices.cluster ] [Rocket Racer] [twitter_history_76816072][1] master [[Rocket Racer][_1cPivHASUOzVcgQ2fK0_A][inet[/10.178.13.201:9300]]] marked shard as started, but shard have not been created, mark shard as failed
[2013-05-23 22:13:07,446][WARN ][cluster.action.shard ] [Rocket Racer] sending failed shard for [twitter_history_76816072][1], node[_1cPivHASUOzVcgQ2fK0_A], [P], s[STARTED], reason [master [Rocket Racer][_1cPivHASUOzVcgQ2fK0_A][inet[/10.178.13.201:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2013-05-23 22:13:07,446][WARN ][cluster.action.shard ] [Rocket Racer] received shard failed for [twitter_history_76816072][1], node[_1cPivHASUOzVcgQ2fK0_A], [P], s[STARTED], reason [master [Rocket Racer][_1cPivHASUOzVcgQ2fK0_A][inet[/10.178.13.201:9300]] marked shard as started, but shard have not been created, mark shard as failed]

The best guess I have is that some random data is breaking the creation of new mappings. The same is happening on the same index in both production and test environments. Other indexes with the same initial mappings but different Twitter data content are not exhibiting the same error.

I'm trying to capture the data, although there is a mass of it. Therefore before I go too far, has anybody got any clue how I can start to identify the source of the problem?

@philayres

This comment has been minimized.

Copy link
Author

philayres commented May 24, 2013

An additional note. I found this similar issue for another user: http://www.marshut.com/kpnht/problems-with-rabbitmq-river-90rc2.html that I think is unrelated to their RabbitMQ river, but is the same issue as I am seeing. No resolution over there apart from to unplug the river.

@ghost ghost assigned s1monw May 24, 2013

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented May 24, 2013

no matter why this happens it's bad to fail with a corrupt index (which is essentially what an exception during merge is) We should track down the race but first of all I think we should return the default postings format instead of failing with an exception

s1monw added a commit to s1monw/elasticsearch that referenced this issue May 24, 2013

Never throw an IAE if the IndexMapper isn't present in PostingsFormat
If we throw an exception in the PostingsFormat during a merge we essentially
fail the entire merge which can lead to a corrupt index. We should rather
return the default postings format for the new segment and log a warning.

Closes elastic#3088

@s1monw s1monw closed this in 6e366ba May 24, 2013

s1monw added a commit that referenced this issue May 24, 2013

Never throw an IAE if the IndexMapper isn't present in PostingsFormat
If we throw an exception in the PostingsFormat during a merge we essentially
fail the entire merge which can lead to a corrupt index. We should rather
return the default postings format for the new segment and log a warning.

Closes #3088
@philayres

This comment has been minimized.

Copy link
Author

philayres commented May 25, 2013

Thanks for the incredibly fast response.

I see from the commits we are now logging the issue rather than raising an exception. What effect does this have on the automatic creation of a new element in the mapping if we hit this condition? Do we just skip it, or is the mapping now updated accordingly? My concern is just that we would be arbitrarily not indexing elements in a document based on this error. Or is this not really an error condition and I'm misinterpreting?

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented May 25, 2013

the condition you are seening is certainly not correc, the mapper for a given field is not present in the mapping which can have multiple reasons which I am trying to figure out at the moment. My suspicion is that due to some index deleting or type deletion the mappings are pruned and therefor a concurrently running merge can't access this mapping. So what happens if we return the default postings format rather than a per field format is in the worst case the default field format for the current lucene version you are using (which in your case is very likely identical). That won't be a problem since we pruned the mapper anyways so this field will likely not be accessed anymore. I am still trying to reproduce this but your docs will be updated correctly.

@philayres

This comment has been minimized.

Copy link
Author

philayres commented May 25, 2013

Simon, I'd be happy to spin up a copy of this server so you can take a look directly at the mappings and data if that would help to reproduce this situation. Just let me know.

Just in case it is useful, I spotted the field in question (entities.description.urls.display_url) potentially having its mapping set when the data being passed included a 'special' character. I don't know if the actual field value could affect the mapping creation. The data was a regular ascii string ending with an elipsis (hex E2 80 A6). This may be unrelated, it just seemed unusual.

@ajhalani

This comment has been minimized.

Copy link

ajhalani commented May 30, 2013

This issue affected us for regular indexing as well. We suspect this was caused when in 0.20.x, we overwrote existing mapping with new mapping which had fewer fields. The data somehow still had the old fields and merge failed with 0.90.0.
With 0.90.1, we are now getting warnings but doesn't fail. thanks!

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented May 30, 2013

cool thanks for reporting

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Never throw an IAE if the IndexMapper isn't present in PostingsFormat
If we throw an exception in the PostingsFormat during a merge we essentially
fail the entire merge which can lead to a corrupt index. We should rather
return the default postings format for the new segment and log a warning.

Closes elastic#3088
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.