Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NullPointerException during concurrent merges #3555

Closed
rdeaton opened this Issue Aug 21, 2013 · 5 comments

Comments

Projects
None yet
2 participants
@rdeaton
Copy link

rdeaton commented Aug 21, 2013

Looks like it may be related to the new suggestion stuff. I'm on 0.90.3, Java 7. Will be trying to reproduce on master shortly, but let me know if there's other stuff you need to debug.

[2013-08-21 22:02:05,166][WARN ][index.engine.robin ] [qanticharm] [quizlet][9] failed engine
org.apache.lucene.index.MergePolicy$MergeException: java.lang.NullPointerException
at org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:99)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.NullPointerException
at org.elasticsearch.search.suggest.completion.AnalyzingCompletionLookupProvider$1$1.finish(AnalyzingCompletionLookupProvider.java:134)
at org.elasticsearch.search.suggest.completion.Completion090PostingsFormat$SuggestFieldsConsumer$1.finish(Completion090PostingsFormat.java:152)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:204)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2013-08-21 22:02:05,180][WARN ][cluster.action.shard ] [qanticharm] sending failed shard for [quizlet][9], node[27LWb8rFQiafhZhgRH4vjg], [R], s[STARTED], reason [engine failure, message [MergeException[java.lang.NullPointerException]; nested: NullPointerException; ]]

@rdeaton

This comment has been minimized.

Copy link
Author

rdeaton commented Aug 22, 2013

The problem persists on master.

@ghost ghost assigned s1monw Aug 22, 2013

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 22, 2013

I think I know what the problem is.... your segment might have no docs in that field... I will try to write a test and come up with a patch

@s1monw s1monw closed this in fc3133d Aug 22, 2013

s1monw added a commit that referenced this issue Aug 22, 2013

Prevent NPE if all docs for the field are pruned during merge
During segment merges FieldConsumers for all fields from the source
segments are created. Yet, if all documents that have a value in a
certain field  are deleted / pruned during the merge the FieldConsumer
will not be called at all causing the internally used FST Builder to
return `null` from it's `build()` method. This means we can't store
it and run potentially into a NPE. This commit adds handling for
this corner case both during merge and during suggest phase since
we also don't have a Lookup instance for this segments field.

Closes #3555
@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 22, 2013

@rdeaton can you try if that fixes your issue? Thanks

@rdeaton

This comment has been minimized.

Copy link
Author

rdeaton commented Aug 22, 2013

Seems to be working. Thanks a bunch.

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 22, 2013

thanks for verifying!!!

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Prevent NPE if all docs for the field are pruned during merge
During segment merges FieldConsumers for all fields from the source
segments are created. Yet, if all documents that have a value in a
certain field  are deleted / pruned during the merge the FieldConsumer
will not be called at all causing the internally used FST Builder to
return `null` from it's `build()` method. This means we can't store
it and run potentially into a NPE. This commit adds handling for
this corner case both during merge and during suggest phase since
we also don't have a Lookup instance for this segments field.

Closes elastic#3555
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.