Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NullPointerException in RamAccounntingTermsEnum #5326

Closed
justinuang opened this issue Mar 3, 2014 · 8 comments

Comments

@justinuang
Copy link

commented Mar 3, 2014

In one of my custom scripts, I call:

(ScriptDocValues.Strings) doc().get("_uid");

which then throws:

Caused by: java.lang.NullPointerException
    at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.bytesPerValue(PagedBytesIndexFieldData.java:147)
    at org.elasticsearch.index.fielddata.RamAccountingTermsEnum.next(RamAccountingTermsEnum.java:84)
    at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:99)
    at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:41)
    at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:139)
    at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:135)
    at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4739)
    at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
    at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
    at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
    at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2195)

It looks like the offending code is in org.elasticsearch.index.fielddata.RamAccountingTermsEnum, which has logic like

public BytesRef next() throws IOException {
    BytesRef term = termsEnum.next();
    if (term == null && this.flushBuffer != 0) {
        // We have reached the end of the termsEnum, flush the buffer
        flush();
    } else {
        this.flushBuffer += estimator.bytesPerValue(term);
        if (this.flushBuffer >= FLUSH_BUFFER_SIZE) {
            flush();
        }
    }
    return term;
}

What happens when term == null but this.flushBuffer == 0? Won't it throw an exception? Is there an invariant that I'm not aware of? Unfortunately, I can't find a way to repro this.

I have attached the full stack trace below:

2014-03-01 11:31:28,820 +0000 ERROR [elasticsearch[Jacqueline Falsworth][search][T#17]] search.MyScript - Exception encountered while executing MyScript
org.elasticsearch.ElasticsearchException: java.lang.NullPointerException
    at org.elasticsearch.index.fielddata.AbstractIndexFieldData.load(AbstractIndexFieldData.java:75)
    at org.elasticsearch.search.lookup.DocLookup.get(DocLookup.java:101)
    at com.MyCompany.MyScript.getStringForField(MyScript.java:158) // hidden name
    at com.MyCompany.MyScript.getIdentifierForDocument(MyScript.java:140)
    at com.MyCompany.MyScript.run(MyScript.java:101)
    at org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:184)
    at org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67)
    at org.elasticsearch.common.lucene.docset.AndDocIdSet$AndBits.get(AndDocIdSet.java:106)
    at org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60)
    at org.apache.lucene.search.Scorer.score(Scorer.java:65)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:122)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:322)
    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:304)
    at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:71)
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
    at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
    at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3934)
    at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4736)
    at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:135)
    at org.elasticsearch.index.fielddata.AbstractIndexFieldData.load(AbstractIndexFieldData.java:69)
    ... 25 more
Caused by: java.lang.NullPointerException
    at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.bytesPerValue(PagedBytesIndexFieldData.java:147)
    at org.elasticsearch.index.fielddata.RamAccountingTermsEnum.next(RamAccountingTermsEnum.java:84)
    at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:99)
    at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:41)
    at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:139)
    at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:135)
    at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4739)
    at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
    at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
    at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
    at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
    ... 29 more
@justinuang

This comment has been minimized.

Copy link
Author

commented Mar 3, 2014

What happens if the buffer is flushed right after the last non-null element in the termsEnum is iterated over? Won't this.flushBuffer == 0, and a NullPointerException will be thrown?

I see that this circuit breaking functionality was added by @dakrone in commit a754224.

@dakrone

This comment has been minimized.

Copy link
Member

commented Mar 4, 2014

@justinuang I see the issue in bytesPerValue in PagedBytesIndexFieldData, I will fix this.

@dakrone dakrone added bug labels Mar 4, 2014
@justinuang

This comment has been minimized.

Copy link
Author

commented Mar 4, 2014

Thanks so much! If you don't mind, can you explain what the bug is?

@dakrone

This comment has been minimized.

Copy link
Member

commented Mar 4, 2014

It is like you said in the .next() method, if the term is null right after a flush we still end up calling the .bytesPerValue() function on it. I'm going to add a check in .next() that for null terms and returns null immediately.

@justinuang

This comment has been minimized.

Copy link
Author

commented Mar 4, 2014

Awesome. Thanks for your expertise. I'm guessing that your previous comment doesn't really apply to this issue then?

@justinuang I see the issue in bytesPerValue in PagedBytesIndexFieldData, I will fix this.

@dakrone

This comment has been minimized.

Copy link
Member

commented Mar 4, 2014

No, it does, that's where the actual NPE is coming from, because I assume term can never be null in .bytesPerValue() (which was incorrect)

@justinuang

This comment has been minimized.

Copy link
Author

commented Mar 4, 2014

Ah I see. Thanks again!

@justinuang

This comment has been minimized.

Copy link
Author

commented Mar 4, 2014

Cool. Do you know when the next release will be? It's hitting us in production.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.