Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPE in BytesRefOrdValComparator #3470

Closed
ghost opened this Issue Aug 9, 2013 · 13 comments

Comments

Projects
None yet
6 participants
@ghost
Copy link

ghost commented Aug 9, 2013

This issue relates to issue #3189 and I'm experiencing it with v1.0.0.Beta, but I guess that applies to 0.90 branch as well.

While sorting on some trivial fields I get the following error message:

{
   "error": "SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[UIomKhmqSzumD0uzjmRbfA][brisa][0]: QueryPhaseExecutionException[[brisa][0]: query[filtered(ConstantScore(+NotFilter(cache(discarded:T))))->cache(_type:patients)],from[0],size[20],sort[<custom:\"lastname\": org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource@1d8f4c01>]: Query Failed [Failed to execute main query]]; nested: NullPointerException; }]",
   "status": 500
}

lastname mapping is as follows:

"lastname": {
  "type": "string",
  "index": "not_analyzed",
  "omit_norms": true,
  "index_options": "docs"
}

Fairly standard IMO.

Now the funny thing is that I have another name mapping which is exactly the same and everything works just fine. From a data point of view, I cannot see a difference between one field and the other (upper case strings).

The NPE happens in BytesRefOrdValComparator at line 388 because MultiDocs.ordinals() returns null.

Now, i don't know why lastname happens to use MultiOrdinals (or MultiDocs for that matter) to do sorting, but making MultiDocs return its MultiOrdinals instance seems to fix the issue, although I'm not sure whether the null pointer was intentional and the problem is somewhere else.

Thoughts?

Thanks a lot in advance.

@ghost ghost assigned s1monw Aug 9, 2013

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 9, 2013

Thanks for opening this! This is in-fact a bug that is triggered depending on how sparse your data in that field is. I will come up with a patch very soon and I try to increase the test coverage for this to make sure we are not running into this again.

s1monw added a commit to s1monw/elasticsearch that referenced this issue Aug 9, 2013

Added random sort test for dense and sparse fields.
This test triggers a MultiDoc / MultiOrds in-memory representation
even if the field is not multivalued

Relates to elastic#3470

@s1monw s1monw closed this in 417c193 Aug 9, 2013

s1monw added a commit that referenced this issue Aug 9, 2013

Return ordinals from MultiOrdinals.MultiDocs
MultiOrdinals.MultiDocs returned 'null' ordinals which caused
a NPE if the field was single valued and would allow a significantly
smaller in memory representation than single packed int ordinals.

Closes #3470

s1monw added a commit that referenced this issue Aug 9, 2013

Added random sort test for dense and sparse fields.
This test triggers a MultiDoc / MultiOrds in-memory representation
even if the field is not multivalued

Relates to #3470
@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Aug 9, 2013

thanks for reporting this so quickly!

@ghost

This comment has been minimized.

Copy link
Author

ghost commented Aug 9, 2013

Hey, thank YOU for fixing it so quickly!

ES is an amazing project, I'm just glad I can help.

--- Original Message ---

From: "Simon Willnauer" notifications@github.com
Sent: August 9, 2013 9:44 AM
To: "elasticsearch/elasticsearch" elasticsearch@noreply.github.com
Cc: "dngdevelopment" kayowas@hotmail.com
Subject: Re: [elasticsearch] NPE in BytesRefOrdValComparator (#3470)

thanks for reporting this so quickly!


Reply to this email directly or view it on GitHub:
#3470 (comment)

@elasticjava

This comment has been minimized.

Copy link
Contributor

elasticjava commented Aug 13, 2013

Many thanks Simon! that error filled our log!! +1

fizx added a commit to fizx/elasticsearch that referenced this issue Aug 24, 2013

Return ordinals from MultiOrdinals.MultiDocs
MultiOrdinals.MultiDocs returned 'null' ordinals which caused
a NPE if the field was single valued and would allow a significantly
smaller in memory representation than single packed int ordinals.

Closes elastic#3470
@tarunjangra

This comment has been minimized.

Copy link

tarunjangra commented Sep 2, 2013

Great guys, when you would be able to release this bug. We are using this very heavily. +1

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Sep 2, 2013

@tarunjangra it's coming very soon I hope we can move towards a release next week or so.

@chrisamccoy

This comment has been minimized.

Copy link

chrisamccoy commented Sep 3, 2013

@tarunjangra

This comment has been minimized.

Copy link

tarunjangra commented Sep 3, 2013

@s1monw i would like to share this behaviour. Query i am doing:
{"query":{"match_all":{}},"sort":{"title.untouched":{"order":"desc"}}}

And result i am getting:

QueryPhaseExecutionException[[es_index][11]: query[ConstantScore(cache(org.elasticsearch.index.search.nested.NonNestedDocsFilter@953b6c6c))],from[0],size[10],sort[<custom:"title.untouched": org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource@1ca8b405>!]: Query Failed [Failed to execute main query]]; nested: NullPointerException;

Some thing interesting i just noticed. I am also getting some stats about Shards.

total: 15
successful: 10
failed: 5

But yesterday i was getting

total: 15
successful: 6
failed: 9

the question is, why these stats are changing? Is that some how usual ?

Thank you.

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Sep 3, 2013

@tarunjangra this issue depends on a lot of things if your field data implementation uses the "non-broken" code paths due to segment merges you won't see the failure on all shards so this can change over time.

@pbellora

This comment has been minimized.

Copy link

pbellora commented Sep 3, 2013

Is there a known workaround in the meantime? I was also using not_analyzed for sorting, and that's breaking with an NPE in BytesRefFieldComparatorSource (I'm assuming it's the same issue). I tried using keyword analysis instead but got the same failures.

@tarunjangra

This comment has been minimized.

Copy link

tarunjangra commented Sep 4, 2013

@s1monw Yep: It make sense. We are waiting for next elasticsearch release. Would like to recognise all elasticsearch team for such a great community efforts.

Thank you.

@mjebrini

This comment has been minimized.

Copy link

mjebrini commented Dec 25, 2013

Hi I am searching with following query:

{
"query": {
"filtered": {
"query": {
"match_all": []
},
"filter": {
"and": [
{
"query": {
"match": {
"status": "published"
}
}
},
{
"numeric_range": {
"rate": {
"gte": 0,
"lte": 5
}
}
}
]
}
}
},
"from": 1,
"size": 5,
"sort": [
{
"activity_id": "desc"
}
],
"fields": [
"activity_id"
]
}

After all I get into this exception:

QueryPhaseExecutionException[[activity][0]: query[ConstantScore(+QueryWrapperFilter(status:published) +rate:[0 TO 5])],from[1],size[5],sort[<custom:"activity_id": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@bda96b>!]: Query Failed [Failed to execute main query]]; nested: ElasticSearchException[java.lang.NumberFormatException: Invalid shift value in prefixCoded bytes (is encoded value really an INT?)]; nested: UncheckedExecutionException[java.lang.NumberFormatException: Invalid shift value in prefixCoded bytes (is encoded value really an INT?)]; nested: NumberFormatException[Invalid shift value in prefixCoded bytes (is encoded value really an INT?)];

Actually this is weird, i was facing the problem at 0.90.5 .. then I upgraded my elastic search binary to 0.90.9 and I still face the same error.

@mjebrini

This comment has been minimized.

Copy link

mjebrini commented Dec 25, 2013

I think the sorting term is the cause of the problem!

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Return ordinals from MultiOrdinals.MultiDocs
MultiOrdinals.MultiDocs returned 'null' ordinals which caused
a NPE if the field was single valued and would allow a significantly
smaller in memory representation than single packed int ordinals.

Closes elastic#3470

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Added random sort test for dense and sparse fields.
This test triggers a MultiDoc / MultiOrds in-memory representation
even if the field is not multivalued

Relates to elastic#3470
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.