Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPE in BytesRefOrdValComparator #3470

Closed
ghost opened this issue Aug 9, 2013 · 13 comments
Closed

NPE in BytesRefOrdValComparator #3470

ghost opened this issue Aug 9, 2013 · 13 comments

Comments

@ghost
Copy link

ghost commented Aug 9, 2013

This issue relates to issue #3189 and I'm experiencing it with v1.0.0.Beta, but I guess that applies to 0.90 branch as well.

While sorting on some trivial fields I get the following error message:

{
   "error": "SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[UIomKhmqSzumD0uzjmRbfA][brisa][0]: QueryPhaseExecutionException[[brisa][0]: query[filtered(ConstantScore(+NotFilter(cache(discarded:T))))->cache(_type:patients)],from[0],size[20],sort[<custom:\"lastname\": org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource@1d8f4c01>]: Query Failed [Failed to execute main query]]; nested: NullPointerException; }]",
   "status": 500
}

lastname mapping is as follows:

"lastname": {
  "type": "string",
  "index": "not_analyzed",
  "omit_norms": true,
  "index_options": "docs"
}

Fairly standard IMO.

Now the funny thing is that I have another name mapping which is exactly the same and everything works just fine. From a data point of view, I cannot see a difference between one field and the other (upper case strings).

The NPE happens in BytesRefOrdValComparator at line 388 because MultiDocs.ordinals() returns null.

Now, i don't know why lastname happens to use MultiOrdinals (or MultiDocs for that matter) to do sorting, but making MultiDocs return its MultiOrdinals instance seems to fix the issue, although I'm not sure whether the null pointer was intentional and the problem is somewhere else.

Thoughts?

Thanks a lot in advance.

@ghost ghost assigned s1monw Aug 9, 2013
@s1monw
Copy link
Contributor

s1monw commented Aug 9, 2013

Thanks for opening this! This is in-fact a bug that is triggered depending on how sparse your data in that field is. I will come up with a patch very soon and I try to increase the test coverage for this to make sure we are not running into this again.

s1monw added a commit to s1monw/elasticsearch that referenced this issue Aug 9, 2013
This test triggers a MultiDoc / MultiOrds in-memory representation
even if the field is not multivalued

Relates to elastic#3470
@s1monw s1monw closed this as completed in 417c193 Aug 9, 2013
s1monw added a commit that referenced this issue Aug 9, 2013
MultiOrdinals.MultiDocs returned 'null' ordinals which caused
a NPE if the field was single valued and would allow a significantly
smaller in memory representation than single packed int ordinals.

Closes #3470
s1monw added a commit that referenced this issue Aug 9, 2013
This test triggers a MultiDoc / MultiOrds in-memory representation
even if the field is not multivalued

Relates to #3470
@s1monw
Copy link
Contributor

s1monw commented Aug 9, 2013

thanks for reporting this so quickly!

@ghost
Copy link
Author

ghost commented Aug 9, 2013

Hey, thank YOU for fixing it so quickly!

ES is an amazing project, I'm just glad I can help.

--- Original Message ---

From: "Simon Willnauer" notifications@github.com
Sent: August 9, 2013 9:44 AM
To: "elasticsearch/elasticsearch" elasticsearch@noreply.github.com
Cc: "dngdevelopment" kayowas@hotmail.com
Subject: Re: [elasticsearch] NPE in BytesRefOrdValComparator (#3470)

thanks for reporting this so quickly!


Reply to this email directly or view it on GitHub:
#3470 (comment)

@elasticjava
Copy link
Contributor

Many thanks Simon! that error filled our log!! +1

fizx pushed a commit to fizx/elasticsearch that referenced this issue Aug 24, 2013
MultiOrdinals.MultiDocs returned 'null' ordinals which caused
a NPE if the field was single valued and would allow a significantly
smaller in memory representation than single packed int ordinals.

Closes elastic#3470
@tarunjangra
Copy link

Great guys, when you would be able to release this bug. We are using this very heavily. +1

@s1monw
Copy link
Contributor

s1monw commented Sep 2, 2013

@tarunjangra it's coming very soon I hope we can move towards a release next week or so.

@chrisamccoy
Copy link

@s1monw Yes, we'd love a fix for this bug. We're using it heavily at Yoursports.com http://techcrunch.com/2012/10/11/sports-section-2-0-yoursports-launches-its-ambitious-project-to-build-the-facebook-of-sports/

@tarunjangra
Copy link

@s1monw i would like to share this behaviour. Query i am doing:
{"query":{"match_all":{}},"sort":{"title.untouched":{"order":"desc"}}}

And result i am getting:

QueryPhaseExecutionException[[es_index][11]: query[ConstantScore(cache(org.elasticsearch.index.search.nested.NonNestedDocsFilter@953b6c6c))],from[0],size[10],sort[<custom:"title.untouched": org.elasticsearch.index.fielddata.fieldcomparator.BytesRefFieldComparatorSource@1ca8b405>!]: Query Failed [Failed to execute main query]]; nested: NullPointerException;

Some thing interesting i just noticed. I am also getting some stats about Shards.

total: 15
successful: 10
failed: 5

But yesterday i was getting

total: 15
successful: 6
failed: 9

the question is, why these stats are changing? Is that some how usual ?

Thank you.

@s1monw
Copy link
Contributor

s1monw commented Sep 3, 2013

@tarunjangra this issue depends on a lot of things if your field data implementation uses the "non-broken" code paths due to segment merges you won't see the failure on all shards so this can change over time.

@pbellora
Copy link

pbellora commented Sep 3, 2013

Is there a known workaround in the meantime? I was also using not_analyzed for sorting, and that's breaking with an NPE in BytesRefFieldComparatorSource (I'm assuming it's the same issue). I tried using keyword analysis instead but got the same failures.

@tarunjangra
Copy link

@s1monw Yep: It make sense. We are waiting for next elasticsearch release. Would like to recognise all elasticsearch team for such a great community efforts.

Thank you.

@mjebrini
Copy link

Hi I am searching with following query:

{
"query": {
"filtered": {
"query": {
"match_all": []
},
"filter": {
"and": [
{
"query": {
"match": {
"status": "published"
}
}
},
{
"numeric_range": {
"rate": {
"gte": 0,
"lte": 5
}
}
}
]
}
}
},
"from": 1,
"size": 5,
"sort": [
{
"activity_id": "desc"
}
],
"fields": [
"activity_id"
]
}

After all I get into this exception:

QueryPhaseExecutionException[[activity][0]: query[ConstantScore(+QueryWrapperFilter(status:published) +rate:[0 TO 5])],from[1],size[5],sort[<custom:"activity_id": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@bda96b>!]: Query Failed [Failed to execute main query]]; nested: ElasticSearchException[java.lang.NumberFormatException: Invalid shift value in prefixCoded bytes (is encoded value really an INT?)]; nested: UncheckedExecutionException[java.lang.NumberFormatException: Invalid shift value in prefixCoded bytes (is encoded value really an INT?)]; nested: NumberFormatException[Invalid shift value in prefixCoded bytes (is encoded value really an INT?)];

Actually this is weird, i was facing the problem at 0.90.5 .. then I upgraded my elastic search binary to 0.90.9 and I still face the same error.

@mjebrini
Copy link

I think the sorting term is the cause of the problem!

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
MultiOrdinals.MultiDocs returned 'null' ordinals which caused
a NPE if the field was single valued and would allow a significantly
smaller in memory representation than single packed int ordinals.

Closes elastic#3470
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
This test triggers a MultiDoc / MultiOrds in-memory representation
even if the field is not multivalued

Relates to elastic#3470
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants