New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPE in AggregationSpout when there is not any status index created #597

Closed
jcruzmartini opened this Issue Aug 3, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@jcruzmartini
Contributor

jcruzmartini commented Aug 3, 2018

When we are running crawler topology and there is not any index status created yet in ES, AggregationSpout is throwing a NPE.
We are using as index name in configuration file status* in order to get all indexes that match with that criteria.
The exception that we are getting is

17027 [I/O dispatcher 5] ERROR c.d.s.e.p.AggregationSpout - Exception with ES query
java.io.IOException: Unable to parse response body for Response{requestLine=POST /status*/status/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&preference=_shards%3A0&search_type=query_then_fetch&batched_reduce_size=512 HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK}
	at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:582) [stormcrawler-1.1.0.jar:?]
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:621) [stormcrawler-1.1.0.jar:?]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:375) [stormcrawler-1.1.0.jar:?]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [stormcrawler-1.1.0.jar:?]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
Caused by: java.lang.NullPointerException
	at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.onResponse(AggregationSpout.java:191) ~[stormcrawler-1.1.0.jar:?]
	at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.onResponse(AggregationSpout.java:71) [stormcrawler-1.1.0.jar:?]
	at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:580) [stormcrawler-1.1.0.jar:?]
	... 18 more
17028 [I/O dispatcher 1] ERROR c.d.s.e.p.AggregationSpout - Exception with ES query
java.io.IOException: Unable to parse response body for Response{requestLine=POST /status*/status/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&preference=_shards%3A2&search_type=query_then_fetch&batched_reduce_size=512 HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK}
	at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:582) [stormcrawler-1.1.0.jar:?]
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:621) [stormcrawler-1.1.0.jar:?]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:375) [stormcrawler-1.1.0.jar:?]
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [stormcrawler-1.1.0.jar:?]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [stormcrawler-1.1.0.jar:?]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
Caused by: java.lang.NullPointerException
	at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.onResponse(AggregationSpout.java:191) ~[stormcrawler-1.1.0.jar:?]
	at com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout.onResponse(AggregationSpout.java:71) [stormcrawler-1.1.0.jar:?]
	at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:580) [stormcrawler-1.1.0.jar:?]
	... 18 more

I think that the fix for this issue should be done in onResponse checking that results.getAggregations() is not null.

Aggregations aggregs = response.getAggregations();

Thanks in advance @jnioche and let me know if that works for you I can create a PR for this too

@jnioche jnioche added this to the 1.11 milestone Aug 5, 2018

@jcruzmartini

This comment has been minimized.

Contributor

jcruzmartini commented Aug 10, 2018

We added a workaround for this issue in our code, basically what we did is extend AggregationSpout and overrride onResponse method.

  @Override
    public void onResponse(SearchResponse response) {

        Aggregations aggregs = response.getAggregations();
		if (aggregs == null) {
			isInESQuery.set(false);
			return;
		}
		super.onResponse(response);
    }

this solved the issue, response.getAggregations() is null when there is no any index

@jnioche jnioche closed this in 796c2b8 Aug 15, 2018

@jnioche

This comment has been minimized.

Member

jnioche commented Aug 15, 2018

Fixed, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment