ES 2.x broken since 6.4.0

[codefromthecrypt]

I believe df2e678 broke elasticsearch 2.x support. It introduces the following state check which is invalid in 2.x when you initially encounter a result:

        if (responseHits > 0) {
            return new Scroll(scrollId, totalHits, results, responseHits, skippedHits);
        } else {
            // Scroll had no hits in the response, it must have concluded.
            return new Scroll(scrollId, totalHits, true);
        }

The "Scroll had no hits in the response, it must have concluded." assumption is invalid. As you'll notice, the initial response from ES 2.x has no contents in the hits section. Flipping the true to false works for ES 2.x, but I think it would be better to pass in the version and perhaps 5.x forward use the assumption that it is indeed complete.

4:12:46,847 TRACE [Executor task launch worker for task 4] commonshttp.CommonsHttpTransport - Tx [POST]@[localhost:32898][test_zipkin:span-2019-04-13/span/_search]?[search_type=scan&scroll=5m&size=50&preference=_shards%3A3%3B_local] w/ payload [{"query":{"match_all":{}}}]
14:12:46,850 DEBUG [Executor task launch worker for task 4] httpclient.Wire - >> "POST /test_zipkin:span-2019-04-13/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A3%3B_local HTTP/1.1[\r][\n]"
14:12:46,855 DEBUG [Executor task launch worker for task 4] httpclient.Wire - >> "Content-Type: application/json[\r][\n]"
14:12:46,857 DEBUG [Executor task launch worker for task 4] httpclient.Wire - >> "Accept: application/json[\r][\n]"
14:12:46,857 DEBUG [Executor task launch worker for task 4] httpclient.Wire - >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
14:12:46,857 DEBUG [Executor task launch worker for task 4] httpclient.Wire - >> "Host: localhost:32898[\r][\n]"
14:12:46,858 DEBUG [Executor task launch worker for task 4] httpclient.Wire - >> "Content-Length: 26[\r][\n]"
14:12:46,858 DEBUG [Executor task launch worker for task 4] httpclient.Wire - >> "[\r][\n]"
14:12:46,858 DEBUG [Executor task launch worker for task 4] httpclient.Wire - >> "{"query":{"match_all":{}}}"
14:12:46,929 DEBUG [Executor task launch worker for task 4] httpclient.Wire - << "HTTP/1.1 200 OK[\r][\n]"
14:12:46,929 DEBUG [Executor task launch worker for task 4] httpclient.Wire - << "HTTP/1.1 200 OK[\r][\n]"
14:12:46,929 DEBUG [Executor task launch worker for task 4] httpclient.Wire - << "Content-Type: application/json; charset=UTF-8[\r][\n]"
14:12:46,930 DEBUG [Executor task launch worker for task 4] httpclient.Wire - << "Content-Length: 202[\r][\n]"
14:12:46,930 DEBUG [Executor task launch worker for task 4] httpclient.Wire - << "[\r][\n]"
14:12:46,930 DEBUG [Executor task launch worker for task 4] httpclient.Wire - << "{"_scroll_id":"c2NhbjsxOzU6SEN2cmo0LVlRYUdUZEYyQXhScUdPdzsxO3RvdGFsX2hpdHM6MTs=","took":63,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":1,"max_score":0.0,"hits":[]}}"

As this glitch is rev-locking zipkin, it would be very appreciated if it could be sorted asap.

[jbaiera]

Thanks for reporting @adriancole, we'll take a look at this soon. Just an FYI - ES-Hadoop will be transitioning to a model where it only supports querying Elasticsearch back to the latest minor version on the previous major release. I'll see what I can do in this case since most of the BWC code has not been removed yet; I just want you to be aware of that change going forward.

[codefromthecrypt]

Thanks for the warning about the policy change on supported versions. Please keep things working for at least a little while longer so that we don't revlock people. We are trying to migrate zipkin to apache, and this policy might cause people to not move because they don't want us to break their old ES installs.

We will discuss how to communicate with users a kind of forced migration off ES 2.x and 5.x on our mailing list, but you can imagine that this is not something all sites will be willing to do.

[codefromthecrypt]

https://lists.apache.org/thread.html/73c2efa69e3ff0a519c6b6c2f5e551159c34902c29df01b2703e9126@%3Cdev.zipkin.apache.org%3E is a start on our discussion about this

[codefromthecrypt]

I'd expect this is a big issue for you.. any chance you'll be fixing this soon?

[jbaiera]

This should be fixed with #1287