Scroll Id regression on Elasticsearch 2.4.5

[codefromthecrypt]

Zipkin is currently testing Elasticsearch 6.x for our dependency linking spark job and we noticed that latest snapshots break 2.x. Looking at wire logs, it seems this is around scroll ids, as you'll notice that with snapshot no scrolls are used, eventhough 5.5.1 is ok. I reverted 1190be7 and things are back to normal. I suspect there's something subtle missed.

Log from version 5.5.1, note presence of POST /_search/scroll?scroll=5m

17/08/08 13:34:40 INFO ElasticsearchDependenciesJob: Processing spans from test_zipkin_dependency-2017-08-08/span
17/08/08 13:35:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/08 13:35:11 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "GET /_cluster/health/test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Length: 389[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "[\r][\n]"
17/08/08 13:35:11 DEBUG content: << "{"cluster_name":"elasticsearch","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}"
17/08/08 13:35:11 DEBUG header: >> "GET / HTTP/1.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Length: 364[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "[\r][\n]"
17/08/08 13:35:11 DEBUG content: << "{[\n]"
17/08/08 13:35:11 DEBUG content: << "  "name" : "Amber Hunt",[\n]"
17/08/08 13:35:11 DEBUG content: << "  "cluster_name" : "elasticsearch",[\n]"
17/08/08 13:35:11 DEBUG content: << "  "cluster_uuid" : "yhWaWc2jRaWNz2u6W2TbLQ",[\n]"
17/08/08 13:35:11 DEBUG content: << "  "version" : {[\n]"
17/08/08 13:35:11 DEBUG content: << "    "number" : "2.4.5",[\n]"
17/08/08 13:35:11 DEBUG content: << "    "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276",[\n]"
17/08/08 13:35:11 DEBUG content: << "    "build_timestamp" : "2017-04-24T16:18:17Z",[\n]"
17/08/08 13:35:11 DEBUG content: << "    "build_snapshot" : false,[\n]"
17/08/08 13:35:11 DEBUG content: << "    "lucene_version" : "5.5.4"[\n]"
17/08/08 13:35:11 DEBUG content: << "  },[\n]"
17/08/08 13:35:11 DEBUG content: << "  "tagline" : "You Know, for Search"[\n]"
17/08/08 13:35:11 DEBUG content: << "}[\n]"
17/08/08 13:35:11 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08/span HTTP/1.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "GET /test_zipkin_dependency-2017-08-08/_search_shards HTTP/1.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Length: 1153[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "[\r][\n]"
17/08/08 13:35:11 DEBUG content: << "{"nodes":{"sZf5lCQfRomZi7ZzPag3MQ":{"name":"Amber Hunt","transport_address":"172.17.0.6:9300","attributes":{}}},"shards":[[{"state":"STARTED","primary":true,"node":"sZf5lCQfRomZi7ZzPag3MQ","relocating_node":null,"shard":0,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"uLa2nlToQp-BYhCG_Xgb3w"}}],[{"state":"STARTED","primary":true,"node":"sZf5lCQfRomZi7ZzPag3MQ","relocating_node":null,"shard":1,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"sQtBMY8NR52OQ2_0l7IvQw"}}],[{"state":"STARTED","primary":true,"node":"sZf5lCQfRomZi7ZzPag3MQ","relocating_node":null,"shard":2,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"7gYXJ4AiR6WoEhXU0AtSQA"}}],[{"state":"STARTED","primary":true,"node":"sZf5lCQfRomZi7ZzPag3MQ","relocating_node":null,"shard":3,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"SLpufC8ySVmV7krxWD7kHQ"}}],[{"state":"STARTED","primary":true,"node":"sZf5lCQfRomZi7ZzPag3MQ","relocating_node":null,"shard":4,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"gnOnhHCsTsaigWlIdRaFvg"}}]]}"
17/08/08 13:35:11 DEBUG header: >> "GET /test_zipkin_dependency-2017-08-08/span/_mapping HTTP/1.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:11 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "Content-Length: 925[\r][\n]"
17/08/08 13:35:11 DEBUG header: << "[\r][\n]"
17/08/08 13:35:11 DEBUG content: << "{"test_zipkin_dependency-2017-08-08":{"mappings":{"span":{"_all":{"enabled":false},"properties":{"annotations":{"type":"nested","dynamic":"false","properties":{"endpoint":{"dynamic":"false","properties":{"serviceName":{"type":"string","index":"not_analyzed","ignore_above":256}}},"value":{"type":"string","index":"not_analyzed","ignore_above":256}}},"binaryAnnotations":{"type":"nested","dynamic":"false","properties":{"endpoint":{"dynamic":"false","properties":{"serviceName":{"type":"string","index":"not_analyzed","ignore_above":256}}},"key":{"type":"string","index":"not_analyzed","ignore_above":256},"value":{"type":"string","index":"not_analyzed","ignore_above":256}}},"duration":{"type":"long"},"id":{"type":"string"},"name":{"type":"string","index":"not_analyzed","ignore_above":256},"timestamp_millis":{"type":"date","format":"epoch_millis"},"traceId":{"type":"string","index":"not_analyzed","ignore_above":256}}}}}}"
17/08/08 13:35:12 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A2%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A3%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A0%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A1%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A4%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:35:12 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzExOnNaZjVsQ1FmUm9tWmk3WnpQYWczTVE7MTt0b3RhbF9oaXRzOjE7","took":4,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":1,"max_score":0.0,"hits":[]}}"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzE0OnNaZjVsQ1FmUm9tWmk3WnpQYWczTVE7MTt0b3RhbF9oaXRzOjA7","took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}"
17/08/08 13:35:12 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzEyOnNaZjVsQ1FmUm9tWmk3WnpQYWczTVE7MTt0b3RhbF9oaXRzOjA7","took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzEzOnNaZjVsQ1FmUm9tWmk3WnpQYWczTVE7MTt0b3RhbF9oaXRzOjA7","took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzE1OnNaZjVsQ1FmUm9tWmk3WnpQYWczTVE7MTt0b3RhbF9oaXRzOjA7","took":4,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}"
17/08/08 13:35:12 DEBUG header: >> "POST /_search/scroll?scroll=5m HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 80[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG content: >> "{"scroll_id":"c2NhbjsxOzExOnNaZjVsQ1FmUm9tWmk3WnpQYWczTVE7MTt0b3RhbF9oaXRzOjE7"}"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 750[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG content: << "{"_scroll_id":"c2NhbjswOzE7dG90YWxfaGl0czoxOw==","took":15,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":1,"max_score":0.0,"hits":[{"_index":"test_zipkin_dependency-2017-08-08","_type":"span","_id":"AV3AVnw4E4f7903zZKvx","_score":0.0,"_source":{"timestamp_millis":1502150400050,"traceId":"000000000000000a","id":"000000000000000a","name":"","annotations":[{"timestamp":1502150400050000,"value":"cs","endpoint":{"serviceName":"web","ipv4":"124.13.90.3","port":80,"ipv6":"2001:db8::c001"}},{"timestamp":1502150400072000,"value":"error","endpoint":{"serviceName":"app","ipv4":"172.17.0.2","port":8080}},{"timestamp":1502150400100000,"value":"sr","endpoint":{"serviceName":"app","ipv4":"172.17.0.2","port":8080}}]}}]}}"
17/08/08 13:35:12 DEBUG header: >> "DELETE /_search/scroll HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 50[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG content: >> "{"scroll_id":["c2NhbjswOzE7dG90YWxfaGl0czoxOw=="]}"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 404 Not Found[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 404 Not Found[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 2[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 INFO ElasticsearchDependenciesJob: Saving dependency links to test_zipkin_dependency-2017-08-08/dependencylink
17/08/08 13:35:12 DEBUG header: >> "GET / HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 364[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG content: << "{[\n]"
17/08/08 13:35:12 DEBUG content: << "  "name" : "Amber Hunt",[\n]"
17/08/08 13:35:12 DEBUG content: << "  "cluster_name" : "elasticsearch",[\n]"
17/08/08 13:35:12 DEBUG content: << "  "cluster_uuid" : "yhWaWc2jRaWNz2u6W2TbLQ",[\n]"
17/08/08 13:35:12 DEBUG content: << "  "version" : {[\n]"
17/08/08 13:35:12 DEBUG content: << "    "number" : "2.4.5",[\n]"
17/08/08 13:35:12 DEBUG content: << "    "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276",[\n]"
17/08/08 13:35:12 DEBUG content: << "    "build_timestamp" : "2017-04-24T16:18:17Z",[\n]"
17/08/08 13:35:12 DEBUG content: << "    "build_snapshot" : false,[\n]"
17/08/08 13:35:12 DEBUG content: << "    "lucene_version" : "5.5.4"[\n]"
17/08/08 13:35:12 DEBUG content: << "  },[\n]"
17/08/08 13:35:12 DEBUG content: << "  "tagline" : "You Know, for Search"[\n]"
17/08/08 13:35:12 DEBUG content: << "}[\n]"
17/08/08 13:35:12 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "PUT /test_zipkin_dependency-2017-08-08/dependencylink/_bulk HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 103[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG content: >> "{"index":{"_id":"web|app"}}[\n]"
17/08/08 13:35:12 DEBUG content: >> "{"id":"web|app","parent":"web","child":"app","callCount":1,"errorCount":0}[\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 207[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG content: << "{"took":7,"errors":false,"items":[{"index":{"_index":"test_zipkin_dependency-2017-08-08","_type":"dependencylink","_id":"web|app","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"status":201}}]}"
17/08/08 13:35:12 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/_refresh HTTP/1.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Host: 192.168.99.100:32981[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "Content-Length: 0[\r][\n]"
17/08/08 13:35:12 DEBUG header: >> "[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "Content-Length: 50[\r][\n]"
17/08/08 13:35:12 DEBUG header: << "[\r][\n]"
17/08/08 13:35:12 DEBUG content: << "{"_shards":{"total":10,"successful":5,"failed":0}}"

Log from version 6.0.0.BUILD-SNAPSHOT, note absense of POST /_search/scroll?scroll=5m

17/08/08 13:32:39 INFO ElasticsearchDependenciesJob: Processing spans from test_zipkin_dependency-2017-08-08/span
17/08/08 13:32:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/08 13:32:40 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "GET /_cluster/health/test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:40 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "Content-Length: 389[\r][\n]"
17/08/08 13:32:40 DEBUG header: << "[\r][\n]"
17/08/08 13:32:40 DEBUG content: << "{"cluster_name":"elasticsearch","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}"
17/08/08 13:32:41 DEBUG header: >> "GET / HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 358[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG content: << "{[\n]"
17/08/08 13:32:41 DEBUG content: << "  "name" : "Rock",[\n]"
17/08/08 13:32:41 DEBUG content: << "  "cluster_name" : "elasticsearch",[\n]"
17/08/08 13:32:41 DEBUG content: << "  "cluster_uuid" : "7wqy9S9VS_OEg_x0onupow",[\n]"
17/08/08 13:32:41 DEBUG content: << "  "version" : {[\n]"
17/08/08 13:32:41 DEBUG content: << "    "number" : "2.4.5",[\n]"
17/08/08 13:32:41 DEBUG content: << "    "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276",[\n]"
17/08/08 13:32:41 DEBUG content: << "    "build_timestamp" : "2017-04-24T16:18:17Z",[\n]"
17/08/08 13:32:41 DEBUG content: << "    "build_snapshot" : false,[\n]"
17/08/08 13:32:41 DEBUG content: << "    "lucene_version" : "5.5.4"[\n]"
17/08/08 13:32:41 DEBUG content: << "  },[\n]"
17/08/08 13:32:41 DEBUG content: << "  "tagline" : "You Know, for Search"[\n]"
17/08/08 13:32:41 DEBUG content: << "}[\n]"
17/08/08 13:32:41 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08 HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "HEAD /test_zipkin_dependency-2017-08-08/span HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: text/plain; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 0[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "GET /test_zipkin_dependency-2017-08-08/_search_shards HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 1147[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG content: << "{"nodes":{"X9QcQ6N6Spm-4YkmOwVdpw":{"name":"Rock","transport_address":"172.17.0.6:9300","attributes":{}}},"shards":[[{"state":"STARTED","primary":true,"node":"X9QcQ6N6Spm-4YkmOwVdpw","relocating_node":null,"shard":0,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"U93DnwDITxKnyBEIhXvppA"}}],[{"state":"STARTED","primary":true,"node":"X9QcQ6N6Spm-4YkmOwVdpw","relocating_node":null,"shard":1,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"priFUo5zT5W7jDpyE1VcZg"}}],[{"state":"STARTED","primary":true,"node":"X9QcQ6N6Spm-4YkmOwVdpw","relocating_node":null,"shard":2,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"o2YQUDI-Raij9iqkw9RPVQ"}}],[{"state":"STARTED","primary":true,"node":"X9QcQ6N6Spm-4YkmOwVdpw","relocating_node":null,"shard":3,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"vnhG0BDvSyW181nYiWBajQ"}}],[{"state":"STARTED","primary":true,"node":"X9QcQ6N6Spm-4YkmOwVdpw","relocating_node":null,"shard":4,"index":"test_zipkin_dependency-2017-08-08","version":2,"allocation_id":{"id":"rYX8S2TFRcKmHWLvZ142gw"}}]]}"
17/08/08 13:32:41 DEBUG header: >> "GET /test_zipkin_dependency-2017-08-08/span/_mapping HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 925[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG content: << "{"test_zipkin_dependency-2017-08-08":{"mappings":{"span":{"_all":{"enabled":false},"properties":{"annotations":{"type":"nested","dynamic":"false","properties":{"endpoint":{"dynamic":"false","properties":{"serviceName":{"type":"string","index":"not_analyzed","ignore_above":256}}},"value":{"type":"string","index":"not_analyzed","ignore_above":256}}},"binaryAnnotations":{"type":"nested","dynamic":"false","properties":{"endpoint":{"dynamic":"false","properties":{"serviceName":{"type":"string","index":"not_analyzed","ignore_above":256}}},"key":{"type":"string","index":"not_analyzed","ignore_above":256},"value":{"type":"string","index":"not_analyzed","ignore_above":256}}},"duration":{"type":"long"},"id":{"type":"string"},"name":{"type":"string","index":"not_analyzed","ignore_above":256},"timestamp_millis":{"type":"date","format":"epoch_millis"},"traceId":{"type":"string","index":"not_analyzed","ignore_above":256}}}}}}"
17/08/08 13:32:41 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A0%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A1%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A3%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A2%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "POST /test_zipkin_dependency-2017-08-08/span/_search?search_type=scan&scroll=5m&size=50&preference=_shards%3A4%3B_local HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 26[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"query":{"match_all":{}}}"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzExOlg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjE7","took":6,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":1,"max_score":0.0,"hits":[]}}"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzEzOlg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjA7","took":8,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}"
17/08/08 13:32:41 DEBUG header: >> "DELETE /_search/scroll HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "DELETE /_search/scroll HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 82[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 82[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"scroll_id":["c2NhbjsxOzEzOlg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjA7"]}"
17/08/08 13:32:41 DEBUG content: >> "{"scroll_id":["c2NhbjsxOzExOlg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjE7"]}"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzEyOlg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjA7","took":9,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}"
17/08/08 13:32:41 DEBUG header: >> "DELETE /_search/scroll HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 82[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"scroll_id":["c2NhbjsxOzEyOlg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjA7"]}"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzE0Olg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjA7","took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 201[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG content: << "{"_scroll_id":"c2NhbjsxOzE1Olg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjA7","took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}}"
17/08/08 13:32:41 DEBUG header: >> "DELETE /_search/scroll HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 2[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 2[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 82[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"scroll_id":["c2NhbjsxOzE0Olg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjA7"]}"
17/08/08 13:32:41 DEBUG header: >> "DELETE /_search/scroll HTTP/1.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Type: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Accept: application/json[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Host: 192.168.99.100:32978[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "Content-Length: 82[\r][\n]"
17/08/08 13:32:41 DEBUG header: >> "[\r][\n]"
17/08/08 13:32:41 DEBUG content: >> "{"scroll_id":["c2NhbjsxOzE1Olg5UWNRNk42U3BtLTRZa21Pd1ZkcHc7MTt0b3RhbF9oaXRzOjA7"]}"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 2[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 2[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "HTTP/1.1 200 OK[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Type: application/json; charset=UTF-8[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "Content-Length: 2[\r][\n]"
17/08/08 13:32:41 DEBUG header: << "[\r][\n]"
17/08/08 13:32:41 INFO ElasticsearchDependenciesJob: No spans found at test_zipkin_dependency-2017-08-08/span

[jbaiera]

I think I see where the problem is here - The internal scroll management code assumes that if the first scroll request does not contain any records, then the rest of the scroll will not (regardless if the total number of hits is non-zero). This is surprising because we have tests that should recreate this behavior, but it doesn't reproduce with them. Can you share your reproduction steps here? I'll start working on a fix.

[codefromthecrypt]

Hi james. Thanks for looking into it. In hopes that this doesnt cause you too much work.. Clone https://github.com/openzipkin/zipkin-dependencies Set this to the snapshot version https://github.com/openzipkin/zipkin-dependencies/blob/master/elasticsearch/pom.xml Run this https://github.com/openzipkin/zipkin-dependencies/blob/master/elasticsearch/src/test/java/zipkin/storage/elasticsearch/http/ElasticsearchV2DependenciesTest.java Note this uses docker! If this is too much I can try to isolate a test given an example test case for things like this.

[jbaiera]

Isolating the test would be appreciated in this situation since it will need to be isolated for us to add it as a test case to ES-Hadoop

[codefromthecrypt]

This hack of AbstractJavaEsSparkTest.testEsRDDZRead seems to do the trick.

  @Test
  public void testEsRDDZReadJson_groupBy() throws Exception {
    String target = "spark-test-java-basic-json-read/data";

    RestUtils.touch("spark-test-java-basic-json-read");
    RestUtils.postData(target,
        "{\"message\" : \"Hello World\",\"message_date\" : \"2014-05-25\"}".getBytes());
    RestUtils.postData(target,
        "{\"message\" : \"Goodbye World\",\"message_date\" : \"2014-05-25\"}".getBytes());
    RestUtils.refresh("spark-test*");

    assertThat(JavaEsSpark.esJsonRDD(sc, target).groupBy(pair -> pair._2).count(), is(2L));
  }

[jbaiera]

@adriancole Thanks for the reproducing test! The fix should make its way into a snapshot build soon.

[codefromthecrypt]

verified that current master works. Really excited for the next beta, as when that's out folks can try Elasticsearch 6.x with Zipkin (right now, they can but the service graph functionality doesn't work due to this)