Invalid Docs Received counter in MapReduce jobs using EsInputFormat

[panthony]

What kind an issue is this?

Bug

Issue description

In MapReduce job using EsinputFormat, the Docs received counter value does not match Map input records.

Steps to reproduce

Initial request done here:

• https://github.com/elastic/elasticsearch-hadoop/blob/master/mr/src/main/java/org/elasticsearch/hadoop/rest/ScrollQuery.java#L93

Do not set read property nor increment stats.docsReceived like requests in while loop does:

• https://github.com/elastic/elasticsearch-hadoop/blob/master/mr/src/main/java/org/elasticsearch/hadoop/rest/ScrollQuery.java#L122

Version Info

JVM : openjdk version "1.8.0_242"
Hadoop/Spark: 2.9.3
ES-Hadoop : 6.8.7 (exists on master)
ES : 5.6

[masseyke]

Hi @panthony -- I would like to work on this one but am not sure how to reproduce it. Do you have steps to reproduce it?

[panthony]

@masseyke Hi, I simply had:

• An ElasticSearch with few documents inserted (enough for the scroll query to paginate)
• A MapReduce job reading from this index using EsInputFormat (do not care about the output)

At the end of the job, when you compare the counters for stats.docsReceived and Map input record, you have different values whereas they should match since all docs received by EsInputFormat are fed to the mapper.

[masseyke]

It looks like this was introduced by this commit in #738 -- dd1d603#diff-5fdedebe55454f3ee66be59277cca29f728d005b0ef2bf96f6c613eb49a088a1. If I'm not mistaken, the first pass through the scroll doesn't update the counter. I haven't tried reproducing it yet, but does that line up with what you're seeing @panthony? The value of stats.docsReceived is always less than Map input records?

[masseyke]

Oh and I see now that that's what your original description was saying -- sorry I missed that!