Filter count become incorrect when a skip happen in the writer [BATCH-2663]

[spring-projects-issues]

Alexis NICOLAS opened BATCH-2663 and commented

Hello,

So I have an application for which I recently added the possibility to have a skip in the writer and I've saw that the filter count value became completely wrong, it become even way bigger than the read count which doesn't make much sense, especially since I have processorNonTransactional set to true which means the processor is not called again in this case.

I did a test with a chunk size of 30, with 10 items filtered on the chunk and an exception in the initial call to the writer to force the scanning mode (chunk items processed one by one to find the one responsible for the exception and skip it), what I saw after the full chunk was a filter count of 210 which seems to be 10 + (30-10)*10 meaning that the filter count of 10 would have been added each time when processing again an item during the scanning.

Best regards,

Alexis NICOLAS

---

Affects: 3.0.8

Attachments:

• batch-2663.zip (50.15 kB)

Referenced from: pull request #565, and commits c78701a

[spring-projects-issues]

Alexis NICOLAS commented

Attached as [^batch-2663.zip] is a very simple application to reproduce the issue. It produce a filter count of 210 when the read count is only 30.

[spring-projects-issues]

Mahmoud Ben Hassine commented

Hi Nyamiou,

Thank you for reporting this issue and for your code example! Indeed, the filter count is incorrect when scanning the chunk and is applied for each item.

I would like to make a small clarification though. When the item processor is idempotent (which is the case in your example), setting it to be transactional or non transactional does not influence the filter count (but it does influence other metrics like processSkipCount) since:

• Filtered items are not part of the scanned chunk anyway.
• Non filtered items before the scan will not be filtered when re-processed during the scan (Since the processor is idempotent).

In your example, if you use a transactional processor, the filter count will still be 210, which we acknowledge is incorrect.

We are working on this issue and we will let you know when a fix is published.

Kr
Mahmoud