Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter count become incorrect when a skip happen in the writer [BATCH-2663] #941

spring-projects-issues opened this issue Dec 13, 2017 · 2 comments


Copy link

Alexis NICOLAS opened BATCH-2663 and commented


So I have an application for which I recently added the possibility to have a skip in the writer and I've saw that the filter count value became completely wrong, it become even way bigger than the read count which doesn't make much sense, especially since I have processorNonTransactional set to true which means the processor is not called again in this case.

I did a test with a chunk size of 30, with 10 items filtered on the chunk and an exception in the initial call to the writer to force the scanning mode (chunk items processed one by one to find the one responsible for the exception and skip it), what I saw after the full chunk was a filter count of 210 which seems to be 10 + (30-10)*10 meaning that the filter count of 10 would have been added each time when processing again an item during the scanning.

Best regards,


Affects: 3.0.8


Referenced from: pull request #565, and commits c78701a

Copy link
Collaborator Author

Alexis NICOLAS commented

Attached as [^] is a very simple application to reproduce the issue. It produce a filter count of 210 when the read count is only 30.

Copy link
Collaborator Author

Mahmoud Ben Hassine commented

Hi Nyamiou,

Thank you for reporting this issue and for your code example! Indeed, the filter count is incorrect when scanning the chunk and is applied for each item.

I would like to make a small clarification though. When the item processor is idempotent (which is the case in your example), setting it to be transactional or non transactional does not influence the filter count (but it does influence other metrics like processSkipCount) since:

  • Filtered items are not part of the scanned chunk anyway.
  • Non filtered items before the scan will not be filtered when re-processed during the scan (Since the processor is idempotent).

In your example, if you use a transactional processor, the filter count will still be 210, which we acknowledge is incorrect.

We are working on this issue and we will let you know when a fix is published.


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

1 participant