Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FlatFileItemRead isnt respecting the maxitemcounter [BATCH-2801] #805

Open
spring-issuemaster opened this issue Mar 25, 2019 · 0 comments
Open

Comments

@spring-issuemaster
Copy link
Collaborator

@spring-issuemaster spring-issuemaster commented Mar 25, 2019

Jesús Ponte opened BATCH-2801 and commented

I'm doing a process with Spring Batch multi-thread. My process receives a large txt file (> 100k lines). I want each thread to process X lines of the file and do X process with the information to gain time.

I am using the FlatFileItemRead class passing the initial and final line to each thread. I'm doing tests with 19k lines, 3 and 4 threads and I cannot see why the first thread if it starts and stops in the correct line, but the rest of threads start in the correct line but not end in the correct one, read the file until the end.

The xml configuration and the rangepartitioner files are attached.

My example with 19k lines in file and 3 threads:

Thread: 1
Initial Index: 1
Final Index: 6333

Thread: 2
Index Initial: 6334
Final Index: 12666

Thread: 3
Index Initial: 12667
Final Index: 19000

Thread 1 reads up to 6333. Thread 2 reads up to 19000 instead of reading up to 12666 and thread 3 reads up to 19000.

You can see an example created by Mahmoud Ben Hassine on his GitHub see this

His example (15 lines in file and 3 threads):

  • SimpleAsyncTaskExecutor-1 reads from 1 to 5 as expected
  • SimpleAsyncTaskExecutor-2 reads from 6 to 15 which is incorrect (it should read from 5 to 10 only)
  • SimpleAsyncTaskExecutor-3 reads from 11 to 15 as expected

 

 


Affects: 2.1.9, 4.1.1

Reference URL: https://stackoverflow.com/questions/55227133/why-do-these-threads-read-the-entire-file-instead-of-maxitemcounter

Attachments:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.