FlatFileItemReader with TaskExecutor [BATCH-2578]

[spring-projects-issues]

Kyriakos Mandalas opened BATCH-2578 and commented

There are a lot of examples which use FlatFileItemReader along with TaskExecutor. I provide samples below (both with XML and Java Config):

• http://www.oracle.com/technetwork/articles/cloudcomp/coherence-springbatch-1400985.html
• https://examples.javacodegeeks.com/enterprise-java/spring/batch/spring-batch-multithreading-example/

I have used it my self as well with XML configuration for large CSVs (GB size) writing to database with the out-of-the-box JpaItemWriter. There seem to be no issues whatsoever (i.e. errors) even without setting save-state = false or taking any kind of special handling.

Now, FlatFileItemReader is documented as not thread-safe. My guess was that JpaItemWriter was "covering" the issue by persisting Sets i.e. collections with no duplicates if the hashCode() and equals() were covering the business key of the Entity. However, even this way it is not enough to prevent duplicates due to non-thread safe reading and processing.

Could you please clarify: is it proper/correct/safe to use the out-of-the-box FlatFileItemReader within a Tasklet which has assigned a TaskExecutor? Regardless of the Writer. If not, how could we explain in theory the lack of errors when a JPAItemWriter is used?

---

2 votes, 4 watchers

[spring-projects-issues]

Michael Minella commented

It is not propper/correct/safe to use the out-of-the-box FlatFileItemReader with a TaskExecutor with saveState set to true. The reason you aren't seeing errors is that no error will occur on a single run. The issue arises restartability. As multiple threads read from the file, the state of the reader (how many lines have been read, etc) is being updated but may not match what is successfully processed. Because of that (the shared state), the ItemReader is not considered thread safe and should only be used in multithreaded use cases when saveState is set to false.