Support for multiple I/O files in a single jobRun for a particular scheduleDate. [BATCH-34]

[spring-projects-issues]

Dave Syer opened BATCH-34 and commented

Scenario: Read 100 rows from the database and write to file, Output file can have at most 50 lines(rows). So does spring batch support creating 2 files each containing 50 lines (filename.txt.001 & filename.txt.002).

This is supported, however, there isn't any kind of 'max record count' that would signal to the FileOutputSource that a new file should be created. The outputsource would simply need to be closed and reopened. However, a wrapper could easily be written that would do this for you.
Edit/Delete Message

---

Affects: 1.0-m2

Attachments:

• flatfile_os_refactored_071024.patch (47.23 kB)

[spring-projects-issues]

Peter Zozom commented

I have created new flat file output sources (see attached patch).

• SingleFlatFileOutputSource has same functionality as FlatFileOutputSource, but it is refactored to be extendable
• SplittableFlatFileOutputSource extends SingleFlatFileOutputSource and allows to specify 'max record count'. When record limit is reached new file is created. Output Source attaches part number to filename as extension so file names are filename.txt.001, filename.txt.002. Output source also supports rollback to last commit point located in different file than currently opened file (e.g. we are processing filename.txt.005 and last commit point was somwere in filename.txt.002)

[spring-projects-issues]

Lucas Ward commented

Peter, I'll take a look at the solution as soon as possible. At first glance it seems to me that writing a simple wrapper for the input source would be the simplest approach. The wrapper would delegate read calls to it's inputSource and call close when the limit is reached, and open again on a new resource. However, extending may be an easier to configure solution. In my mind this is somewhat related to BATCH-16. The approach taken in the xml input sources was to use a static utils class to help with restarting, etc. However, it seems to me that common file handling concerns for reading in from both xml and flat files could be handled using a common abstract class. This would allow a wrapper, such as one that could be created to fix this issue, to work for both xml, flat file, and anything else that extended the common abstract file input source interface.

[spring-projects-issues]

Lucas Ward commented

I understand your point about extending the output source so that you can span rollbacks and commits across files. However, I'm not sure if we should be doing that. Personally, I would prefer that any limit given simply means that we will close the file at the first commit point after the limit is reached. However, I would be interested to hear if anyone has a use case that requires an exact file size or record count.

[spring-projects-issues]

Lucas Ward commented

Moving to 2.0, as there seems to be no requests for the feature.

[spring-projects-issues]

Robert Kasanicky commented

Is this still a feature we care about implementing? It's being pushed forwards for almost a year now...

[spring-projects-issues]

Dave Syer commented

We scheduled it for 2.0.0.M3. If it isn't really difficult I think we should do it. But I would concentrate on more important things first (this could always go into 2.1).

[spring-projects-issues]

Robert Kasanicky commented

There is now basic implementation commited.

TODO:
-strategise naming of created output resources
-test with StaxEventItemWriter

[spring-projects-issues]

Robert Kasanicky commented

There is now MultiResourceItemWriter that works with both FlatFileItemWriter and StaxEventItemWriter and has customizable suffix creation. One can set an item count limit per resource and a new resource is created after this limit is exceeded, but only on chunk boundary.