Streaming API based ItemWriter for JSON, with restart support [BATCH-1850]

[spring-projects-issues]

Vasanth Raj opened BATCH-1850 and commented

Since JSON has become a widely adapted format among Integration technologies, a Streaming API based ItemWriter with restart support for JSON has to be developed. On the similar lines to StaxEventItemWriter.

Usecase:

1. Online Application initiating the batch job
2. ItemReader reads the list of IDs to be process from underlying DB or Service
3. ItemProcessor does further processing using DB and/or Service
4. ItemWriter writes JSON file out of the processed records
5. Callback mechanism would drop the file into underlying rabbitMQ
6. Consumers would parse the JSON File for downstream applications.

Writing a simple Itemwriter wouldnt suffice. As in most batch process the datasets could be enormous,so the job should support restarts.

If the above has to be achieved with XML as the output format, it can be easily achieved using StaxEventItemWriter.

A similar implementation for JSON would be highly valuable.

---

Affects: 2.1.8

Reference URL: http://forum.springsource.org/showthread.php?125495-StaxEventItemWriter-equivalent-for-JSON-for-restartable-jobs

Issue Links:

• BATCH-2692 Streaming JSON Writer
  ("is duplicated by")

2 votes, 3 watchers

[spring-projects-issues]

Dave Syer commented

I didn't get the RabbitMQ piece (5), but can't you do everything with a FlatFileItemWriter plus a custom LineAggregator (and maybe header/footer callbacks to complete the JSON object)? We could definietly add one of those (to complement the existing JsonLineMapper).

[spring-projects-issues]

Vasanth Raj commented

RabbitMQ part isnt related to the issue, but mentioned it for use case to look complete, end to end. the output file needs to be placed in a Queue, probably a JobListener should do the job.

Using the technique you suggested above should work for simple scenarios, but letting a parser do the work would make the solution more accommodating for future changes.

eg: If there will be 2 item processors writing 2 different fragments to the same file ??

[spring-projects-issues]

Vasanth Raj commented

Apologies, I meant the approach wont accommodate changes such as multiple steps working on the same file.

[spring-projects-issues]

Mahmoud Ben Hassine commented

This feature was implemented in BATCH-2692. I'm closing this ticket.