Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming API based ItemWriter for JSON, with restart support [BATCH-1850] #1739

Closed
spring-issuemaster opened this issue Apr 17, 2012 · 4 comments

Comments

@spring-issuemaster
Copy link
Collaborator

@spring-issuemaster spring-issuemaster commented Apr 17, 2012

Vasanth Raj opened BATCH-1850 and commented

Since JSON has become a widely adapted format among Integration technologies, a Streaming API based ItemWriter with restart support for JSON has to be developed. On the similar lines to StaxEventItemWriter.

Usecase:

  1. Online Application initiating the batch job
  2. ItemReader reads the list of IDs to be process from underlying DB or Service
  3. ItemProcessor does further processing using DB and/or Service
  4. ItemWriter writes JSON file out of the processed records
  5. Callback mechanism would drop the file into underlying rabbitMQ
  6. Consumers would parse the JSON File for downstream applications.

Writing a simple Itemwriter wouldnt suffice. As in most batch process the datasets could be enormous,so the job should support restarts.

If the above has to be achieved with XML as the output format, it can be easily achieved using StaxEventItemWriter.

A similar implementation for JSON would be highly valuable.


Affects: 2.1.8

Reference URL: http://forum.springsource.org/showthread.php?125495-StaxEventItemWriter-equivalent-for-JSON-for-restartable-jobs

Issue Links:

  • BATCH-2692 Streaming JSON Writer
    ("is duplicated by")

2 votes, 3 watchers

@spring-issuemaster
Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Apr 17, 2012

Dave Syer commented

I didn't get the RabbitMQ piece (5), but can't you do everything with a FlatFileItemWriter plus a custom LineAggregator (and maybe header/footer callbacks to complete the JSON object)? We could definietly add one of those (to complement the existing JsonLineMapper).

@spring-issuemaster
Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Apr 17, 2012

Vasanth Raj commented

RabbitMQ part isnt related to the issue, but mentioned it for use case to look complete, end to end. the output file needs to be placed in a Queue, probably a JobListener should do the job.

Using the technique you suggested above should work for simple scenarios, but letting a parser do the work would make the solution more accommodating for future changes.

eg: If there will be 2 item processors writing 2 different fragments to the same file ??

@spring-issuemaster
Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Apr 17, 2012

Vasanth Raj commented

Apologies, I meant the approach wont accommodate changes such as multiple steps working on the same file.

@spring-issuemaster
Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Oct 5, 2018

Mahmoud Ben Hassine commented

This feature was implemented in BATCH-2692. I'm closing this ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.