Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include limit of items read in JmsItemReader [BATCH-1752] #1839

Open
spring-issuemaster opened this issue May 16, 2011 · 3 comments
Open

Include limit of items read in JmsItemReader [BATCH-1752] #1839

spring-issuemaster opened this issue May 16, 2011 · 3 comments

Comments

@spring-issuemaster
Copy link
Collaborator

@spring-issuemaster spring-issuemaster commented May 16, 2011

Paulo Pimenta opened BATCH-1752 and commented

I developed a batch that needs to read objects from a JMS Queue and write them in a File.

The batch runs every 1 hour and should reads at most 500.000 records by execution and write them in the file.

I had to extend JmsItemReader, create limit and counter attributes and change the read method to read at most the limit attribute.

I would like to suggest to change JmsItemReader to have this feature.


Affects: 2.1.7

Attachments:

@spring-issuemaster
Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented May 16, 2011

Lucas Ward commented

Its slightly trickier than that. Adding some type of limit would be making the reader stateful. As it stands now, its stateless and as a result is one of the few readers that can easily be used in a multi-threaded step. So, adding potential stateful scenarios may not be wise. However, it might make sense to create a new reader extending the old called StatefulJmsItemReader, with a simple interface allowing for various implementations to decide when the reading can be considered 'done'. It would also likely need to store this in the ExecutionContext, because otherwise it wouldn't handle restart scenarios correctly, and once you've read and removed the items from the queue, you can't exactly go back. I'm curious what Dave thinks as well, since he has had more experience in the multi-threaded area with Spring Batch, and wrote the original implementation.

@spring-issuemaster
Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented May 16, 2011

Lucas Ward commented

I just took a look at your attached class. My concerns from above about the ExecutionContext are valid for you implementation as well. Because you're not storing the readCounter on itemStream#update(), the counter would start over again at 0 if the job failed and had to be restarted.

@spring-issuemaster
Copy link
Collaborator Author

@spring-issuemaster spring-issuemaster commented Jun 1, 2011

Dave Syer commented

I think we can add a counter. As long as it is only incemented (and is implemented in a thread safe wrapper) I think the net result will be that a restart will finish the job so that the total number of items is at most the max limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.