Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job fails when reader not able to read next record #175

Closed
nfleury opened this Issue Feb 25, 2016 · 6 comments

Comments

2 participants
@nfleury
Copy link

nfleury commented Feb 25, 2016

Hi !

First of all thanks for this nice and useful framework !

I would like to understand why does a Job fails/stops when the reader cannot read the next record with RecordReader::readNextRecord() in default JobImpl ?

Don't you think this is a strange behavior (or at least not very flexible) to stop the job only if a single record cannot be retrieved among many other valid ones ?

What do you think if current behavior is activated for strict mode only otherwise it continues to loop

you would have somthing like this:

  //read next record
                Record currentRecord;
                try {
                    currentRecord = readNextRecord();
                    recordCount++;
                } catch (RecordReadingException e) {

                    eventManager.fireOnRecordReadingException(e);                    

                    if (parameters.isStrictMode()){
                        return report;                        
                    }else{
                        continue;    
                    }


                }

Thanks !

@benas

This comment has been minimized.

Copy link
Contributor

benas commented Feb 25, 2016

Hi,

First of all thanks for this nice and useful framework !

Thank you, I'm glad you like it 😄

I would like to understand why does a Job fails/stops when the reader cannot read the next record with RecordReader::readNextRecord() in default JobImpl ? Don't you think this is a strange behavior (or at least not very flexible) to stop the job only if a single record cannot be retrieved among many other valid ones ?

I do agree with you, it's a bit weird. Stopping the entire job is a brutal solution. It is not flexible and probably a design issue. I have planned to introduce a retry policy in case the data source is temporarily down.

What do you think if current behavior is activated for strict mode only otherwise it continues to loop

The strictMode is designed for "functional" errors (business validation, mapping, etc). May be I should emphasize this detail in the documentation.

The snippet above will immediately continue to read the next record. It is sometimes helpful to wait a bit before retrying. I believe a retry mechanism (with a configurable backoff policy, timeout, number of retries) is more appropriate for this case. What do you think?

Kind regards
Mahmoud

@benas benas added the enhancement label Feb 25, 2016

@benas benas added this to the 4.1.0 milestone Feb 25, 2016

@benas benas self-assigned this Feb 25, 2016

benas added a commit that referenced this issue Jun 14, 2016

@benas

This comment has been minimized.

Copy link
Contributor

benas commented Jun 14, 2016

Hi,

It took some time but it is finally here. I've added the RetryPolicy API that makes the reader retry reading records whenever the data source is temporarily down. We can specify a backoff delay to wait between attemps.

You can find two examples here.

I've deployed version 4.1.0-SNAPSHOT with the working prototype. It would be great if you can give it a try! Many thanks upfront.

Looking forward to hearing your feedback.
Kind regards
Mahmoud

@nfleury

This comment has been minimized.

Copy link
Author

nfleury commented Jun 14, 2016

Hi,

Thanks ! Looks nice ! Will try that and give you some feeback !

benas added a commit that referenced this issue Jun 17, 2016

@benas

This comment has been minimized.

Copy link
Contributor

benas commented Jun 17, 2016

Hello,

Just FYI, the work on this feature has been merged in the master branch and will be released in the next version. Keep tuned for the upcoming release candidate 😉

Kind regards
Mahmoud

@benas

This comment has been minimized.

Copy link
Contributor

benas commented Jul 7, 2016

Will try that and give you some feeback !

@nfleury Have you got a chance to try out the new RetryPolicy API ?

@benas benas closed this Jul 18, 2016

@benas benas removed the in progress label Jul 18, 2016

@benas

This comment has been minimized.

Copy link
Contributor

benas commented Sep 15, 2016

Looks like retries in case of read failure are not supported in Spring batch:

http://forum.spring.io/forum/spring-projects/batch/75256-retry-in-case-of-a-failed-reading

It is left to the user and not part of the framework. Probably this is a good decision IMO.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.