Job fails when reader not able to read next record

[nfleury]

Hi !

First of all thanks for this nice and useful framework !

I would like to understand why does a Job fails/stops when the reader cannot read the next record with RecordReader::readNextRecord() in default JobImpl ?

Don't you think this is a strange behavior (or at least not very flexible) to stop the job only if a single record cannot be retrieved among many other valid ones ?

What do you think if current behavior is activated for strict mode only otherwise it continues to loop

you would have somthing like this:

  //read next record
                Record currentRecord;
                try {
                    currentRecord = readNextRecord();
                    recordCount++;
                } catch (RecordReadingException e) {

                    eventManager.fireOnRecordReadingException(e);                    

                    if (parameters.isStrictMode()){
                        return report;                        
                    }else{
                        continue;    
                    }


                }

Thanks !

[fmbenhassine]

Hi,

First of all thanks for this nice and useful framework !

Thank you, I'm glad you like it 😄

I would like to understand why does a Job fails/stops when the reader cannot read the next record with RecordReader::readNextRecord() in default JobImpl ? Don't you think this is a strange behavior (or at least not very flexible) to stop the job only if a single record cannot be retrieved among many other valid ones ?

I do agree with you, it's a bit weird. Stopping the entire job is a brutal solution. It is not flexible and probably a design issue. I have planned to introduce a retry policy in case the data source is temporarily down.

What do you think if current behavior is activated for strict mode only otherwise it continues to loop

The strictMode is designed for "functional" errors (business validation, mapping, etc). May be I should emphasize this detail in the documentation.

The snippet above will immediately continue to read the next record. It is sometimes helpful to wait a bit before retrying. I believe a retry mechanism (with a configurable backoff policy, timeout, number of retries) is more appropriate for this case. What do you think?

Kind regards
Mahmoud

[fmbenhassine]

Hi,

It took some time but it is finally here. I've added the RetryPolicy API that makes the reader retry reading records whenever the data source is temporarily down. We can specify a backoff delay to wait between attemps.

You can find two examples here.

I've deployed version 4.1.0-SNAPSHOT with the working prototype. It would be great if you can give it a try! Many thanks upfront.

Looking forward to hearing your feedback.
Kind regards
Mahmoud

[nfleury]

Hi,

Thanks ! Looks nice ! Will try that and give you some feeback !

[fmbenhassine]

Hello,

Just FYI, the work on this feature has been merged in the master branch and will be released in the next version. Keep tuned for the upcoming release candidate 😉

Kind regards
Mahmoud

[fmbenhassine]

Will try that and give you some feeback !

@nfleury Have you got a chance to try out the new RetryPolicy API ?