Support RetryOperations within SimpleJobRepository [BATCH-1329]

[spring-projects-issues]

Ian Brandt opened BATCH-1329 and commented

See the referenced forum post for motivation and details. If SimpleJobRepository had a get/setRetryOperations that was used to wrap all DAO calls it would have a very flexible and durable way to deal with transient failures such as CannotSerializeTransactionException (which is to be anticipated given the default transaction SERIALIZABLE isolation level set by AbstractJobRepositoryFactoryBean). Of course AbstractJobRepositoryFactoryBean would need equivalent getters and setters, and in the absence of given configuration would set a default. The default could be a RetryTemplate with the NeverRetryPolicy to preserve current behavior, or one could get fancy and start handling given TransientDataAccessException subclasses with an ExceptionClassifierRetryPolicy. The latter would require the determination of sensible defaults. I'm not sure if that would result in endless back and forth by users with different needs, or if a "retry 3 times with 1-2 second pause" would be a generally welcomed default?

Per Dave's post earlier in the referenced forum thread an alternative would be put it on the user to wrap the JobLauncher with RetryOperations. Given that transient failures can be anticipated for JobRepository in particular I think it would be better to handle them at a lower level out of the box (i.e. I'm in favor of a default that's more robust than NeverRetryPolicy). Perhaps there is a case to add RetryOperations to SimpleJobLauncher instead of SimpleJobRepository? I didn't notice a JobLauncherFactoryBean equivalent to AbstractJobRepositoryFactoryBean.

---

Affects: 2.0.0

Reference URL: http://forum.springsource.org/showpost.php?p=249913

[spring-projects-issues]

Dave Syer commented

SERIALIZABLE is only the default level for one isolation in the JobRepository, and that is for a reason. Isn't the exception telling you that you almost launched a job with the same parameters twice? On a retry won't it just throw JobExecutionAlreadyRunningException or JobInstanceAlreadyCompleteException?

[spring-projects-issues]

Ian Brandt commented

I don't believe so. This job is launched multiple times in a row from a single Quartz trigger based on iteration over a set of PKs queried from the DB. The PKs are included in the job parameters to guarantee a new job instance for each run. The code is pretty simple and well tested. It's worked reliably before and since, and hasn't been changed. I have no cause to suspect we launched the same job instance twice in succession.

As I understand it INITRANS is a data block level header. A quick stats check shows we're running around 28 rows per block for BATCH_JOB_INSTANCE. So taking the documentation at face value it would seem reasonable to get this exception even for transactions that aren't affecting the same rows. I'm not an Oracle expert so I could stand to be corrected, but this is what I believe happened in our case.

[spring-projects-issues]

Dave Syer commented

I see. That sounds about right. However, since the retry operations can be added declaratively by users who want that feature I don't think we need it hard-coded in the JobRepository or JobLauncher. I would probably add the interceptor to the JobLauncher if I needed this feature. I can supply code samples if anyone is interested.

[spring-projects-issues]

Giovanni Dall'Oglio Risso commented

we solved this problem as told by Dave a moment ago: using the RetryTemplate with AOP, around the JobOperator

now we can see the log telling "error occurred... retry... now is ok!"

Here the fragment:

        <bean id="retryAdvice"
            class="org.springframework.batch.retry.interceptor.RetryOperationsInterceptor">
            <property name="retryOperations">
                  <bean id="retryTemplate" class="org.springframework.batch.retry.support.RetryTemplate">
                        <property name="retryPolicy">
                             <bean id="retryPolicy"
                                   class="org.springframework.batch.retry.policy.SimpleRetryPolicy">
                                   <property name="maxAttempts" value="3" />
                                   <property name="retryableExceptions">
                                         <map>
                                               <entry
                                                     key="org.springframework.dao.CannotSerializeTransactionException"
                                                     value="true" />
                                         </map>
                                   </property>
                             </bean>
                        </property>
                  </bean>
            </property>
      </bean>

      <aop:config>
            <aop:pointcut id="retryPointcut"
                  expression="execution(* org.springframework.batch.core.launch.JobOperator.start*(..))" />
            <aop:advisor advice-ref="retryAdvice" pointcut-ref="retryPointcut" />
      </aop:config>

[spring-projects-issues]

Mahmoud Ben Hassine commented

I agree with Dave Syer on this, I would let users who need this feature to add it declaratively around the JobLauncher/JobOperator instead of adding it in those classes.

The AOP sample by Giovanni Dall'Oglio Risso LGTM (thanks for sharing!). Another way to implement the feature is suggested in PR 440 in which the idea is to decorate the JobLauncher in a FaultTolerantJobLauncher configurable with a RetryTemplate.

@Ian Brandt Does this make sense? Can we close this ticket?

[spring-projects-issues]

Mahmoud Ben Hassine commented

In addition to the AOP sample above, here is a quick example with a retry template around the JobLauncher:

import java.util.Map;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.JobParametersBuilder;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.retry.policy.SimpleRetryPolicy;
import org.springframework.retry.support.RetryTemplate;

@Configuration
@EnableBatchProcessing
public class MyJob {

	private final JobBuilderFactory jobs;

	private final StepBuilderFactory steps;

	@Autowired
	public MyJob(JobBuilderFactory jobs, StepBuilderFactory steps) {
		this.jobs = jobs;
		this.steps = steps;
	}

	@Bean
	public Step step() {
		return steps.get("step")
				.tasklet((contribution, chunkContext) -> {
					Map<String, Object> jobParameters = chunkContext.getStepContext().getJobParameters();
					String name = (String) jobParameters.get("name");
					System.out.println("Hello " + name);
					throw new Exception("Boom!");
				})
				.build();
	}

	@Bean
	public Job job() {
		return jobs.get("job")
				.start(step())
				.build();
	}

	public static void main(String[] args) throws Throwable {
		RetryTemplate retryTemplate = new RetryTemplate();
		retryTemplate.setRetryPolicy(new SimpleRetryPolicy(3));

		ApplicationContext applicationContext = new AnnotationConfigApplicationContext(MyJob.class);
		JobLauncher jobLauncher = applicationContext.getBean(JobLauncher.class);
		Job job = applicationContext.getBean(Job.class);
		JobParameters jobParameters = new JobParametersBuilder().addString("name", "world").toJobParameters();

		retryTemplate.execute(retryContext -> {
			JobExecution jobExecution = jobLauncher.run(job, jobParameters);
			if (!jobExecution.getAllFailureExceptions().isEmpty()){
				System.out.println("Job failed, retrying..");
				throw jobExecution.getAllFailureExceptions().iterator().next();
			}
			return jobExecution;
		});
	}

}

This prints:

Hello world
Job failed, retrying..
Hello world
Job failed, retrying..
Hello world
Job failed, retrying..

Hope this helps.

I'm closing this ticket for now.

[javaHelper]

@benas - Thanks for sharing the sample example. We've a multiple chunk based steps in a single Job. Assume if 2nd step got failed, will retryTemplate.execute ensure that it will restart at the point where it has failed ? Could you please confirm if passing the same job parameters will be sufficient?

[mminella]

It's not the RetryTemplate that enables that support. It's the statefulness of the components (those that implement ItemStream. These questions are better asked on StackOverflow and tagged with spring-batch.

[javaHelper]

@mminella - Ok thanks, will you please guide here: https://stackoverflow.com/questions/63705853/how-can-we-resume-spring-batch-job-where-it-had-failed ?