JobOperatorTestUtils only allows one Job per Spring context

[prismspike]

Bug description
After upgrading to Spring Boot 4 and Spring Batch 6 multiple tests are failing in our project, using JobOperatorTestUtils. After wiring multiple Jobs and using JobOperatorTestUtils to start them, only the first job is successfully executed, while every job after that is failing with a JobInstanceAlreadyCompleteException. The logged JobParameters are always the params of the first job run, even when attaching another Job to JobOperatorTestUtils.

Message: A job instance already exists and is complete for identifying parameters={JobParameter{name='batch.random', value=4546055881725385948, type=class java.lang.Long, identifying=true}}

Environment

• Spring Boot 4.0.0
• spring-boot-starter-batch 6.0.0
• spring-batch-test 6.0.0
• MariaDB 11.4

Steps to reproduce

• Add two Job Beans via JobBuilder with different names and qualifiers
• Wire both jobs in your Spring JUnit Test
• Wire JobOperatorTestUtils
• Add two tests and set each job on the utils, once for each test
• Start the job

Expected behavior
The setting of the job on JobOperatorTestUtils should override the previous config and be able to run again

Minimal Complete Reproducible example

@SpringBatchTest
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class BatchTest {
    @Autowired
    lateinit var jobOperatorTestUtils: JobOperatorTestUtils

    @Autowired
    @Qualifier("jobA")
    private lateinit var jobA: Job

    @Autowired
    @Qualifier("jobB")
    private lateinit var jobB: Job

    @Test
    @Throws(Exception::class)
    fun testJob() {
        jobOperatorTestUtils.setJob(jobA)
        val params = JobParametersBuilder()
            .addLocalDate("runDate", LocalDate.now())
            .addString("CID", UUID.randomUUID().toString())
            .toJobParameters()

        jobOperatorTestUtils.startJob(params)
    }

    @Test
    @Throws(Exception::class)
    fun testJob2() {
        jobOperatorTestUtils.setJob(jobB)
        val params = JobParametersBuilder()
            .addLocalDate("runDate", LocalDate.now())
            .addString("CID", UUID.randomUUID().toString())
            .toJobParameters()

        jobOperatorTestUtils.startJob(params)
    }
}

[benelog]

@prismspike

Are you running your tests with JobOperatorTestUtils backed by a ResourcelessJobRepository?
If so, the behavior is expected, as ResourcelessJobRepository does not take JobParameters into account. Its getJobInstance(...) method returns the previously executed JobInstance:

@Override
@Nullable
public JobInstance getJobInstance(String jobName, JobParameters jobParameters) {
    return this.jobInstance;
}

Try switching to the JDBC-backed SimpleJobRepository instead.
The spring-boot-starter-batch-jdbc module (available since Spring Boot 4.0) should make this easier.

[monnetchr]

Strange, using ResourcelessJobRepository in Batch 5 / Boot 3 the behavior was different. In fact ResourcelessJobRepository did not even implement getJobInstance, and the default throws UOE, I guess it was never called but now it is?
We want to continue using ResourcelessJobRepository, what are the options when running several batch runs in a single test?

[prismspike]

@benelog

Thank you for the information. I didn't know that spring-boot-starter-batch-jdbc is a new dependency that is needed for this. As @monnetchr said, our current implementation worked before with Batch 5 and Spring Boot 3 without any issues and we just followed the deprecation info to update all our classes. This side effect was unfortunately not very apparent.

I'm a bit confused about the need for ResourcelessJobRepository for testing purposes, since it only seems to be available once per job for a Spring application context. Our first idea, was to use a DirtiesContext annotation, but that dramatically increases test runtime.

[benelog]

@monnetchr
In Spring Batch v6, the JobRepository.getJobInstance(String, JobParameters) method is invoked within TaskExecutorJobLauncher.run(Job, JobParameters).

This behavior change from Spring Batch v5 was introduced by the following commit:

90d8959

[benelog]

@prismspike
Unfortunately, unless the implementation of ResourcelessJobRepository is significantly revised, it does not seem suitable for tests where JobOperatorTestUtils.startJob(...) is invoked multiple times.

I resolved this issue by avoiding the use of ResourcelessJobRepository in my tests. Instead, I replaced it with a JobRepository configuration backed by an H2 database.

[fmbenhassine]

@prismspike

In Spring Batch 6 / Spring Boot 4, the default batch infrastructure has been changed to use the resourceless infrastructure. This is mainly to avoid having to use an in-memory database even without the need for batch meta-data (that was a very strong constraint on our users).

This change is documented in:

• Spring Batch 6 migration guide: https://github.com/spring-projects/spring-batch/wiki/Spring-Batch-6.0-Migration-Guide#changes-related-to-the-default-batch-configuration
• Spring Boot 4 migration guide: https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.0-Migration-Guide#spring-batch

The ResourcelessJobRepository is designed for a very specific use case: a single execution of a Spring Batch job. This is mentioned in its Javadoc as well as the reference docs. It does exactly that and nothing else (and I believe it does very well as it performs two orders of magnitude better than any other repository implementation). So using it for anything else than what it was designed for is incorrect.

Here are some recommendations:

• The ResourcelessJobRepository is very lightweight and can be defined as a prototype bean in your test context. Every job can use a different instance of it and this is completely fine. I think of it like a virtual thread but for jobs: you can have as many disposable resourcessless job repositories as needed, no need to reuse them or pool them, etc. They will be GCed anyway.
• Use an JDBC-based batch infrastructure with an in-memory database or a docker-based setup if your tests involve job restarts, sharing execution context between steps, etc.
• Last but not least, and this is a more general advice, I highly recommend having a single job per test context. I really don't understand the reason when I see some users loading multiple job definitions in the same test context and complaining about confusing test setups. Test utilities provided by Spring Batch expect a single job per context to (try to) enforce best practices (see my comment here Multiple Job unit testing with @SpringBatchTest #3699 (comment)). If that is not possible, then you can always set the job to test using JobOperatorTestUtils#setJob prior to each test method.

I hope this clarifies things a bit more, but if you need further support or you think we can improve the docs, please let me know.

I am going to close this issue as the change is intended and I believe is documented both on the Batch and Boot sides. I understand that migrations to major releases can be daunting, but I am committed to do my best to help.