Batch execution of tests when running in ExecutionMode.CONCURRENT

[Adriantega12]

Issue

Hi JUnit project! I was reading into how to implement parallel execution of tests and how to work around when these depend on a single resource. Partially, the solution to these is @ResourceLock annotation which helps specify which test case or test class depends on which resource which helps decide which tests can run with which ones. If there are many resources, @ResourceLock annotations can be stacked.

This, however does not always result in an ideal execution. There is a small issue with the current implementation of @ResourceLock, when a test depends on two resources that the rest of the tests depend on. Here is an example of that:

@Execution(ExecutionMode.CONCURRENT)
public class MyTests {
    @ResourceLock("a")
    public void Test1() {}
    
    @ResourceLock("b")
    public void Test2() {}
    
    @ResourceLock("a")
    @ResourceLock("b")
    public void Test3() {}
}

In my testing, from what I've seen, two kinds of execution orders can happen:

1. First:
   1. Test1 and Test2 run together.
   2. Test3 runs
2. Second: (any ordering of this where Test3 is in the middle)
   1. Test1 runs
   2. Test3 runs
   3. Test2 runs

Here, the point being that ideally it doesn't matter when Test3 runs but for runtime it matters a lot if Test1 and Test2 run together or not.

Note: Being fair, "Second" order rarely happens if only there are three tests like this, JUnit handles those pretty well, but my live example where I found this issue I have 6 tests that depend either on a or b, and 2 tests that depend on a and b. I can post that example if required, I just wanted to simplify the issue as much as possible.

Workarounds I've tried

What I've manage to do to work around this is the utilization of @nested class annotation. So, for example, reworking the initial code:

public class MyTests {

    @Nested
    @Execution(ExecutionMode.CONCURRENT)
    class ConcurrentTests() {
        @ResourceLock(value = "a")
        @ResourceLock(value = "dummy", mode = ResourceAccessMode.READ)
        public void Test1() {}

        @ResourceLock("b")
        @ResourceLock(value = "dummy", mode = ResourceAccessMode.READ)
        public void Test2() {}
    }
     
    @Nested
    @ResourceLock("a")
    @ResourceLock("b")
    @ResourceLock(value = "dummy", mode = ResourceAccessMode.READ_WRITE)
    class DependsOnBothTests() {
         public void Test3() {}
     }
}

This, however comes with it's own set of issues:

• Does not scale at all. (what happens if I need a 3rd "batch"?)
• "dummy" resource isn't really a resource
• In Test3, @ResourceLock for "a" and "b" becomes redundant but if removed then it isn't explicit to future maintainers.
• @order annotation for either class or method does not really work in here because that only stablishes priority, not a serial order of execution or batchable order of execution.

Deliverables

• Add a way to separate test "batches" with it's own set of properties and serialized.

public class MyTests {

    @Nested
    @Execution(ExecutionMode.CONCURRENT)
    @Batch
    class ConcurrentTests() {
        @ResourceLock(value = "a")
        @ResourceLock(value = "dummy", mode = ResourceAccessMode.READ)
        public void Test1() {}

        @ResourceLock("b")
        @ResourceLock(value = "dummy", mode = ResourceAccessMode.READ)
        public void Test2() {}
    }
     
    @Nested
    @Batch
    class DependsOnBothTests() {
         public void Test3() {}
     }
}

Desired behavior would be @Batch DependsOnBothTests and @Batch ConcurrentTests would always run independently and not at the same time.

Let me know if there are any further details required and or suggestions on how to work around this!

[marcphilipp]

Thanks for the extensive issue description! I wonder if this could be solved by ordering tests differently so tests that should run together get scheduled directly after one another. Are your affected test methods all in the same test class? If so, you could experiment with a custom method orderer.

[Adriantega12]

@marcphilipp Didn't quite know about that one! Will look into it for sure and report back. Thanks for the suggestion.

[github-actions]

If you would like us to be able to process this issue, please provide the requested information. If the information is not provided within the next 3 weeks, we will be unable to proceed and this issue will be closed.

[github-actions]

Closing due to lack of requested feedback. If you would like to proceed with your contribution, please provide the requested information and we will re-open this issue.