Switch Mockito test suite over to JUnit5

[TimvdLippe]

With the MockitoExtension in development for JUnit 5 (#1221), we can then switch over Mockito's internal test suite to JUnit5. We will need to figure out how this all works, as we also have the runner and junit5 is currently proposed as subproject (because of Java 6 vs 8 compatibility issues). This is mainly a tracking issue to make the switch someday.

[bohsen]

Repost!
@hennr Just curious. What are the advantages of using junit5 over junit4 (with respect to an updated junit dependency like junit 4.13.2)?

[hennr]

It's simply needed at some point in time as JUnit 4 will be deprecated more and more.
Spring-Boot 2.4 e.g. comes with an disabled JUnit 4 runner.

To the advantages of JUnit 5: Support for parameterized tests and nested tests are the highlights for me, but you should check the docs and judge for yourself.

[bohsen]

@hennr Thanks for the insight.
As for support in Spring Boot 2.4 that's really good to know. One of my next projects will probably be using Spring Boot. Time to get to know the Junit5 API then.

I like the @DisplayName feature. As for parameterized and nested tests Junit4 still works though:

@RunWith(Enclosed::class)
class AccessionNumberGeneratorTest {

    /**
     * Parameterized tests for [AccessionNumberGenerator]
     */
    @RunWith(Parameterized::class)
    internal class ParameterizedTests(private val accessionnumber: String, private val isValid: Boolean) {

        companion object {
            @JvmStatic
            @Parameters
            fun data() = listOf(
                arrayOf("TEST.0A246375", true),
                arrayOf("TEST.0G792765", true),
                arrayOf("TEST.0G255659", true),
                arrayOf("TEST.0G007934", true),
                arrayOf("TEST.0G590154", true),
                arrayOf("TEST.0G030354", true),
                arrayOf("TEST.0G055715", true),
                arrayOf("TEST.0G560262", true),
                arrayOf("TEST.0G643459", true),
                arrayOf("TEST.0G416015", true),
                arrayOf("TEST.0G205348", true),
                arrayOf("TEST.0A419210", true),
                arrayOf("TEST.0G291484", true),
                arrayOf("TEST.0D191435", true),
                arrayOf("TEST.0G898640", true),
                arrayOf("TEST.0D845404", true),
                arrayOf("TEST.0G695237", true),
                arrayOf("TEST.0D523122", true),
                arrayOf("TEST.0G019153", true),
                arrayOf("TEST.0E827546", true),
                arrayOf("TEST.0G454679", true),
                arrayOf("TEST.0C748137", true),
                arrayOf("TEST.0G828180", true),
                arrayOf("TEST.0B183991", true),
                arrayOf("TEST.0E929085", true),
                arrayOf("TEST.0Z721175", true),
                arrayOf("sTEST.0G246375", false),
                arrayOf("TEsST.0G792765", false),
                arrayOf("TEST.s0G255659", false),
                arrayOf("TEST.04007934", false),
                arrayOf("TES.0G590154", false),
                arrayOf("TEST.0G0303AB", false),
                arrayOf("TEST.G055715", false),
                arrayOf("TEST.560262", false),
                arrayOf("TEST,0G019153", false),
                arrayOf("TEST:0E827546", false),
                arrayOf("TEST0G643459", false)
            )
        }
    }

    internal class RegularTests {

        @Test
        fun whenGeneratingAccessionnumber_shouldReturnCorrectPrefix() {
            val actualPrefixSubstring = AccessionNumberGenerator.generate().substring(0, 9)
            assertThat(
                "Generated prefix does not match expected prefix.",
                AccessionNumberGenerator.accnumPrefix, `is`(actualPrefixSubstring)
            )
        }

        @Test
        fun whenGeneratingAccessionnumber_shouldReturnCorrectLength() {
            val expectedLength = 15
            val actualLength = AccessionNumberGenerator.generate().length
            assertThat(
                "Generated length does not match expected lenght.",
                actualLength, `is`(expectedLength)
            )
        }

        @Test
        fun whenGeneratingAccessionnumber_shouldMatchCorrectPattern() {
            val accessionnumber = AccessionNumberGenerator.generate()
            assertThat(
                "Generated accessionnumber does not match expected pattern.",
                accessionnumber, RegexMatcher("^TEST\\.\\d[A-Z]\\d{6}$")
            )
        }
    }
}

internal class RegexMatcher(private val regex: String) : TypeSafeMatcher<String>() {

    override fun describeTo(description: Description) {
        description.appendText("matches regex = `$regex`")
    }

    override fun matchesSafely(item: String): Boolean = item.matches(Regex(regex))
}

[TimvdLippe]

There are several considerations I have as both an open-source maintainer and user of JUnit:

1. The Mockito userbase still largely uses JUnit4 (we will need definitive data to confirm/disprove this statement)
2. The Mockito test suite uses JUnit4. The largest benefit of this approach is that Mockito uses the same version of JUnit as the majority of our users. This helps in catching potential issues that match our users setup. Ever since we shipped our JUnit5 extension, users have reported subtle issues that we are not catching ourselves, which a larger Mockito test suite built on top of JUnit5 can potentially catch
3. If we update the Mockito test suite to use JUnit5, then the big question is: what do we with supporting JUnit4? Supporting multiple versions of JUnit is time-consuming (we already see the effects of mainly supporting JUnit4 and a bit of JUnit5) and I am personally sceptical that we can commit to spending more time solely supporting on JUnit
4. If Mockito updates to JUnit5 and we assume that Mockito can only support 1 version of JUnit, this can cause significant friction for users who want to stay on JUnit4. They would lose out on support and might consider alternatives to Mockito. I think fragmentation of our userbase is something we would like to avoid whenever possible
5. (This is a personal opinion and I don't speak for the other maintainers) Currently, JUnit5 does not provide enough value compared to JUnit4. The friction of an upgrade does not outweigh the benefits of using a later version, based on my current analysis of the features that JUnit5 provides

Based on the above, I am sadly not positive for an upgrade to JUnit5 at this moment in time. I guess the biggest challenge for the Java ecosystem as a whole is figuring out how we migrate to JUnit5, without causing significant disruption. While JUnit5 has a runner to support JUnit4, not all of users are able to use this runner. A consequence of this is the requirement that the adoption of JUnit5 APIs and package upgrade must be atomic, which is only suitable option in smaller projects. A big bang upgrade will not be suitable for larger projects, which warrants a gradual migration path. Since Mockito is heavily used in enterprise, there are a significant amount of users with large projects, which in turn warrant a gradual migration path for Mockito itself. Therefore, an upgrade to JUnit5 in Mockito is more complicated than it initially looks.

For a more positive outlook, some scenarios that would make me reconsider upgrading to JUnit5:

1. The wider Java ecosystem has largely migrated to JUnit5 on its own
2. JUnit5 provides significant value on its own that warrants Mockito taking the lead and upgrading, hopefully causing our userbase to follow our steps. This is very risky and can cause fragmentation of the Mockito userbase, in a similar fashion to what happened with Mockito 2. I would like to avoid a repeat of the amount of fragmentation that Mockito 2 introduced (despite the concrete benefits that Mockito 2 provided over Mockito 1, it caused significant disruption for our users)
3. There is a gradual migration path available that makes support of both versions very cheap. We currently already have a JUnit5 extension, but there are subtle issues and we are missing them ourselves because we use JUnit4 in our test suite. At the moment, I am sceptical this scenario will become a reality, given our current experiences with having to maintain multiple implementations

Given the fragmentation risk and my expectation that scenario 3 is less likely to occur, I am personally in the camp of waiting for scenario 1. E.g. wait for a significant portion of our userbase to migrate to JUnit5 before we make the full switch ourselves and we are comfortable dropping JUnit4 support. Until then, we are in the current state where we mainly support JUnit4, but we have an extension for JUnit5 users and we will do our best to support it to the best of our abilities.

[hennr]

@bohsen This issue, for me, is not really about features, both JUnit 4 and 5 work well and (almost) everything is doable in both versions. This is more about progress and moving forward as JUnit 5 has been around for quite some time now and is more and more adopted. No hard feelings here though :)

@TimvdLippe Thanks summing up your points.
I can't follow one point yet, though. Why do you think that it wouldn't be possbile to use the vintage test runner in some projects?

In general I guess the opinion of a JUnit dev would be interesting here. Any thoughts on migrating mockito's test suite to JUnit5 @stefanbirkner?

[TimvdLippe]

@hennr My reading of https://junit.org/junit5/docs/current/user-guide/#migrating-from-junit4-running leads me to believe that test engine discovery is based on the classpath. While classpath discovery works, not every project can make use of that as I understand it. Users that don't use classpath discovery in JUnit4 (and rather use the programmatic JUnit4 API) would not work with junit-vintage-engine.

Please be aware that I might be missing crucial information here that would make it possible, so please let me know if the above reading is incorrect and how that would work in practice.

[stefanbirkner]

TLDR; You can switch your internal tests to JUnit Jupiter but you don't have to. People can use Mockito with JUnit 4 and/or JUnit Jupiter independent from the test framework that you use for your internal tests.

Running Mockito's test with JUnit Jupiter and supporting JUnit Jupiter are two independent topics.

Support for JUnit 4 and JUnit Jupiter

IIRC the only integration with the testing framework is setting up annotated fields (@mock, @Injectmocks, ...). Mockito supports integration with JUnit 4 by the MockitoRule and the MockitoRunner and JUnit Jupiter with the MockitoExtension. You can still support both when you switch the internal tests to JUnit Jupiter (like you already do with your current test setup). It should not make a lot of difference. From my experience you may have JUnit 4 and JUnit Jupiter users. It is very unlikely that almost all codebases move from JUnit 4 to JUnit Jupiter. Most of the developers usually don't care much about the testing framework. They use what they find or know.

Run the internal tests with JUnit Jupiter

JUnit 4 is still an option for writing tests. Most likely we will not add new features but if the current feature set fulfills your needs that should not be a problem. The library is very stable. You're also not forced to migrate to JUnit Jupiter because JUnit 4 is still supported by the major build tools (there are still people who have JUnit 3 tests and they are also still supported by the build tools).
There are new features which make the switch appealing: nested tests, no public modifier, display name, better support for parameterized tests. Switching to JUnit Jupiter is usually a straight forward task and it does not affect your users. You only switch the test framework that you're using for testing Mockito.

[hennr]

Thanks for summing things up, @stefanbirkner !

[Devamparikh]

Hey there, I have read some issues and discussions, you guys are planning to migrate Junit4 to Junit5, Can outsiders work on this issue? I want to help in migrating Junit4 to Junit5. Can I get some good first issues to work on? Thanks :)

[TimvdLippe]

At the moment, we are not planning to migrate to JUnit 5, since it is a complete revamp. For now, JUnit 4 is serving our needs.