Instrument dynamic JUnit 5 tests

[billybong]

Description
Great project, I really like the idea and the configuration model.

I'd love to have the alternative to instrument dynamic tests as well, which AFAIK does not respect annotations but would instead need some programmatic API alternative.
Alternatively provide an example on how these could be wired up manually with the current model and some before and after lambdas?

https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests

[jeanbisutti]

Happy that QuickPerf can be useful!

Interesting idea @billybong!

I have a look into the code. Code needs refactoring to have a programmatic API for SQL. It seems also possible for JVM part and to have it working with JUnit 5 dynamic tests but it is more complex because each test is executed in a dedicated JVM.

Do you have a specific use case in mind? A use case that would be very helpful for you?

[billybong]

In my particular use-case we have a ton of tests not written in JUnit (think of it like a system test) that loop through a set of inputs from disk.
I'd like to run these but instrumented to get metrics during these runs and a simple way to do that would be to generate a test for each input rather than manually creating a separate test method for each variant.

Since I've seen you've been involved with Camel the analogy would be i.e loading a bunch of different camel routes from xml files and test their runtime characteristics.

[loicmathieu]

You can intercept Dynamic Tests so it should be possible.
I'll give it a try if I find some time.

[loicmathieu]

I have a quick look at this one and it's easily feasible to support SQL annotations but not JVM one (as it needs to fork the VM and we misses some information from JUnit to know which test is being executable and we need this information to know which method to execute is the new VM).

If it is of interest to provides support for SQL annotations only in JUnit5 dynamic test, I can create a PR for that.

[jeanbisutti]

Great @loicmathieu! Perhaps you may give an example of how it would be possible to use QuickPerf SQL features together with dynamic JUnit 5 tests. With @billybong we could give feedback on this API.

[loicmathieu]

In fact I do have a raw example as I have a basic prototype to support dynamic tests (and it should be easy to support test templates the same way).

The prototype is here: master...loicmathieu:junit5/dynamic-tests

There is a lot of different way to defines dynamic tests, if you define one dynamic test per test factory method it will works the same way standard tests works.

    @QuickPerfTest
    public static class SqlSelectJUnit5 extends SqlTestBaseJUnit5 {

        @ExpectSelect(5) // will be used by the DynamicTest
        @TestFactory
        public DynamicTest execute_one_select_but_five_select_expected() {
            return DynamicTest.dynamicTest("Dynamic test name", () -> {
                EntityManager em = emf.createEntityManager();
                Query query = em.createQuery("FROM " + Book.class.getCanonicalName());
                query.getResultList();
            });
        }

    }

If you want to define a single TestFactory method that generates multiple dynamic tests, then the annotation approach will not be works very will as the same annotation will be used by all dynamic tests defined by the single test factory method (needs to be tested but should be the truth).

    @QuickPerfTest
    public static class SqlSelectJUnit5 extends SqlTestBaseJUnit5 {

        @ExpectSelect(5) // this annotation will be used for both tests
        @TestFactory
        public List<DynamicTest> execute_one_select_but_five_select_expected() {
            return List.of(
                DynamicTest.dynamicTest("Dynamic test name", () -> {
                    EntityManager em = emf.createEntityManager();
                    Query query = em.createQuery("FROM " + Book.class.getCanonicalName());
                    query.getResultList();
                }),
               DynamicTest.dynamicTest("Dynamic test name", () -> {
                    EntityManager em = emf.createEntityManager();
                    Query query = em.createQuery("FROM " + Person.class.getCanonicalName());
                    query.getResultList();
                })
            );
        }

    }

So with my current approach we will give a partial support for dynamic test as if you want to defines multiple dynamic test inside the same test factory method with different annotations for each it will not be possible.

[jeanbisutti]

This first implementation of dynamic tests seems already interesting!

So with my current approach we will give a partial support for dynamic test as if you want to defines multiple dynamic test inside the same test factory method with different annotations for each it will not be possible.

An interesting use case would be to apply the recommended global annotations. @loicmathieu Is global annotation configuration applied to dynamic tests? The answer is probably yes. If not, the recommended global annotations could be added on the test method annotated with@TestFactory.

@billybong This implementation of @loicmathieu could be already useful for you?

[loicmathieu]

@jeanbisutti it needs more tests to cover all possible ways to defines dynamic tests but I think it will also works with global annotation (I don't see why it will not works).

I'll implements more tests and also provides an example application as it will be easier for user to understand what is supported by providing an example application.

[jeanbisutti]

@loicmathieu
Could you please update the documentation for the next release?
I copied the wiki pages in this repo.
The JUnit 5 wiki page is here. You should be able to edit the wiki pages directly from GitHub.
You can also clone this wiki: git clone https://github.com/jeanbisutti/quick-perf-doc-1.0.wiki.git

[loicmathieu]

@jeanbisutti doc done.
As soon as a version is available I can also provides some example inside the example repository

[jeanbisutti]

@billybong

QuickPerf 1.0.0 was released. QuickPerf SQL annotations work with JUnit 5 dynamics tests. The documentation is here.

About dynamic tests and QuickPerf JVM annotations, we created the #115 QuickPerf issue. It should be fixed on JUnit 5.8, see junit-team/junit5#2445.

I am going to close this issue. If you need anything else, don't hesitate to create a new one.