Cross JRE testing matrix

[ryandens]

Use Case

We should run our tests (especially, tests in the instrumentation module) on multiple JVMs after compiling them with JDK 11. This was first brought up on #334 in this comment.

Proposal

One possible method of doing this is leveraging Gradle toolchains. Gradle toolchains support registering Test tasks that require certain JDK versions (see Specify custom toolchains for individual tasks). This has the additional benefit of working on developer's machines as well. Notably, if the toolchain is not detected on the developer's machine, Gradle will automatically download a matching toolchain from AdoptOpenJDK

Questions to address (if any)

There may be other methods of accomplishing this level of cross-jvm testing with Gradle, such as using the GitHub Actions setup-java matrix. But, it's not clear to me how this would be implemented effectively while still compiling with JDK 11 so we can detect niche bytecode compatibility bugs. In addition, depending on the setup, it could be difficult for developers to run these tests locally to debug errors. However, without a specific example test workflow in mind, it is difficult to fairly compare alternate solutions. If folks have code snippets/examples of how this setup could work in practice, please feel free to propose them here

[pavolloffay]

But, it's not clear to me how this would be implemented effectively while still compiling with JDK 11 so we can detect niche bytecode compatibility bugs. I

Our tested deliverable is bytecode compiled on JVM 11 into JVM 8 compatible bytecode. What we have to test is that if the deliverable works on other JVMs - all TLS versions and the latest released version.

[ryandens]

Our tested deliverable is bytecode compiled on JVM 11 into JVM 8 compatible bytecode. What we have to test is that if the deliverable works on other JVMs - all TLS versions and the latest released version.

Yeah, I'm aware of that, the solution I'm proposing accomplishes that using Gradle toolchains, allowing us to define a matrix of JVMs to test. What I'm trying to understand is what your solution would look like in practice

[pavolloffay]

I think you got it right, I would use gradle toolchains and matrix configuration on CI.