Support Cartesian products of environments in MultiEnvTestEngine #7816
Conversation
...t/java/org/projectnessie/tools/compatibility/internal/TestNessieCompatibilityExtensions.java
Outdated
Show resolved
Hide resolved
| * <p>Actual test environments are expected to be managed by JUnit 5 extensions, implementing the | ||
| * {@link MultiEnvTestExtension} interface. | ||
| */ | ||
| public class MultiEnvTestEngine implements TestEngine { | ||
|
|
||
| private static final Logger LOGGER = LoggerFactory.getLogger(MultiEnvTestEngine.class); | ||
| private static final boolean FAIL_ON_MISSING_ENVIRONMENTS = | ||
| !Boolean.getBoolean("org.projectnessie.junit.engine.ignore-empty-environments"); | ||
| private static final MultiEnvExtensionRegistry FILTER_REGISTRY = new MultiEnvExtensionRegistry(); |
There was a problem hiding this comment.
@dimas-b you mentioned that this was not a constant. Did you just want the casing changed to filterRegistry, or were you concerned by the addition of a second currentRegistry below?
I found that two registries were needed, one for the test environment creation and one that keeps track of all known MultiEnvTestExtensions to use for test filtering.
There was a problem hiding this comment.
I'd prefer to keep the old lower case name... or perhaps globalRegistry if you want to make it easier to distinguish from currentRegistry.
There was a problem hiding this comment.
It was not needed. But I found a couple interesting things:
- The registry needs to be cleared between each test. I found tests like
bothOnJunitJupiterwould fail if run alone, but would pass when the wholeTestMultiEnvTestEngineclass was run. Clearing the registries between tests exposes this issue. - The filter needs its own registry in case multi-env tests are selected on other engines (also see
bothOnJunitJupiter)
| JupiterConfiguration nodeConfiguration = | ||
| new CachingJupiterConfiguration( | ||
| new MultiEnvJupiterConfiguration( | ||
| configurationParameters, "TODO")); // TODO |
There was a problem hiding this comment.
I still need to look at test naming here.
There was a problem hiding this comment.
Thanks for your contribution @colan-dremio !
This is just a preliminary quick review from my side :)
| * </ul> | ||
| * will result in the following IDs: | ||
| * <ul> | ||
| * <li>[engine:nessie-multi-env][segmentA:1][segmentB:1][segmentC:1]</li> |
There was a problem hiding this comment.
I believe MultiEnvTestFilter needs to be updated to check whether all multi-env ID segments are recognised... Currently it checks that any match.
| descriptor -> { | ||
| if (descriptor instanceof ClassBasedTestDescriptor) { | ||
| Class<?> testClass = ((ClassBasedTestDescriptor) descriptor).getTestClass(); | ||
| registry.registerExtensions(testClass); | ||
| FILTER_REGISTRY.registerExtensions(testClass); | ||
| currentRegistry.registerExtensions(testClass); |
There was a problem hiding this comment.
Why exactly do we need two registries?
currentRegistry is used only during discovery, but MultiEnvTestFilter should use the same extensions for filtering, which is why the registry was static before. See comments in MultiEnvTestFilter for more details about the rationale.
There was a problem hiding this comment.
(replied in #7816 (comment) instead accidentally)
...g/multi-env-test-engine/src/main/java/org/projectnessie/junit/engine/MultiEnvTestEngine.java
Outdated
Show resolved
Hide resolved
|
|
||
| public void clear() { | ||
| registry = MutableExtensionRegistry.createRegistryWithDefaultExtensions( | ||
| new DefaultJupiterConfiguration(new EmptyConfigurationParameters())); |
There was a problem hiding this comment.
Let's call this from the constructor to avoid duplicating (re-)initialization logic.
| * UniqueID. Higher numbers will appear earlier in the JUnit UniqueId. For extensions with the | ||
| * same order value, the segmentType will be sorted alphabetically. | ||
| */ | ||
| default int getOrder() { |
There was a problem hiding this comment.
How about priority() or weight()?
- the
getprefix does not align with theallEnvironmentIds()method. - "order" sounds a bit too strong. This value affects the placement of the extension's segment in the test
UniqueIdvalues... it does not affect execution order of tests or extensions at all (cf. the JUnit5@Orderannotation).
| MultiEnvExtensionRegistry registry = registry(); | ||
|
|
||
| // Keep separate registry from MultiEnvTestEngine in case we are running on another engine and still need to filter. | ||
| registry.registerExtensions(testClass); | ||
| if (id.getEngineId().map("junit-jupiter"::equals).orElse(false)) { | ||
| if (registry.stream(testClass).findAny().isPresent()) { |
There was a problem hiding this comment.
WDYT about moving segmentType() to an annotation on MultiEnvTestExtension? Then a registry instance will not be required here at all. The stream method could probably be refactored to offer a static part for discovering annotations.
Currently,
MultiEnvTestEngineonly supports a singleMultiEnvTestExtensionat a time. This adds support for multipleMultiEnvTestExtensions. It will perform a Cartesian product of the registered environment IDs.For example, it would become possible to test multiple Nessie client versions with multiple Nessie server versions through a combination of
OlderNessieClientsExtensionandOlderNessieServersExtension. Withnessie.versionsproperty set to1,2, the following test environments would run:This is also enables downstream projects to use their own implementations of
MultiEnvTestExtension. For example, a downstream project could test multiple Nessie versions against multiple types of object storage.