New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JUnit 5 extension #887

Merged
merged 41 commits into from Nov 2, 2018

Conversation

@britter
Copy link
Contributor

britter commented Sep 28, 2018

This is related to #636 but takes a different approach: instead of upgrading testcontainers core to JUnit 5 APIs, this introduces a new module containing a JUnit 5 extension that uses testcontainers core.

Static fields are shared among test methods (started only once and stopped at the end), while instance fields are started and stopped for every method. This approach has the advantage that it is relatively easy to implement. However, the downside of this is, that shared containers can't be used inside of @Nested test cases since these must not be declared as static inner classes and hence can not have static fields.

Feedback welcome @kiview, @sormuras, @nicolaiparlog, @marcphilipp

britter added some commits Sep 28, 2018

@britter britter requested review from bsideup , kiview and rnorth as code owners Sep 28, 2018

@kiview

This comment has been minimized.

Copy link
Member

kiview commented Sep 28, 2018

Hey @britter, great that you've contributed the JUnit5 extension, I told you it's not too much work 😉
And cool approach similar to the Spock implementation, let's see what the others will think about this.

I'll do the review later this weekend 🙂

@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Sep 29, 2018

Hi @britter! Thanks for contributing your vision of JUnit 5 support 👍
I'm totally okay to replace my PR with yours :) Could you please migrate the test cases tho? I've tried to reflect most of the use cases there, and (when I was thinking about the implementation) some use cases were not possible to cover with an extension, at least with the annotation-based one, but maybe I'm wrong :)

@britter

This comment has been minimized.

Copy link
Contributor

britter commented Sep 29, 2018

Hi @bsideup,

sure thing. Which test cases do you mean? Those you added in #636?
The Codacy analysis fails but the failures seem to be false positives. Can you confirm?

@nicolaiparlog
Copy link

nicolaiparlog left a comment

Since I was @'ed, I figured I'd give a quick review. I know little about Testcontainers (except that once they have proper JUnit 5 integration and no longer rely on JUnit 4, I will start using them immediately) and so focused on the Jupiter extension. Straightforward and well implemented. 👍

* {@code @Testcontainers} is a JUnit Jupiter extension to activate automatic
* startup and stop of test containers used in a test case.
*
* <p>The test containers extension finds all fields of typ

This comment has been minimized.

@nicolaiparlog
class TestcontainersExtension implements BeforeAllCallback, BeforeEachCallback, AfterAllCallback, AfterEachCallback {

@Override
public void beforeAll(final ExtensionContext context) throws IllegalAccessException {

This comment has been minimized.

@nicolaiparlog

nicolaiparlog Sep 30, 2018

You could reduce code duplication between beforeAll and afterAll by having a method forEachSharedContainer(Consumer<Field>) that does the same thing as this method, but instead of startContainer calls Consumer::consume with the field.

}

@Override
public void beforeEach(final ExtensionContext context) throws IllegalAccessException {

This comment has been minimized.

@nicolaiparlog

nicolaiparlog Sep 30, 2018

Same as for beforeAll.

import static org.junit.jupiter.api.Assertions.assertEquals;

@Testcontainers
class ComposeContainerIT {

This comment has been minimized.

@nicolaiparlog

nicolaiparlog Sep 30, 2018

I don't know how Testcontainers in general or these new tests in particular are structured, so my point may be moot, but here it goes: I moved away from using test class names to distinguish unit and integration test. Instead I apply Jupiter tags (e.g. with @Tag("integration")) and configure the build tool to filter by tags instead of file names. If you're interested in that, let me know - I have more than these two cents to give. 😉

This comment has been minimized.

@kiview

kiview Oct 1, 2018

Member

We build Testcontainers using Gradle and currently don't have dedicated source sets for Unit- and Integration-Tests. I would assume if we would retstructure our testing phases, we would use two distinct source sets for Unit- and Integration-Tests, so we can use it for all modules.

This comment has been minimized.

@marcphilipp

marcphilipp Oct 3, 2018

Alternatively, you could use tags and define two separate test tasks.

This comment has been minimized.

@bsideup

bsideup Oct 29, 2018

Member

I would avoid IT suffix (a rudiment from Maven)

This comment has been minimized.

@kiview

kiview Oct 29, 2018

Member

I think actually just copied over from the Spock tests 🙂

@rnorth

This comment has been minimized.

Copy link
Member

rnorth commented Sep 30, 2018

@britter thanks for raising this! I haven't had a chance to review/play with this yet, but re the Codacy warnings: yes, please consider these false positives. There's nothing there that would worry me for test-scoped code.


/**
* {@code @Testcontainers} is a JUnit Jupiter extension to activate automatic
* startup and stop of test containers used in a test case.

This comment has been minimized.

@kiview

kiview Oct 1, 2018

Member

I would omit the 'test' from 'test container' here.

import static org.junit.jupiter.api.Assertions.assertTrue;

/**
* This test verifies, that setup and cleanup of containers works correctly.

This comment has been minimized.

@kiview

kiview Oct 1, 2018

Member

I wonder right know, if this comment is still true for the existing Spock tests structure. Either way, this is kind of hackish and I think we should refactor this test in the Spock module and don't want to duplicate it for this new module :D

@kiview
Copy link
Member

kiview left a comment

I think @bsideup is referring to the inheritance and abstract class tests:
https://github.com/testcontainers/testcontainers-java/pull/636/files#diff-d6b833ecb1659337d0093eab00102635

I'm very happy with the PR, since the integration surface is super slim and the API is the same than our Spock implementation.

One thing to keep in mind is, that VncRecording will not work right now. But we can add the support in a later PR.

@britter

This comment has been minimized.

Copy link
Contributor

britter commented Oct 1, 2018

I'm doing another iteration. @nicolaiparlog has pointed me to the TestInstancePostProcessor extension point in JUnit Jupiter. This way I can implement shared containers as instance fields instead of static fields. This way the @Nested use case can also be implemented.

I'll also adapt the inheritance test case making sure that this use case is also supported!

@britter

This comment has been minimized.

Copy link
Contributor

britter commented Oct 1, 2018

After another iteration I came up with a solution that is based on TestInstancePostProcessor. I've not pushed my solution, it has other draw backs which I'd like to outline below. I think we need to decide which approach we want to use and live with the drawbacks:

Solution 1 uses static fields to enable shared containers. This way shared containers are bound the the test class. The problem with this approach is, that shared containers can't be used inside nested test cases, because nested test cases must not be declared as static classes and can therefore not have static fields.

Solution 2 uses the TestInstancePostProcessor extension point to create shared containers. Shared containers can be declared as instance fields, so they are bound to the test instance rather then to the test class. The problem with this is, that containers can only be shared if the test is annotated with @TestInstance(Lifecycle.PER_CLASS) because otherwise it would be recreated every time the test class is instantiated. The the drawback here is, that users have to be aware of this and that they are forced to change the test instance lifecycle for shared containers to work. On the other hand with this approach shared containers also work in nested test cases.

I currently don't know which solution we should implement... So I'm happy for every feedback.

@britter britter referenced this pull request Oct 1, 2018

Merged

Introduce JUnit Platform Test Kit #1392

6 of 7 tasks complete

britter added some commits Oct 1, 2018

@kiview

This comment has been minimized.

Copy link
Member

kiview commented Oct 2, 2018

I think nested tests is a super convenient and powerful feature, however I'm not sure how many users will actually use them or know about this feature? Is it very prominent in JUnit5?

I have no real problem with **Solution 2 ** if we document it accordingly and would prefer it in order to support nested tests.

@rnorth @bsideup WDYT?

@marcphilipp
Copy link

marcphilipp left a comment

Regarding solution 2: There's no way around the instance variable being instantiated twice. You could store the first one in the extension context of the nested test class and replace it using a TestInstancePostProcessor. However, I would consider that a hack. 😉

Thus, I'm in favor of using static fields and adding documentation that those won't work in nested classes.

*
* <p>The test containers extension finds all fields of typ
* {@link org.testcontainers.lifecycle.Startable} and calls their container
* lifecylce methods. Containers declared as static fields will be shared

This comment has been minimized.

@marcphilipp

marcphilipp Oct 3, 2018

Typo: "lifecylce"

@Override
public void beforeAll(final ExtensionContext context) throws IllegalAccessException {
Class<?> testClass = context.getRequiredTestClass();
for (final Field field : testClass.getDeclaredFields()) {

This comment has been minimized.

@marcphilipp

marcphilipp Oct 3, 2018

This should take into account fields declared in super classes, shouldn't it? Same for the other methods in this class.

@Override
public void afterEach(final ExtensionContext context) throws IllegalAccessException {
Object testInstance = context.getRequiredTestInstance();
for (Field field : testInstance.getClass().getDeclaredFields()) {

This comment has been minimized.

@marcphilipp

marcphilipp Oct 3, 2018

Alternatively, you could store the started containers in the ExtensionContext and retrieve them from there. You could even store CloseableResource implementations and let Jupiter take care of stopping the containers.

import static org.junit.jupiter.api.Assertions.assertEquals;

@Testcontainers
class ComposeContainerIT {

This comment has been minimized.

@marcphilipp

marcphilipp Oct 3, 2018

Alternatively, you could use tags and define two separate test tasks.

@kiview

kiview approved these changes Oct 26, 2018

@kiview

This comment has been minimized.

Copy link
Member

kiview commented Oct 26, 2018

Great, can you have a final look, @rnorth @bsideup?

Show resolved Hide resolved modules/junit-jupiter/README.md Outdated
@rnorth

This comment has been minimized.

Copy link
Member

rnorth commented Oct 27, 2018

This looks amazing - thank you all for contributing and refining this! I only had one minor comment on the docs; this is already a well polished PR by the time I've got to it 😄.

I'd like to have a little bit more of a play with this tonight, but I'll be looking forward to us merging and releasing this ASAP.

Thank you 🙇

import static org.junit.jupiter.api.Assertions.assertTrue;

@Testcontainers
class TestcontainersNestedRestaredContainerIT {

This comment has been minimized.

@rnorth

rnorth Oct 27, 2018

Member

Tiny typo!

Suggested change Beta
class TestcontainersNestedRestaredContainerIT {
class TestcontainersNestedRestartedContainerIT {

rnorth and others added some commits Oct 28, 2018

Shared containers have to be declared static
Co-Authored-By: britter <beneritter@gmail.com>

@testcontainers testcontainers deleted a comment from codacy-bot Oct 28, 2018

@kiview kiview added the type/feature label Oct 29, 2018

@kiview kiview added this to the next milestone Oct 29, 2018

@bsideup bsideup referenced this pull request Oct 29, 2018

Closed

[WIP] JUnit 5 #636

store.put(TEST_INSTANCE, testInstance);

findSharedContainers(testInstance)
.forEach(adapter -> store.getOrComputeIfAbsent(adapter.key, k -> adapter.start()));

This comment has been minimized.

@bsideup

bsideup Oct 29, 2018

Member

Any reason to use field instead of getter?

This comment has been minimized.

@britter

britter Oct 30, 2018

Contributor

The StoreAdapter class is a private class inside of TestcontainersExtension. So all fields are visible from TestcontainersExtension and there is no gain from an information hiding perspective if we add getters.

This comment has been minimized.

@bsideup

bsideup Oct 30, 2018

Member

The problem I see is that it is inconsistent with the rest of our code base because we always use getters :)

}

private static Predicate<Field> isContainer() {
return field -> Startable.class.isAssignableFrom(field.getType()) && AnnotationSupport.isAnnotated(field, Container.class);

This comment has been minimized.

@bsideup

bsideup Oct 29, 2018

Member

I would return any field annotated with @Container and throw and error if it is not Startable, otherwise it will simply ignore it

@sormuras

This comment has been minimized.

Copy link

sormuras commented Oct 29, 2018

I know, this PR is in a late and almost finished state -- but I promised @britter to spike a different way to handle shared containers, which is also thread safe. See https://github.com/junit-team/junit5-samples/compare/singleton-extension for a work-in-progress implementation using a StringBuilder as a shared resource.

Transferred to this container context, the test class could read like:

@ExtendWith(SingletonExtension.class)
class SingletonExtensionTests {

	@Test
	void test1(@Singleton(MySQL123.class) MySQLContainer shared) {}

	@Test
	void test2(@Singleton(MySQL123.class) MySQLContainer shared) {}

	@Test
	void test3(@New(MySQL123.class) MySQLContainer fresh) {}

}

The MySQL123 class is used as resource factory:

public class MySQL123 implements SingletonExtension.Resource<MySQLContainer> {

	private final MySQLContainer container = new MySQLContainer();

	@Override
	public MySQLContainer get() {
		return container;
	}
}

Edit: refactored the implementation and filed junit-team/junit5-samples#87 as a sample for discussion

britter added some commits Oct 30, 2018

@codacy-bot

This comment has been minimized.

Copy link

codacy-bot commented Oct 30, 2018

Codacy Here is an overview of what got changed by this pull request:

Issues
======
- Added 29
           

See the complete overview on Codacy

@kiview kiview merged commit d7c4708 into testcontainers:master Nov 2, 2018

6 of 8 checks passed

Codacy/PR Quality Review Not up to standards. This pull request quality could be better.
Details
WIP work in progress
Details
ci/circleci: core Your tests passed on CircleCI!
Details
ci/circleci: modules-jdbc-test Your tests passed on CircleCI!
Details
ci/circleci: modules-no-jdbc-test-no-selenium Your tests passed on CircleCI!
Details
ci/circleci: okhttp Your tests passed on CircleCI!
Details
ci/circleci: selenium Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

kiview pushed a commit that referenced this pull request Nov 2, 2018

Kevin Wittek

rnorth added a commit that referenced this pull request Nov 4, 2018

Small improvements to Jupiter extension code style (#947)
* Small improvements to Jupiter extension code style

Follow up to #887

* Only raise exception if annotated field is not Startable
@jrehwaldt

This comment has been minimized.

Copy link

jrehwaldt commented Dec 17, 2018

I am trying to use this extension together with @SpringBootTest. How to ensure testcontainer starts first before executing SpringBootTest extension?

From what I see in the JUnit 5 documentation the annotation order matters. Since Java sorts annotations in bytecode alphabetically Spring will always run first unless it is possible to explicitly import @ExtendWith(TestcontainersExtension.class). Currently, this cannot be done due to TestcontainersExtension being package-private.

Question is: How to ensure testcontainers ordering relative to other extensions with the current extension mechanism?

@bsideup

This comment has been minimized.

Copy link
Member

bsideup commented Dec 17, 2018

Hi @jrehwaldt!
Are you sure that you need JUnit 5 extension here? If you need containers for your Spring Boot app, you can easily add it inside a static block, test framework free:
https://testcontainers.gitbook.io/workshop/step-6-adding-redis

@jrehwaldt

This comment has been minimized.

Copy link

jrehwaldt commented Dec 17, 2018

Thanks. That's exactly what I am currently doing. I got pointed to this new feature and thought I'd give it a shot. Imho, this extension provides test lifecycle support out of the box without copy and pasting start-stop logic in @[Before|After][All|Each] annotations from test to test. Unfortunately, it seems to me the interplay of multiple extensions wasn't part of the spec'ing, which is why I asked here for clarification.

Am I right in assuming the scenario I describe is simply not supported as of now? I believe making TestcontainersExtension public is all that's needed for users to be able to explicitly define the order of multiple extensions.

@marcphilipp

This comment has been minimized.

Copy link

marcphilipp commented Dec 18, 2018

@jrehwaldt Since this PR is merged, please open a new issue.

@jrehwaldt

This comment has been minimized.

Copy link

jrehwaldt commented Dec 18, 2018

I did in #1017. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment