Plans on testing strategy and approach

[emmanuelbernard]

We need a guide of course capturing that.
Potentially some frameworks helping.

Sub issues for first release

• Guide Write a how to test guide #555
• JUnit 5 Support JUnit 5 and allow legacy JUnit 4 #553

Other sub issues

• Arquillian like subset of the app with no zipping Support Classloader based resource filtering #554
• @Inject in unit tests Add ability to Inject application bean in unit test #620
• Configuration test in Kubernetes Define strategy to do deployment tests in Kubernetes #557
• Mock and Bean alternatives Integrate mock and testing #568
• Inject configured REST client Ability to inject a configured REST client for testing #569
• Reset data between tests via Hibernate ORM Consider offering a way to reset DB at Hibernate ORM level for testing #583

[gsmet]

Currently, we only support JUnit 4, we should add support for JUnit 5.

[cescoffier]

Wondering about things like Cucumber, or let's say "spec based testing". I don't know if it's still popular.

[cescoffier]

+1 for Junit5

[mkouba]

Hey @manovotn, I know you've spent quite some time working on JUnit5 extension for weld-junit. It would be great if you could watch this issue, provide feedback, etc. ;-)

[gunnarmorling]

I was recently looking into testing CDI and the persistence layer locally, i.e. without a container deployment but still allowing for testing things like transactional observers, DI in entity listeners etc. It also should give DI into tests themselves. I've blogged about it here:

http://in.relation.to/2019/01/23/testing-cdi-beans-and-persistence-layer-under-java-se/

The take away is that it's doable, but requires a bit of set-up, that shouldn't be needed if we want to give a great "it works out of the box" experience. I've opened an issue with the Weld JUnit project, hoping the story can be improved there. The same things should work when testing Protean apps:

• DI into the test class
• Can test CDI beans including things like events, interceptors etc.
• Can inject EM and EMF into test and beans
• Can DI into JPA entity listeners
• Can test transactional observers, declarative TX (@Transactional)
• Can rollback TX in tests (that'd require a new @Transactional annotation with with a new TX behavior ROLLBACK as that's not foreseen by JTA)

Such tests are more than unit tests but allow for testing CDI and JPA in conjunction, resulting in high fidelity (no mocking) and ensured functioning of the app components integrated.

[geoand]

I was recently looking into testing CDI and the persistence layer locally, i.e. without a container deployment but still allowing for testing things like transactional observers, DI in entity listeners etc. It also should give DI into tests themselves. I've blogged about it here:

http://in.relation.to/2019/01/23/testing-cdi-beans-and-persistence-layer-under-java-se/

The take away is that it's doable, but requires a bit of set-up, that shouldn't be needed if we want to give a great "it works out of the box" experience. I've opened an issue with the Weld JUnit project, hoping the story can be improved there. The same things should work when testing Protean apps:

• DI into the test class
• Can test CDI beans including things like events, interceptors etc.
• Can inject EM and EMF into test and beans
• Can DI into JPA entity listeners
• Can test transactional observers, declarative TX (@Transactional)
• Can rollback TX in tests (that'd require a new @Transactional annotation with with a new TX behavior ROLLBACK as that's not foreseen by JTA)

Such tests are more than unit tests but allow for testing CDI and JPA in conjunction, resulting in high fidelity (no mocking) and ensured functioning of the app components integrated.

From a Spring user's perspective, this makes perfect sense.
Spring Boot provides some very convenient abilities much like you describe. See things like: @DataJpaTest, @WebMvcTest @JdbcTest etc

[emmanuelbernard]

@gunnarmorling so I think you first 4 bullet points would be covered by #620

I don't understand what is entailed by bullet point 5.
On bullet point 6, can you also describe what you want more about it.

[emmanuelbernard]

@geoand I'm not sure I follow, can you describe more in details what these test specific annotations or runners are doing to address Gunnar's problems?

[geoand]

@geoand I'm not sure I follow, can you describe more in details what these test specific annotations or runners are doing to address Gunnar's problems?

When adding @DataJpaTest or @JdbcTest Spring can automatically rollback the database after each test (or after the entire class) is run.
Also it allows injection of the EntityManager or EntityManagerFactory.

DI of course of any bean into the test class works out of the box.

Not exactly related but similar, for the web layer when using @WebMvcTest a Spring MVC controller can be tested with it's dependencies mocked - which is meant to test that the annotations on the controller have been properly added.
So in this case you have something of a hybrid test - since Spring is obviously in play but not the entire framework and the user can supply their own mocks.

[gunnarmorling]

I don't understand what is entailed by bullet point 5.

Essentially I'd like to be able to test a bean like this which as @Transactional methods and transactional observer methods (the test is here).

For this to work, an interceptor must be set up which handles @Transactional and starts/stops TX as needed. This also needs to let Weld register its TX synchronization needed in order to run the "after-transaction" observer methods. For the blog post I did that by setting up Narayana and registering a custom TransactionServices with Weld. This should all work out of the box.

On bullet point 6, can you also describe what you want more about it.

Yes, oftentimes you don't want to commit transactions during tests (unless you are testing stuff like after-transaction observer methods of course) but instead set up some data, do your test and rollback the TX. That's practical when not working with an in-memory DB but with a (potentially shared) remote DB which you don't want to leave tainted. Something like this helps in that case:

@Transactional(rollback=true)
@Test
public void someTest() {
    // call the beans
}

That'd require TX semantics of the beans under test to be so they join that TX started by the test runner instead of starting and committing their own.

[gunnarmorling]

I think you first 4 bullet points would be covered by #620

Note that there are no CDI beans for EM and EMF by default, i.e. you'd have to use @PersistenceContext or alternatively a custom producer for these beans (that's what I did for the blog).

[stuartwdouglas]

Yes there are, its not spec compliant but we auto create beans for them if there is only a single EMF.

[gunnarmorling]

Ok, that's something Protean-specific? Cool then.

[emmanuelbernard]

Interesting, we did discuss the objective of rolling back tx for tests but Sanne and I felt it was too limiting. But if that's an annotation on the test, that flies as an option. The alternative is this #583

[Sanne]

Humm it's still limiting.

I need to be ableto test this:

sessionOne.save( x):
tx.commit();
sessionOne.close();

fulltextSession = open...
fulltextSession#runCoolQuery -> Assert Query actually works as intended?

This only flies if you can control tx boundaries within the test.

[gunnarmorling]

This only flies if you can control tx boundaries within the test.

You still could do this. Just don't specify the annotation at the test method then.

But while that may be needed in some cases, in many others it's not. Just insert test data, perhaps flush the EM, do your testing logic and roll back. Often that's just fine.

[Sanne]

Ok that's not too bad then.

I'm not particularly sold on the benefit of rollback though. Just to keep the database clean?

I understand the benefit of isolating tests without the need for resetting the RDBMS state - but if you allow some tests to not rollback tests are not safely isolated anyway.

I guess one could automate that those tests which don't "self-rollback" automatically wipe the DB - yet personally I wonder how confusing this is getting.

[stuartwdouglas]

So the approach I am looking at for allowing injection into tests is to basically just make the test class into a Bean. A neat side effect of this is that I only just realised is that you can apply standard CDI interceptors to test methods.

This could potentially allow us to have a fairly advanced test extension model using standard API's.

[stuartwdouglas]

bc5f769 allows for test managed, rollback only transactions

[gunnarmorling]

bc5f769 allows for test managed, rollback only transactions

Nice! To be sure, there's no handling of the regular @Transactional yet, right? I.e. for any beans under test there would no be TX started upon method invocation?

[stuartwdouglas]

Tests are now CDI beans, @transactional should just work

[stuartwdouglas]

Also for beans under test they will follow the usual TX rules based on whatever annotations or transaction management they have

[gunnarmorling]

Also for beans under test they will follow the usual TX rules based on whatever annotations or transaction management they have

I think there should be a simple way to override this when using your new @TestTransaction annotation. Otherwise any bean with @Transactional will have its changes comitted. I.e. it'd be good to have a way where the beans just join the TX started for @TestTransaction.

[gunnarmorling]

@transactional should just work

Which interceptors is this using, the ones coming with Narayana?

[stuartwdouglas]

That is not how it works, unless they are using REQUIRES_NEW. Any bean with @transactional will just join the outer TX, which will get rolled back. If you are testing REQUIRES_NEW stuff then that is an advanced use case and I think you are on your own.

@transactional will just use the standard one. This is all just standard CDI, so everything should just work (probably).

[gunnarmorling]

If you are testing REQUIRES_NEW stuff then that is an advanced use case and I think you are on your own.

Yes, REQUIRES_NEW was what I had in mind. But yeah, I guess you're right, in that case you'd just have to omit @TestTransaction and deal with it yourself.

@transactional will just use the standard one. This is all just standard CDI, so everything should just work (probably).

Ok, so haven't looked into what you've done in Shamrock, but CDI relies on JTA to make this work, and it requires a bit of plumbing. The interceptors must be present (they are not part of Weld but of the Narayana JTA JAR), and a TransactionServices implementation must be plugged into Weld, exposing the transaction manager essentially (e.g. to make transactional observers work). Might well be of course that's all being done in Shamrock already. Just in case, in the blog post linked above the required steps are described.

[mkouba]

The interceptors must be present...

We have our own interceptors. Also we don't use Weld. And transactional observers are not supported at the moment.

[emmanuelbernard]

So this one waits for @stuartwdouglas writing guide and get @Inject working on ShaprockTest.

[stuartwdouglas]

@Inject works now, guide will be tomorrow

[n1hility]

On triage call, determined will split the issue

[emmanuelbernard]

Renaming title as the first public release is done-ish.

[emmanuelbernard]

and reopen because not enough coffee

[hantsy]

Currently it seems all quickstarts are demonstrating QuarkusTest of JAX-RS.

Hope it includes:

• Mockito integration, mock CDI by some annotations like @MockBean @SpyBean
• Provides embedded server for test purpose, such as H2 for JPA, embedded Mongo for Mongo. etc
• Integrate TestContainers to setup dependent service(such as a MySql server) in Docker directly

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you!
We are doing this automatically to ensure out-of-date issues does not stay around indefinitely.
If you believe this issue is still relevant please put a comment on it on why and if it truly needs to stay request or add 'pinned' label.

[antonin-arquey]

Wondering about things like Cucumber, or let's say "spec based testing". I don't know if it's still popular.

Hi, we use Cucumber for testing in my organization, but I can't get it to work with Quarkus tests. The quarkus context is never started even with the @QuarkusTest annotation.

We use it with SpringBoot thanks to the spring cucumber plugin and it works great !

Would it be possible to do something similar with Quarkus ?

[stuartwdouglas]

It looks like Cucumber is still using JUnit 4, while we are using JUnit 5. Did you try including junit-vintage-engine as the cucumber docs suggest? It's likely that we will need to write some kind of plugin to allow it to work, especially after the class loading changes that are coming in with my class loading PR. Stuart

…

On Mon, 20 Jan 2020 at 07:25, antoninarquey ***@***.***> wrote: Wondering about things like Cucumber, or let's say "spec based testing". I don't know if it's still popular. Hi, we use Cucumber for testing in my organization, but I can't get it to work with Quarkus tests. The quarkus context is never started even with the @QuarkusTest annotation. We use it with SpringBoot thanks to the spring cucumber plugin <https://github.com/cucumber/cucumber-jvm/tree/master/spring> and it works great ! Would it be possible to do something similar with Quarkus ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#461?email_source=notifications&email_token=AACQG62C4LEHRJD4ZIK7VMDQ6SZJTA5CNFSM4GPEGS6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJK3RAY#issuecomment-576043139>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQG64C3PU7ILYR2RV3F4LQ6SZJTANCNFSM4GPEGS6A> .

[emmanuelbernard]

This issue is not mostly obsolete, closing.

[j-martinez-dev]

Wondering about things like Cucumber, or let's say "spec based testing". I don't know if it's still popular.

Hi, we use Cucumber for testing in my organization, but I can't get it to work with Quarkus tests. The quarkus context is never started even with the @QuarkusTest annotation.

We use it with SpringBoot thanks to the spring cucumber plugin and it works great !

Would it be possible to do something similar with Quarkus ?

Did you find a solution for this?

[juanmanuelz]

Wondering about things like Cucumber, or let's say "spec based testing". I don't know if it's still popular.

Hi, we use Cucumber for testing in my organization, but I can't get it to work with Quarkus tests. The quarkus context is never started even with the @QuarkusTest annotation.
We use it with SpringBoot thanks to the spring cucumber plugin and it works great !
Would it be possible to do something similar with Quarkus ?

Did you find a solution for this?

I'm using a workaround. Using the information here https://www.objectivity.co.uk/blog/cucumber-parallel-scenarios-junit5-gradle/ and https://github.com/cucumber/cucumber-jvm/tree/master/junit-platform-engine . In addition to that, I had to create some code to mimic what happens when @QuarkusTest is used (I will write you back with the details, is nothing worth a gist or a reference here because is just a really "crappy" workaround to be used in the meantime this is sorted out properly).

[stuartwdouglas]

Do you have a simple @QuarkusTest + Cucumber example I could look at to see if I could get it to work? I have never used cucumber, so I am not really sure what the expectation is here, but I think I would need to write a custom quarkus-cucmber module to integrate them.