Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support testing for multi module projects #19793

Merged
merged 1 commit into from Sep 29, 2021

Conversation

stuartwdouglas
Copy link
Member

Continuous testing will now run tests from all modules, with the option
to configure the old behaviour.

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 1, 2021

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building 2b61da5

Status Name Step Failures Logs Raw logs
✔️ JVM Tests - JDK 11
JVM Tests - JDK 16 Build Failures Logs Raw logs

Failures

⚙️ JVM Tests - JDK 16 #

- Failing: devtools/gradle 

📦 devtools/gradle

Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:exec (gradle) on project io.quarkus.gradle.plugin: Command execution failed.

@famod
Copy link
Member

famod commented Sep 1, 2021

I gave this a try and it does execute a test from an upstream module, but it fails with an AmbiguousResolutionException:

Ambiguous dependencies for type com.github.famod.modmono_quarkus.core.HelloBean and qualifiers [@Default]
        - java member: com.github.famod.modmono_quarkus.core.testlib.HelloBeanTest#underTest
        - declared on CLASS bean [types=[com.github.famod.modmono_quarkus.core.testlib.HelloBeanTest, java.lang.Object], qualifiers=[@Default, @Any], target=com.github.famod.modmono_quarkus.core.testlib.HelloBeanTest]
        - available beans:
                - CLASS bean [types=[com.github.famod.modmono_quarkus.core.HelloBean, java.lang.Object], qualifiers=[@Default, @Any], target=com.github.famod.modmono_quarkus.core.HelloBean]
                - CLASS bean [types=[com.github.famod.modmono_quarkus.core.HelloBean, java.lang.Object], qualifiers=[@Default, @Any], target=com.github.famod.modmono_quarkus.core.HelloBean]
        at io.quarkus.arc.processor.Beans.resolveInjectionPoint(Beans.java:501)
        at io.quarkus.arc.processor.BeanInfo.init(BeanInfo.java:470)
        at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:254)
        ... 13 more

The reproducer is here: https://github.com/famod/modmono-quarkus/tree/cont-testing-multi

  • ./mvnw clean install
  • ./mvnw quarkus:dev -f dist/
  • resume testing -> fails

PS: Please ignore the mocks profile for now, it's related to #19152.

@stuartwdouglas stuartwdouglas marked this pull request as draft September 2, 2021 03:01
@stuartwdouglas
Copy link
Member Author

This needs some more work, I need to properly resolve an AppModel for each module. I am not sure how practical this will be with the current codebase, it may need to be delayed until after the planned refactoring work.

@tbvl
Copy link

tbvl commented Sep 6, 2021

@stuartwdouglas Sorry for the delay, but now I finally had a little time to try it out. Testing was exploratory.

Observations

  • Overall for us it worked fine for us out of the box. Test in dependent modules gets executed, and re-executed on changes.
  • We did not experience CDI related issues as @famod, but I also have not checked for differences in our setup.
  • Ressource paths can give issues. Some of our unit tests are using relative paths to test files in the target folder, which works as long as unit tests are executed in isolation within each module, like surefire does. This it is bad practice on our part and something we will fix, but it does give some input to the documentation of the limitations multi module ct feature.

Issues
I saw these two (minor) issues:

  1. @ExtendWith not working until added to cp in the module having the Quarkus runtime
    We have a test using Mockito in one of the dependee modules. The classloader seems to not include the mockito libraries.
    image
    It works fine when adding the dependency to mockito to the module where we execute quarkus from.

  2. Custom tags does not get recoqnized
    We have defined our own custom test tags (including @UnitTest for annotating our test classes for easy filtering and consistency, but it seems Quarkus only picks up test classes where at least one method is annotated directly with @test or the class is annotated with @QuarkusTest.
    image

Hope this input will help, and I hope you will find time to work on it again after the refactoring work. We still find the feature very valuable.

@stuartwdouglas
Copy link
Member Author

So issue 1) is the one that I mentioned was a known problem. It might still be worth merging this with it as a documented limitation, possibly with this disabled by default until it is resolved.

  1. is a separate issue, can you open a new issue and I will look at fixing it.

@stuartwdouglas
Copy link
Member Author

BTW for point 2 I think if you add org.junit.platform.commons.annotation.Testable to the annotation it may work.

@stuartwdouglas
Copy link
Member Author

I have added a note about the limitation and changed this so it is not longer the default, however I think it is still worth including.

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 7, 2021

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building 52ce012

Status Name Step Failures Logs Raw logs
Initial JDK 11 Build Build ⚠️ Check → Logs Raw logs

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 7, 2021

This workflow status is outdated as a new workflow run has been triggered.

🚫 This workflow run has been cancelled.

Failing Jobs - Building 52ce012

⚠️ Artifacts of the workflow run were not available thus the report misses some details.

Status Name Step Failures Logs Raw logs
Attach pull request number ⚠️ Check → Logs Raw logs
CI Sanity Check ⚠️ Check → Logs Raw logs

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 7, 2021

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building 529a1be

Status Name Step Failures Logs Raw logs
JVM Tests - JDK 11 Build Failures Logs Raw logs
JVM Tests - JDK 11 Windows Build Failures Logs Raw logs
JVM Tests - JDK 16 Build Failures Logs Raw logs
Maven Tests - JDK 11 Build Failures Logs Raw logs
Maven Tests - JDK 11 Windows Build Failures Logs Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ JVM Tests - JDK 11 #

- Failing: extensions/smallrye-reactive-messaging-kafka/deployment 
! Skipped: docs integration-tests/kubernetes/quarkus-standard-way-kafka integration-tests/reactive-messaging-kafka and 1 more

📦 extensions/smallrye-reactive-messaging-kafka/deployment

io.quarkus.smallrye.reactivemessaging.kafka.deployment.testing.KafkaDevServicesContinuousTestingTestCase.testContinuousTestingScenario1 line 64 - More details - Source on GitHub

org.opentest4j.AssertionFailedError: expected: <1> but was: <0>
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
	at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)

⚙️ JVM Tests - JDK 11 Windows #

- Failing: extensions/amazon-lambda/deployment 
! Skipped: docs extensions/amazon-lambda-http/deployment extensions/amazon-lambda-rest/deployment and 6 more

📦 extensions/amazon-lambda/deployment

io.quarkus.amazon.lambda.deployment.testing.LambdaDevServicesContinuousTestingTestCase.testLambda line 41 - More details - Source on GitHub

org.opentest4j.AssertionFailedError: expected: <0> but was: <1>
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
	at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)

⚙️ JVM Tests - JDK 16 #

- Failing: extensions/hibernate-orm/deployment 
! Skipped: docs extensions/hibernate-envers/deployment extensions/hibernate-reactive/deployment and 82 more

📦 extensions/hibernate-orm/deployment

io.quarkus.hibernate.orm.HibernateHotReloadTestCase.testImportSqlWithContinuousTesting line 87 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Failed to wait for test run 1 State{lastRun=2, running=true, inProgress=true, run=0, passed=0, failed=0, skipped=0, isBrokenOnly=false, isTestOutput=false, isInstrumentationBasedReload=false, isLiveReload=true}
	at io.quarkus.test.ContinuousTestingTestUtils.waitForNextCompletion(ContinuousTestingTestUtils.java:43)
	at io.quarkus.hibernate.orm.HibernateHotReloadTestCase.testImportSqlWithContinuousTesting(HibernateHotReloadTestCase.java:87)

⚙️ Maven Tests - JDK 11 #

- Failing: integration-tests/maven 

📦 integration-tests/maven

io.quarkus.maven.it.DevMojoIT.testThatTheApplicationIsReloadedMultiModule line 531 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Failed to wait for test run 1 State{lastRun=-1, running=false, inProgress=false, run=0, passed=0, failed=0, skipped=0, isBrokenOnly=false, isTestOutput=false, isInstrumentationBasedReload=false, isLiveReload=true}
	at io.quarkus.test.ContinuousTestingTestUtils.waitForNextCompletion(ContinuousTestingTestUtils.java:43)
	at io.quarkus.maven.it.DevMojoIT.testThatTheApplicationIsReloadedMultiModule(DevMojoIT.java:531)

io.quarkus.maven.it.DevMojoIT.testThatTheApplicationIsReloadedMultiModule line 531 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Failed to wait for test run 1 State{lastRun=-1, running=false, inProgress=false, run=0, passed=0, failed=0, skipped=0, isBrokenOnly=false, isTestOutput=false, isInstrumentationBasedReload=false, isLiveReload=true}
	at io.quarkus.test.ContinuousTestingTestUtils.waitForNextCompletion(ContinuousTestingTestUtils.java:43)
	at io.quarkus.maven.it.DevMojoIT.testThatTheApplicationIsReloadedMultiModule(DevMojoIT.java:531)

⚙️ Maven Tests - JDK 11 Windows #

- Failing: integration-tests/maven 

📦 integration-tests/maven

io.quarkus.maven.it.DevMojoIT.testThatTheApplicationIsReloadedMultiModule line 531 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Failed to wait for test run 1 State{lastRun=-1, running=false, inProgress=false, run=0, passed=0, failed=0, skipped=0, isBrokenOnly=false, isTestOutput=false, isInstrumentationBasedReload=false, isLiveReload=true}
	at io.quarkus.test.ContinuousTestingTestUtils.waitForNextCompletion(ContinuousTestingTestUtils.java:43)
	at io.quarkus.maven.it.DevMojoIT.testThatTheApplicationIsReloadedMultiModule(DevMojoIT.java:531)

io.quarkus.maven.it.DevMojoIT.testThatTheApplicationIsReloadedMultiModule line 531 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Failed to wait for test run 1 State{lastRun=-1, running=false, inProgress=false, run=0, passed=0, failed=0, skipped=0, isBrokenOnly=false, isTestOutput=false, isInstrumentationBasedReload=false, isLiveReload=true}
	at io.quarkus.test.ContinuousTestingTestUtils.waitForNextCompletion(ContinuousTestingTestUtils.java:43)
	at io.quarkus.maven.it.DevMojoIT.testThatTheApplicationIsReloadedMultiModule(DevMojoIT.java:531)

@tbvl
Copy link

tbvl commented Sep 7, 2021

BTW for point 2 I think if you add org.junit.platform.commons.annotation.Testable to the annotation it may work.

Tried this but same result, i.e. it does not get picked up.

So issue 1) is the one that I mentioned was a known problem. It might still be worth merging this with it as a documented limitation, possibly with this disabled by default until it is resolved.

  1. is a separate issue, can you open a new issue and I will look at fixing it.
    Done and thanks: Junit custom tags do not get picked up by Continuous Testing engine #19971

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 10, 2021

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building 5892854

Status Name Step Failures Logs Raw logs
Initial JDK 11 Build Build Failures Logs Raw logs

Failures

⚙️ Initial JDK 11 Build #

- Failing: test-framework/maven 
! Skipped: integration-tests/container-image/maven-invoker-way integration-tests/kotlin integration-tests/kubernetes/maven-invoker-way and 2 more

📦 test-framework/maven

Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M3:enforce (enforce) on project quarkus-test-maven: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 24, 2021

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building 78c0223

Status Name Step Failures Logs Raw logs
Gradle Tests - JDK 11 Windows Build Failures Logs Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ Gradle Tests - JDK 11 Windows #

- Failing: integration-tests/gradle 

📦 integration-tests/gradle

io.quarkus.gradle.devmode.BasicKotlinApplicationModuleDevModeTest.main line 18 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference, java.util.concurrent.atomic.AtomicReferencejava.lang.String, java.lang.Stringboolean was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:166)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

io.quarkus.gradle.devmode.JandexMultiModuleProjectDevModeTest.main line 21 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference, java.util.concurrent.atomic.AtomicReferencejava.lang.String, java.lang.Stringboolean was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:166)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 27, 2021

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building 8657eb4

Status Name Step Failures Logs Raw logs
Gradle Tests - JDK 11 Windows Build Failures Logs Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ Gradle Tests - JDK 11 Windows #

- Failing: integration-tests/gradle 

📦 integration-tests/gradle

io.quarkus.gradle.devmode.MultiModuleIncludedBuildTest.main line 24 - More details - Source on GitHub

java.lang.AssertionError: 

Expecting actual:

@stuartwdouglas
Copy link
Member Author

This should be ready to merge

@famod
Copy link
Member

famod commented Sep 27, 2021

I gave this a try and it does execute a test from an upstream module, but it fails with an AmbiguousResolutionException:

Ambiguous dependencies for type com.github.famod.modmono_quarkus.core.HelloBean and qualifiers [@Default]
        - java member: com.github.famod.modmono_quarkus.core.testlib.HelloBeanTest#underTest
        - declared on CLASS bean [types=[com.github.famod.modmono_quarkus.core.testlib.HelloBeanTest, java.lang.Object], qualifiers=[@Default, @Any], target=com.github.famod.modmono_quarkus.core.testlib.HelloBeanTest]
        - available beans:
                - CLASS bean [types=[com.github.famod.modmono_quarkus.core.HelloBean, java.lang.Object], qualifiers=[@Default, @Any], target=com.github.famod.modmono_quarkus.core.HelloBean]
                - CLASS bean [types=[com.github.famod.modmono_quarkus.core.HelloBean, java.lang.Object], qualifiers=[@Default, @Any], target=com.github.famod.modmono_quarkus.core.HelloBean]
        at io.quarkus.arc.processor.Beans.resolveInjectionPoint(Beans.java:501)
        at io.quarkus.arc.processor.BeanInfo.init(BeanInfo.java:470)
        at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:254)
        ... 13 more

The reproducer is here: https://github.com/famod/modmono-quarkus/tree/cont-testing-multi

* ./mvnw clean install

* ./mvnw quarkus:dev -f dist/

* resume testing -> fails

PS: Please ignore the mocks profile for now, it's related to #19152.

⬆️ still fails, I'm afraid...

@stuartwdouglas
Copy link
Member Author

@famod oops, looks like something got missed in a rebase, should be fixed now.

@famod
Copy link
Member

famod commented Sep 28, 2021

@stuartwdouglas I'm now getting the same results, regardless of quarkus.test.only-test-application-module being true or false.
Is this expected?

PS: It always includes the test from that other module (core).

@famod
Copy link
Member

famod commented Sep 28, 2021

...but on the plus side there is no exception anymore! 👍

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 28, 2021

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building 3523830

Status Name Step Failures Logs Raw logs
Gradle Tests - JDK 11 Windows Build Failures Logs Raw logs
JVM Tests - JDK 11 Build Failures Logs Raw logs
JVM Tests - JDK 17 Build ⚠️ Check → Logs Raw logs

Full information is available in the Build summary check run.

⚠️ Errors occurred while downloading the build reports. This report is incomplete.

Failures

⚙️ Gradle Tests - JDK 11 Windows #

- Failing: integration-tests/gradle 

📦 integration-tests/gradle

io.quarkus.gradle.devmode.BasicKotlinApplicationModuleDevModeTest.main line 18 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference, java.util.concurrent.atomic.AtomicReferencejava.lang.String, java.lang.Stringboolean was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:166)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

⚙️ JVM Tests - JDK 11 #

- Failing: extensions/hibernate-orm/deployment 
! Skipped: docs extensions/hibernate-envers/deployment extensions/hibernate-reactive/deployment and 84 more

📦 extensions/hibernate-orm/deployment

io.quarkus.hibernate.orm.HibernateHotReloadTestCase.testImportSqlWithContinuousTesting line 89 - More details - Source on GitHub

org.opentest4j.AssertionFailedError: expected: <0> but was: <1>
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
	at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)

Note that continuous testing supports multi-module projects, so tests in modules other than the application can still
be run when files are changed. The modules that are run can be controlled using config as listed below.

This is enabled by default, and can be disabled via `quarkus.test.only-test-application-module=false`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This is enabled by default, and can be disabled via `quarkus.test.only-test-application-module=false`.
This is enabled by default, and can be disabled via `quarkus.test.only-test-application-module=true`.

@famod
Copy link
Member

famod commented Sep 28, 2021

I'm now getting the same results, regardless of quarkus.test.only-test-application-module being true or false. Is this expected?

My bad, I got confused, partially by the changed docs (see #19793 (comment)).

Copy link
Member

@famod famod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Played with it a little, looking good! Can't comment much on the code though, looks like you had to extract/restructure things for the new runner, right?

So LGTM, just that documented property seems inverted.

@aloubyansky
Copy link
Member

Just in case, I tried this change on top of my bootstrap refactoring branch. The Gradle Kotlin test has passed (I guess it may pass in main on the re-run too) but the import sql one failed as well with

2021-09-28T19:30:00.8228268Z [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.785 s <<< FAILURE! - in io.quarkus.hibernate.orm.HibernateHotReloadTestCase
2021-09-28T19:30:00.8239525Z [ERROR] io.quarkus.hibernate.orm.HibernateHotReloadTestCase.testImportSqlWithContinuousTesting  Time elapsed: 7.116 s  <<< FAILURE!
2021-09-28T19:30:00.8245788Z org.opentest4j.AssertionFailedError: expected: <0> but was: <1>
2021-09-28T19:30:00.8252851Z 	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
2021-09-28T19:30:00.8259080Z 	at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
2021-09-28T19:30:00.8269903Z 	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:166)
2021-09-28T19:30:00.8277951Z 	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:161)
2021-09-28T19:30:00.8285761Z 	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:628)
2021-09-28T19:30:00.8296312Z 	at io.quarkus.hibernate.orm.HibernateHotReloadTestCase.testImportSqlWithContinuousTesting(HibernateHotReloadTestCase.java:89)

Continuous testing will now run tests from all modules, with the option
to configure the old behaviour.

Fixes quarkusio#19362
@stuartwdouglas
Copy link
Member Author

Hopefully all good now.

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 29, 2021

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building 7e804f1

Status Name Step Failures Logs Raw logs
JVM Tests - JDK 11 Build ⚠️ Check → Logs Raw logs
JVM Tests - JDK 17 Build ⚠️ Check → Logs Raw logs

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 29, 2021

Failing Jobs - Building 7e804f1

Status Name Step Failures Logs Raw logs
JVM Tests - JDK 11 Build ⚠️ Check → Logs Raw logs
JVM Tests - JDK 11 Windows Build Failures Logs Raw logs
✔️ JVM Tests - JDK 17

Full information is available in the Build summary check run.

Failures

⚙️ JVM Tests - JDK 11 Windows #

- Failing: extensions/grpc/deployment 
! Skipped: docs integration-tests/devmode integration-tests/grpc-health and 9 more

📦 extensions/grpc/deployment

io.quarkus.grpc.devconsole.DevConsoleUnaryMethodTest.websocketTest line 57 - More details - Source on GitHub

java.lang.AssertionError: 

Expected size: 2 but was: 1 in:

Copy link
Member

@famod famod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, also retested it with my small sample project.

Don't know about that one test failure, looks unrelated.

The mvn part in the other job was actually successful but then the run timed out.

@aloubyansky
Copy link
Member

@stuartwdouglas do you want this to be backported to 2.3? It's too late for the 2.3.0.Final though.

@aloubyansky aloubyansky merged commit 0ffe227 into quarkusio:main Sep 29, 2021
@quarkus-bot quarkus-bot bot added this to the 2.4 - main milestone Sep 29, 2021
@stuartwdouglas
Copy link
Member Author

stuartwdouglas commented Sep 29, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants