Support for hidden dependencies

[ppalaga]

Out of a sudden, mvnd started building in parallel. You perhaps changed the defaults - no matter, I like it. Not sure why exactly this happens, but my spontaneous theory is that this is not a mistake of mnvd, but rather a mistake of how the deps are declared in Camel Quarkus. In particular, the itests do not directly or indirectly depend on the -deployment modules and mnvd is free to assume that the -deployment modules do not need to get built before the itests.

I wonder how this could be fixed in the Camel Quarkus source tree? Adding an explicit -deployment dependency to the itest project would perhaps solve the problem, but I'd find that highly unidiomatic. Do you happen to have a better idea how to fix that?

[INFO] Reactor Summary for Camel Quarkus 0.2.1-SNAPSHOT:
[INFO] 
[INFO] Camel Quarkus ...................................... SUCCESS [  0.016 s]
[INFO] Camel Quarkus :: BOM ............................... SUCCESS [  0.022 s]
[INFO] Camel Quarkus :: Build Parent ...................... SUCCESS [  0.018 s]
[INFO] Camel Quarkus :: Build Parent ...................... SUCCESS [  0.070 s]
[INFO] Camel Quarkus :: Core .............................. SUCCESS [  0.061 s]
[INFO] Camel Quarkus :: Core :: Runtime ................... SUCCESS [  1.091 s]
[INFO] Camel Quarkus :: Core :: Deployment ................ SUCCESS [  2.169 s]
[INFO] Camel Quarkus :: Support :: Jetty .................. SUCCESS [  0.121 s]
[INFO] Camel Quarkus :: Support :: Jetty :: Runtime ....... SUCCESS [  0.720 s]
[INFO] Camel Quarkus :: Support :: Jetty :: Deployment .... SUCCESS [  1.788 s]
[INFO] Camel Quarkus :: Support :: XStream ................ SUCCESS [  0.102 s]
[INFO] Camel Quarkus :: Support :: XStream :: Runtime ..... SUCCESS [  0.713 s]
[INFO] Camel Quarkus :: Support :: XStream :: Deployment .. SUCCESS [  1.865 s]
[INFO] Camel Quarkus :: Extensions ........................ SUCCESS [  0.147 s]
[INFO] Camel Quarkus :: Support ........................... SUCCESS [  0.023 s]
[INFO] Camel Quarkus :: XML :: Common ..................... SUCCESS [  0.118 s]
[INFO] Camel Quarkus :: XML :: Common :: Runtime .......... SUCCESS [  0.718 s]
[INFO] Camel Quarkus :: XML :: Common :: Deployment ....... SUCCESS [  1.135 s]
[INFO] Camel Quarkus :: Core :: Cloud ..................... SUCCESS [  0.109 s]
[INFO] Camel Quarkus :: Core :: Cloud :: Runtime .......... SUCCESS [  1.730 s]
[INFO] Camel Quarkus :: Core :: Cloud :: Deployment ....... SKIPPED
[INFO] Camel Quarkus :: HTTP :: Common .................... SUCCESS [  0.106 s]
[INFO] Camel Quarkus :: HTTP :: Common :: Runtime ......... SUCCESS [  0.709 s]
[INFO] Camel Quarkus :: HTTP :: Common :: Deployment ...... SUCCESS [  0.625 s]
[INFO] Camel Quarkus :: Netty HTTP ........................ SUCCESS [  0.172 s]
[INFO] Camel Quarkus :: Netty HTTP :: Runtime ............. SUCCESS [  1.676 s]
[INFO] Camel Quarkus :: Netty HTTP :: Deployment .......... SUCCESS [  1.560 s]
[INFO] Camel Quarkus :: Infinispan ........................ SUCCESS [  0.091 s]
[INFO] Camel Quarkus :: Infinispan :: Runtime ............. SUCCESS [  2.275 s]
[INFO] Camel Quarkus :: Infinispan :: Deployment .......... SUCCESS [  2.239 s]
[INFO] Camel Quarkus :: AWS ............................... SUCCESS [  0.101 s]
[INFO] Camel Quarkus :: AWS S3 :: Runtime ................. SUCCESS [  0.878 s]
[INFO] Camel Quarkus :: AWS S3 :: Deployment .............. SKIPPED
[INFO] Camel Quarkus :: AWS SNS ........................... SUCCESS [  0.139 s]
[INFO] Camel Quarkus :: AWS SNS :: Runtime ................ SUCCESS [  0.946 s]
[INFO] Camel Quarkus :: AWS SNS :: Deployment ............. SKIPPED
[INFO] Camel Quarkus :: AWS SQS ........................... SUCCESS [  0.137 s]
[INFO] Camel Quarkus :: AWS SQS :: Runtime ................ SUCCESS [  1.350 s]
[INFO] Camel Quarkus :: AWS SQS :: Deployment ............. SKIPPED
[INFO] Camel Quarkus :: AWS ............................... SUCCESS [  0.109 s]
[INFO] Camel Quarkus :: AWS EKS :: Runtime ................ SUCCESS [  0.854 s]
[INFO] Camel Quarkus :: AWS EKS :: Deployment ............. SKIPPED
[INFO] Camel Quarkus :: JDBC .............................. SUCCESS [  0.101 s]
[INFO] Camel Quarkus :: JDBC :: Runtime ................... SUCCESS [  0.647 s]
[INFO] Camel Quarkus :: JDBC :: Deployment ................ SUCCESS [  1.378 s]
[INFO] Camel Quarkus :: Mail .............................. SUCCESS [  0.140 s]
[INFO] Camel Quarkus :: Mail :: Runtime ................... SUCCESS [  0.536 s]
[INFO] Camel Quarkus :: Mail :: Deployment ................ SUCCESS [  1.425 s]
[INFO] Camel Quarkus :: Salesforce ........................ SUCCESS [  0.096 s]
[INFO] Camel Quarkus :: Salesforce :: Runtime ............. SUCCESS [  1.164 s]
[INFO] Camel Quarkus :: Salesforce :: Deployment .......... SUCCESS [  1.991 s]
[INFO] Camel Quarkus :: Servlet ........................... SUCCESS [  0.134 s]
[INFO] Camel Quarkus :: Servlet :: Runtime ................ SUCCESS [  2.093 s]
[INFO] Camel Quarkus :: Servlet :: Deployment ............. SUCCESS [  2.904 s]
[INFO] Camel Quarkus :: Log ............................... SUCCESS [  0.142 s]
[INFO] Camel Quarkus :: Log :: Runtime .................... SUCCESS [  0.593 s]
[INFO] Camel Quarkus :: Log :: Deployment ................. SUCCESS [  1.869 s]
[INFO] Camel Quarkus :: Direct ............................ SUCCESS [  0.089 s]
[INFO] Camel Quarkus :: Direct :: Runtime ................. SUCCESS [  0.467 s]
[INFO] Camel Quarkus :: Direct :: Deployment .............. SKIPPED
[INFO] Camel Quarkus :: Rest .............................. SUCCESS [  0.122 s]
[INFO] Camel Quarkus :: Rest :: Runtime ................... SUCCESS [  0.805 s]
[INFO] Camel Quarkus :: Rest :: Deployment ................ SUCCESS [  1.475 s]
[INFO] Camel Quarkus :: Timer ............................. SUCCESS [  0.114 s]
[INFO] Camel Quarkus :: Timer :: Runtime .................. SUCCESS [  0.473 s]
[INFO] Camel Quarkus :: Timer :: Deployment ............... SUCCESS [  1.428 s]
[INFO] Camel Quarkus :: Twitter ........................... SUCCESS [  0.134 s]
[INFO] Camel Quarkus :: Twitter :: Runtime ................ SUCCESS [  0.588 s]
[INFO] Camel Quarkus :: Twitter :: Deployment ............. SUCCESS [  1.417 s]
[INFO] Camel Quarkus :: Bean .............................. SUCCESS [  0.175 s]
[INFO] Camel Quarkus :: Bean :: Runtime ................... SUCCESS [  1.232 s]
[INFO] Camel Quarkus :: Bean :: Deployment ................ SKIPPED
[INFO] Camel Quarkus :: ZipFile ........................... SUCCESS [  0.150 s]
[INFO] Camel Quarkus :: ZipFile :: Runtime ................ SUCCESS [  0.729 s]
[INFO] Camel Quarkus :: ZipFile :: Deployment ............. SUCCESS [  1.477 s]
[INFO] Camel Quarkus :: CSV ............................... SUCCESS [  0.111 s]
[INFO] Camel Quarkus :: CSV :: Runtime .................... SUCCESS [  1.012 s]
[INFO] Camel Quarkus :: CSV :: Deployment ................. SKIPPED
[INFO] Camel Quarkus :: MicroProfile Metrics .............. SUCCESS [  0.106 s]
[INFO] Camel Quarkus :: MicroProfile Metrics :: Runtime ... SUCCESS [  1.960 s]
[INFO] Camel Quarkus :: MicroProfile Metrics :: Deployment  SUCCESS [  1.797 s]
[INFO] Camel Quarkus :: Paho .............................. SUCCESS [  0.096 s]
[INFO] Camel Quarkus :: Paho :: Runtime ................... SUCCESS [  0.808 s]
[INFO] Camel Quarkus :: Paho :: Deployment ................ SUCCESS [  1.663 s]
[INFO] Camel Quarkus :: Platform HTTP ..................... SUCCESS [  0.142 s]
[INFO] Camel Quarkus :: Platform HTTP :: Component ........ SUCCESS [  0.555 s]
[INFO] Camel Quarkus :: Platform HTTP :: Runtime .......... SUCCESS [  1.969 s]
[INFO] Camel Quarkus :: Platform HTTP :: Deployment ....... SUCCESS [  1.962 s]
[INFO] Camel Quarkus :: Integration Tests ................. SUCCESS [  0.311 s]
[INFO] Camel Quarkus :: Integration Tests :: Core :: Parent SUCCESS [  0.330 s]
[INFO] Camel Quarkus :: Integration Tests :: Core :: Extension :: Runtime SUCCESS [  1.353 s]
[INFO] Camel Quarkus :: Integration Tests :: Core :: Extension :: Deployment SUCCESS [  1.592 s]
[INFO] Camel Quarkus :: Integration Tests :: Core :: Tests  SUCCESS [  5.266 s]
[INFO] Camel Quarkus :: Integration Tests :: Core :: Impl . SUCCESS [  2.080 s]
[INFO] Camel Quarkus :: Integration Tests :: Core CDI ..... SUCCESS [  4.410 s]
[INFO] Camel Quarkus :: Integration Tests :: Netty Http ... SUCCESS [  3.386 s]
[INFO] Camel Quarkus :: Integration Tests :: AWS .......... FAILURE [  1.822 s]
[INFO] Camel Quarkus :: Integration Tests :: Bean ......... SUCCESS [  3.359 s]
[INFO] Camel Quarkus :: Integration Tests :: CSV .......... SUCCESS [  5.334 s]
[INFO] Camel Quarkus :: Integration Tests :: Infinispan ... SUCCESS [  5.739 s]
[INFO] Camel Quarkus :: Integration Tests :: JDBC ......... SUCCESS [  5.748 s]
[INFO] Camel Quarkus :: Integration Tests :: Mail ......... SUCCESS [  4.280 s]
[INFO] Camel Quarkus :: Integration Tests :: Salesforce ... SUCCESS [  5.433 s]
[INFO] Camel Quarkus :: Integration Tests :: Servlet ...... SUCCESS [  3.002 s]
[INFO] Camel Quarkus :: Integration Tests :: Twitter ...... SUCCESS [  4.669 s]
[INFO] Camel Quarkus :: Integration Tests :: zipfile ...... SUCCESS [  4.450 s]
[INFO] Camel Quarkus :: Integration Tests :: microprofile-metrics SUCCESS [  4.625 s]
[INFO] Camel Quarkus :: Integration Tests :: Paho ......... SUCCESS [  4.752 s]
[INFO] Camel Quarkus :: Integration Tests :: platform-http  SUCCESS [  3.292 s]
[INFO] Camel Quarkus :: Documentation ..................... SUCCESS [  0.005 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  18.920 s (Wall Clock)
[INFO] Finished at: 2019-10-01T17:47:14+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:0.23.1:build (default) on project camel-quarkus-integration-test-aws: Failed to build a runnable JAR: Failed to build a runner jar: Failed to augment application classes: JAR entry META-INF/quarkus-build-steps.list not found in /home/ppalaga/.m2/repository/org/apache/camel/quarkus/camel-quarkus-aws-s3-deployment/0.2.1-SNAPSHOT/camel-quarkus-aws-s3-deployment-0.2.1-SNAPSHOT.jar -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :camel-quarkus-integration-test-aws

[ppalaga]

Let me try if the smart builder is considering the plugin dependencies when building the dependency tree. Adding -deployment modules as dependencies of some plugin could be a viable workaround.

[ppalaga]

I thought using the code in https://github.com/gnodet/mvnd/pull/13 and building with -T1 would solve the problem, but no, it looks like the ordering of the modules is non-deterministic and build succeeds only sometimes. Interesting.

[ppalaga]

-T1 -bsinglethreaded seems to work consistently as a workaround.

[gnodet]

Yes, that's one of the problem with a parallel build: all dependencies must be explicitely defined, as we can't rely on the order in which modules are defined.

[ppalaga]

Yes, but Quarkus extensions are a bit special in this respect. The deployment module is not declared as a dependency by design. It does not suit any of the traditional Maven scopes. It is actually only pulled by the Quarkus plugin to be run at built time. So declaring it as a dependency of the Quarkus plugin seems be closest to what it really is (although the plugin does not require that). I will try whether the smart builder will honor the declaration and schedule the deployment module before the itest.

But anyway shouldn't mvnd offer an official way to adjust the build order? I can imagine many projects will have similar problems. E.g. a system property? Something like

<groupId>my-org</groupId>
<artifactId>my-itest</artifactId>
...
<properties>
  <mvnd.schedule-after>my-org:ext1-deployment,my-org:ext2-deployment</mvnd.schedule-after>
</properties>

[ppalaga]

mvnd.schedule-after is stupid. mvnd.schedule.after would be more consistent.

[ppalaga]

Adding the deployment deps as plugin deps seems to solve the scheduling problem, but the effective result seems to be a corrupted local repo. Thinking so because the stock mnv is throwing the same errors in java.io that go away (with the stock maven) when I delete the broken jars from the local repo.

Steps to reproduce should be

1. Checkout this branch https://github.com/ppalaga/camel-quarkus/commits/191003-hidden-deps
2. Remove all quarkus 2.23.2 artifacts from your local repo, e.g. using find . -type d -name 0.23.2 -exec rm -Rf {} \;
3. mvnd clean install -DskipTests

[ppalaga]

The failures in the previous comment are caused by the fact that the Quarkus plugin does not like to have the deployment modules in its class path. So I tried to add the deployment modules as deps of another plugin see here https://github.com/ppalaga/camel-quarkus/commits/191003-hidden-deps.1 and it works! mvnd clean install -DskipTests took ~26 sec instead of ~57 sec with the stock Maven!

[ppalaga]

So adding the hidden deps to some plugin is a valid workaround, but the question is still open whether mvnd should offer an official way to reach the goal without having to add a plugin.

[gnodet]

So adding the hidden deps to some plugin is a valid workaround, but the question is still open whether mvnd should offer an official way to reach the goal without having to add a plugin.

I agree we should provide a way to add dependencies for ordering purposes. Using properties definitely looks like a good way. However, I'd rather move such information in the parent pom rather than dispatched in the various modules. This would also enable to use wildcards to define those additional rules. This would also benefit camel which also has similar problems.

[ppalaga]

I'd rather move such information in the parent pom

I wonder what would it look like?

Would those be honored also when not building from the root dir?

Having it in the dependent module seems to have the following advantages:

• No need to write the dependent module, because it is implicitly clear - it is "this" module
• More robust when renaming/removing the dependent module

[gnodet]

I've added support for top level rules for now. The syntax is the following:

<rules> ::= rule [ <rule-sep> <rule> ]+
<rule-sep> ::= '\n' | ';'
<rule> ::= <projects> <sep> <projects>
<sep> ::= ' before ' | ' -> '
<projects> ::= <project> [ <project-sep>  <project> ]+
<project-sep> ::= ',' | 'and'
<project> ::= [ <groupId> ':' ]? <artifactId>

The groupId and artifactId supports the * wildcard matcher. If the groupId part is omitted, * is implied.

The rules have to be defined as the mvnd.builder.rules maven property.

[gnodet]

A simple top level rule is

org.apache.camel.quarkus:*-deployment before org.apache.camel.quarkus.tests:*

[gnodet]

We could also add support for non top-level rules where the right part of the rule would be implied to the current groupId:artifactId.

[gnodet]

Just added support for local rules defined using mvnd.builder.rule where the right part is implied.