[discussion] run mvnd in docker

[dcaillia]

I have a docker image that contains a regular maven installation, jdk and other build tools.

I use it to run mvn (volume-mounting only my project sources and .m2) both during local development and on the build-server.

I like this because it makes sure my local builds are built exactly like on the build-server (which uses the same "build image"). The Dockerfile of my build image is also important as it completely defines the build environment and tools.

I'm wondering how to get the benefits of mvnd while sticking to that "build image" concept (building in a docker container).

A couple of questions pop up:

• i'ld need to keep the build container alive across builds to enjoy the mvnd speedup
• so to compile n different projects i'ld have to keep n build-containers running
• unless if i'ld mount my whole data dir with all projects - which is something i'ld rather avoid, as by only mounting project x sources, i'm sure the build of project x is not relying on resources outside of project x

So, with mvnd on the host all is awesome - assuming you build (call mvnd) on the host too (and not in some docker container).

I'm wondering if there's a way to have 1 single long-running mvnd container (running all the time), and then when i launch my own build docker container (mounting project x sources), it somehow connects to the long-running mvnd, such that my build container talks to that long-running mvnd to get the mvn job done faster.

[ppalaga]

I'm wondering if there's a way to have 1 single long-running mvnd container (running all the time), and then when i launch my own build docker container (mounting project x sources), it somehow connects to the long-running mvnd, such that my build container talks to that long-running mvnd to get the mvn job done faster.

We currently have only a local, file-based daemon registry, so discovering a daemon running on a different host is not possible. Having some general service discovery mechanism sounds interesting. Have you thought of some specific protocol/implementation?

[gnodet]

Also, if the mvnd daemon is on the host, that would completely defeat the purpose of using docker images to make sure the "local builds are built exactly like on the build-server", because it would use the host environment and not the docker image one. Am I missing something ?

[dcaillia]

The daemon could be inside my build-container (that has other build-tooling i need as part of the mvn build). I just need to:

(1) make sure it never shuts down, because it's a daemon
(2) have a way to make my source code (of the project i need to build) visible into that long-running container when i want to call its mvn, maybe this is feasible with a softlink

[ppalaga]

Also, if the mvnd daemon is on the host, that would completely defeat the purpose of using docker images to make sure the "local builds are built exactly like on the build-server", because it would use the host environment and not the docker image one.

I think @dcaillia could run both the daemon and the mvnd client inside the container. The daemon would be the container’s primary process (PID 1) and mvnd could be invoked via docker exec. In that way the consistency of the env across machines could be guaranteed.

[ppalaga]

(1) make sure it never shuts down, because it's a daemon

There is currently no easy way to start the daemon process directly. But it is not that hard to put together the necessary command. You could e.g. start the daemon via mvnd -v and then find the command that the mvnd client used to start the daemon via ps -e -f | grep mvnd. On my machine, I see this:

/home/ppalaga/.sdkman/candidates/java/21.2.0.r16-grl/bin/java -classpath /home/ppalaga/.sdkman/candidates/mvnd/0.6.0/mvn/lib/ext/mvnd-common-0.6.0.jar:/home/ppalaga/.sdkman/candidates/mvnd/0.6.0/mvn/lib/ext/mvnd-agent-0.6.0.jar -javaagent:/home/ppalaga/.sdkman/candidates/mvnd/0.6.0/mvn/lib/ext/mvnd-agent-0.6.0.jar --add-opens jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.jvm=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED -Xms128M -Xmx6G -Dmvnd.home=/home/ppalaga/.sdkman/candidates/mvnd/0.6.0 -Dmvnd.java.home=/home/ppalaga/.sdkman/candidates/java/21.2.0.r16-grl -Dlogback.configurationFile=/home/ppalaga/.sdkman/candidates/mvnd/0.6.0/conf/logback.xml -Dmvnd.id=ffe0de8d -Dmvnd.daemonStorage=/home/ppalaga/.m2/mvnd/registry/0.6.0 -Dmvnd.registry=/home/ppalaga/.m2/mvnd/registry/0.6.0/registry.bin -Dmvnd.socketFamily=inet -Djdk.java.options= --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.nio.fs=ALL-UNNAMED -Dmvnd.noDaemon=false -Dmvnd.debug=false -Dmvnd.idleTimeout=3h -Dmvnd.keepAlive=100ms -Dmvnd.extClasspath= -Dmvnd.coreExtensions= -Dmvnd.minHeapSize=128M -Dmvnd.maxHeapSize=6G -Dmvnd.jvmArgs=--add-opens jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.jvm=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-opens jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED -Dmvnd.enableAssertions=false -Dmvnd.expirationCheckDelay=10s -Dmvnd.duplicateDaemonGracePeriod=10s -Dmvnd.socketFamily=inet org.mvndaemon.mvnd.common.MavenDaemon

You could then put a similar command to your dockerfile.

(2) have a way to make my source code (of the project i need to build) visible into that long-running container when i want to call its mvn, maybe this is feasible with a softlink

You could perhaps mount your whole home folder or whichever folder containing all your projects? Then you could do something like docker exec CONTAINER cd /mnt/rel/path/to/my/project && mvnd <goals>

[NoSugarCoffee]

So mvnd has less potential on CI, because can‘t benifit from daemon, more suitable for local build and run proeject again and again, is it right?

[ppalaga]

So mvnd has less potential on CI, because can‘t benifit from daemon, more suitable for local build and run proeject again and again, is it right?

Indeed, mvnd was primarily designed as an interactive tool run by humans.
It may still make sense to experiment with other use cases.

[chicobento]

On our CI pipelines, on a single build we launch mvn multiple times across jenkins stages, eg: lint, test, integration test, javadocs and so on, so I believe that we would benefit from mvnd on CI even though in that scenario, the definition of long-running daemon might be not so long.
However, we have a problem similar to the @dcaillia , since the mvn calls are made from containers launched by the jenkins host, so it would be really nice if we had a way of redirecting the mvnd 'client' to reuse a daemon launched by the host (or another container).
My initial idea is to perhaps spawn a mvnd 'daemon' container at the beginning of the pipeline and then be able to redirect the other mvnd 'client' container calls to reuse the daemon container.
Something like:

jenkins pipeline
  - container 0: launch mvnd daemon
  - container 1: mvnd clean
  - container 2: mvnd package install
  - container 3: mvnd test
  - container 4: mvnd dependency:analyze
  - ... and many other mvn plugins we run on our builds

Where container 1-4 calls would reuse the daemon launched by container 0.

Would that be possible ? Is there other strategy that we could benefit from mvnd given our 'restriction' of launching mvn plugins on separate containers ?

[ppalaga]

The existing network protocol between mvnd client and the daemon would work without any change even if daemon runs on a different host. What is missing is a discovery mechanism for the clients to find daemon(s) running on other hosts. Currently there is only a local file-based deamon registry implementation: the client looks into that file for existing daemons and the daemons write their busy/idle state and port there.

So we need some new mechanism.
Perhaps a command line option and/or an env. var for passing the daemon host:port to the client would be enough initially?

[chicobento]

Sounds perfect. That was exactly what I was hoping for 👍🏻