Maven commands fail randomly in CI/CD workflow #422

tschaffter · 2022-07-10T02:21:55Z

Latest error thrown:

Error:  Error executing Maven.
java.lang.NullPointerException
	at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
	at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
	at java.base/java.util.Properties.put(Properties.java:1301)
	at java.base/java.util.Properties.setProperty(Properties.java:229)
	at org.apache.maven.cli.MavenCli.populateProperties(MavenCli.java:1656)
	at org.apache.maven.cli.MavenCli.properties(MavenCli.java:612)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:282)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:47)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:156)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:72)
> nx run shared-java-challenge-util:build
Executing command: ./mvnw package 
Failed to execute command: ./mvnw package 
Error: Command failed: ./mvnw package 
    at checkExecSyncError (node:child_process:828:11)
    at execSync (node:child_process:899:15)
    at runBuilderCommand (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/common/src/lib/core/jvm/utils.js:19:38)
    at runBootPluginCommand (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/utils/boot-utils.js:15:43)
    at /__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:10:62
    at Generator.next (<anonymous>)
    at /__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:117:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:113:16)
    at buildExecutor (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:8:20)
Error
    at /__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:12:19
    at Generator.next (<anonymous>)
    at /__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:117:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:113:16)
    at buildExecutor (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:8:20)
    at /__w/challenge-registry/challenge-registry/node_modules/@nrwl/tao/src/commands/run.js:147:23
    at Generator.next (<anonymous>)
    at /__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:117:75
    at new Promise (<anonymous>)
    at __awaiter (/__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:113:16)
    at runExecutorInternal (/__w/challenge-registry/challenge-registry/node_modules/@nrwl/tao/src/commands/run.js:127:34)
    at Object.<anonymous> (/__w/challenge-registry/challenge-registry/node_modules/@nrwl/tao/src/commands/run.js:219:54)
    at Generator.next (<anonymous>)
    at /__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:117:75
    at new Promise (<anonymous>)

tschaffter · 2022-07-10T02:42:27Z

Is the wrapper the issue? What if I call directly the Maven command after adding it to the devcontainer?
Is it because of concurrent Maven calls executed by Nx?

tschaffter · 2022-08-01T17:12:24Z

Reported to tinesoft/nxrocks#123

tschaffter · 2022-08-01T17:13:10Z

Added to Spring 22.08.

tschaffter · 2022-08-05T14:29:35Z

Another instance of the issue at hand that was solved by a second run:

 Warning: JAVA_HOME environment variable is not set.
Warning: JAVA_HOME environment variable is not set.
Warning: JAVA_HOME environment variable is not set.
Could not set executable permissions for: /root/.m2/wrapper/dists/apache-maven-3.8.5-bin/5i5jha092a3i37g0paqnfr15e0/apache-maven-3.8.5/bin/mvn
Please do this manually if you want to use Maven.
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:47)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:156)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:72)
Caused by: java.lang.NoClassDefFoundError: org/apache/maven/exception/ExceptionHandler
	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3402)
	at java.base/java.lang.Class.getMethodsRecursive(Class.java:3543)
	at java.base/java.lang.Class.getMethod0(Class.java:3529)
	at java.base/java.lang.Class.getMethod(Class.java:2225)
	at org.codehaus.plexus.classworlds.launcher.Launcher.getEnhancedMainMethod(Launcher.java:168)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:261)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
	... 7 more
Caused by: java.lang.ClassNotFoundException: org.apache.maven.exception.ExceptionHandler
	at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
	... 17 more

> nx run challenge-config-service:build

Executing command: ./mvnw package 
Failed to execute command: ./mvnw package 
Error: Command failed: ./mvnw package 
    at checkExecSyncError (node:child_process:828:11)
    at execSync (node:child_process:899:15)
    at runBuilderCommand (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/common/src/lib/core/jvm/utils.js:19:38)
    at runBootPluginCommand (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/utils/boot-utils.js:15:43)
    at /__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:10:62
    at Generator.next (<anonymous>)
    at /__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:118:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:114:16)
    at buildExecutor (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:8:20)
    at /__w/challenge-registry/challenge-registry/node_modules/nx/src/command-line/run.js:101:23
    at Generator.next (<anonymous>)
    at /__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:118:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:114:16)

tschaffter · 2022-08-05T15:49:52Z

Another occurrence:

Build at: 2022-08-05T15:01:36.603Z - Hash: ee193ebcc10c8e1c - Time: 36297ms
Warning: JAVA_HOME environment variable is not set.
Exception in thread "main" java.util.zip.ZipException: zip file is empty
	at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1598)
	at java.base/java.util.zip.ZipFile$Source.findEND(ZipFile.java:1382)
	at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1477)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1315)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1277)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:709)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:243)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:172)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:186)
	at org.apache.maven.wrapper.Installer.unzip(Installer.java:209)
	at org.apache.maven.wrapper.Installer.createDist(Installer.java:102)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:155)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:72)

 > nx run challenge-config-service:build

Executing command: ./mvnw package 
Failed to execute command: ./mvnw package 
Error: Command failed: ./mvnw package 
    at checkExecSyncError (node:child_process:828:11)
    at execSync (node:child_process:899:15)
    at runBuilderCommand (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/common/src/lib/core/jvm/utils.js:19:38)
    at runBootPluginCommand (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/utils/boot-utils.js:15:43)
    at /__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:10:62
    at Generator.next (<anonymous>)
    at /__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:118:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:114:16)
    at buildExecutor (/__w/challenge-registry/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:8:20)
    at /__w/challenge-registry/challenge-registry/node_modules/nx/src/command-line/run.js:101:23
    at Generator.next (<anonymous>)
    at /__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:118:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/__w/challenge-registry/challenge-registry/node_modules/tslib/tslib.js:114:16)

tschaffter · 2022-08-05T16:26:49Z

Troubleshooting

The target that fails is usually build and test. Mainly build in the current series of errors. An example of project-target that fails is challenge-config-service:build. I tried to run many build tasks for different Maven projects with, which is a variant of the build instruction in the CI workflow (here with run-many --projects ... instead of affected).

yarn nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=build --parallel --max-parallel=3 --skip-nx-cache

I could not reproduce the issue this way.

I'm now trying to reproduce the error using act in order to mirror more closely the environment that run the CI workflow on GH.

$ ~/bin/act push -l
Stage  Job ID        Job name      Workflow name              Workflow file                 Events           
0      devcontainer  devcontainer  Build Dev Container Image  build-devcontainer-image.yml  push,pull_request
0      main          main          CI                         ci.yml                        push,pull_request
0      pr            pr            CI                         ci.yml                        push,pull_request
0      svgs          SVGs          Optimize SVGs              svgo.yml                      push,pull_request

Trying to run the main job using a "medium" Docker image:

$ ~/bin/act push -j main
[CI/main] Cannot parse container options: --user root
[CI/main] 🚀  Start image=sagebionetworks/challenge-devcontainer:bf457d84
[CI/main]   🐳  docker pull image=sagebionetworks/challenge-devcontainer:bf457d84 platform= username= forcePull=false
[CI/main]   🐳  docker create image=sagebionetworks/challenge-devcontainer:bf457d84 platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[CI/main]   🐳  docker run image=sagebionetworks/challenge-devcontainer:bf457d84 platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[CI/main]   🐳  docker exec cmd=[chown -R 1000:1001 /workspaces/challenge-registry] user=0 workdir=
[CI/main]   ☁  git clone 'https://github.com/nrwl/nx-set-shas' # ref=v2
[CI/main] ⭐ Run Pre Derive appropriate SHAs for base and head for `nx affected` commands
[CI/main]   ✅  Success - Pre Derive appropriate SHAs for base and head for `nx affected` commands
[CI/main]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/main]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/main]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/main]   ☁  git clone 'https://github.com/coverallsapp/github-action' # ref=master
[CI/main] ⭐ Run Main Checkout [main]
[CI/main]   🐳  docker cp src=/workspaces/challenge-registry/. dst=/workspaces/challenge-registry
[CI/main] close /tmp/act3654623485: file already closed
[CI/main]   🐳  docker exec cmd=[chown -R 1000:1001 /workspaces/challenge-registry] user=0 workdir=
[CI/main]   ✅  Success - Main Checkout [main]
[CI/main] ⭐ Run Main Derive appropriate SHAs for base and head for `nx affected` commands
[CI/main]   🐳  docker cp src=/home/vscode/.cache/act/nrwl-nx-set-shas@v2/ dst=/var/run/act/actions/nrwl-nx-set-shas@v2/
[CI/main] close /tmp/act1025874977: file already closed
[CI/main]   🐳  docker exec cmd=[chown -R 1000:1001 /var/run/act/actions/nrwl-nx-set-shas@v2/] user=0 workdir=
[CI/main] ⭐ Run Main Set base and head SHAs used for nx affected
[CI/main]   🐳  docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/1-composite-setSHAs.sh] user= workdir=
| 
| WARNING: Working directory 'undefined' doesn't exist.
| (node:47) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
| (Use `node --trace-deprecation ...` to show where the warning was created)
[CI/main]   ❗  ::error::Bad credentials
[CI/main]   ❌  Failure - Main Set base and head SHAs used for nx affected
[CI/main] exitcode '1': failure
[CI/main]   ⚙  ::set-output:: head=
[CI/main]   ⚙  ::set-output:: noPreviousBuild=
[CI/main]   ⚙  ::set-output:: base=
[CI/main]   ❌  Failure - Main Derive appropriate SHAs for base and head for `nx affected` commands
[CI/main] exitcode '1': failure
[CI/main] ⭐ Run Post Derive appropriate SHAs for base and head for `nx affected` commands
[CI/main]   🐳  docker cp src=/home/vscode/.cache/act/nrwl-nx-set-shas@v2/ dst=/var/run/act/actions/nrwl-nx-set-shas@v2/
[CI/main] close /tmp/act600441564: file already closed
[CI/main]   🐳  docker exec cmd=[chown -R 1000:1001 /var/run/act/actions/nrwl-nx-set-shas@v2/] user=0 workdir=
[CI/main]   ✅  Success - Post Derive appropriate SHAs for base and head for `nx affected` commands
[CI/main] 🏁  Job failed
Error: Job 'main' failed

Using the pr job which is also affected by the issue being investigated:

$ ~/bin/act pull_request -j pr
[CI/pr] Cannot parse container options: --user root
[CI/pr] 🚀  Start image=sagebionetworks/challenge-devcontainer:bf457d84
[CI/pr]   🐳  docker pull image=sagebionetworks/challenge-devcontainer:bf457d84 platform= username= forcePull=false
[CI/pr]   🐳  docker create image=sagebionetworks/challenge-devcontainer:bf457d84 platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[CI/pr]   🐳  docker run image=sagebionetworks/challenge-devcontainer:bf457d84 platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[CI/pr]   🐳  docker exec cmd=[chown -R 1000:1001 /workspaces/challenge-registry] user=0 workdir=
[CI/pr]   ☁  git clone 'https://github.com/actions/checkout' # ref=v2
[CI/pr]   ☁  git clone 'https://github.com/nrwl/nx-set-shas' # ref=v2
[CI/pr] ⭐ Run Pre Derive appropriate SHAs for base and head for `nx affected` commands
[CI/pr]   ✅  Success - Pre Derive appropriate SHAs for base and head for `nx affected` commands
[CI/pr]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/pr]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/pr]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/pr] ⭐ Run Main actions/checkout@v2
[CI/pr]   🐳  docker cp src=/home/vscode/.cache/act/actions-checkout@v2/ dst=/var/run/act/actions/actions-checkout@v2/
[CI/pr] close /tmp/act1241934835: file already closed
[CI/pr]   🐳  docker exec cmd=[chown -R 1000:1001 /var/run/act/actions/actions-checkout@v2/] user=0 workdir=
[CI/pr]   🐳  docker exec cmd=[node /var/run/act/actions/actions-checkout@v2/dist/index.js] user= workdir=
[CI/pr]   💾  ::save-state name=isPost::true
[CI/pr]   💬  ::debug::GITHUB_WORKSPACE = '/workspaces/challenge-registry'
[CI/pr]   💬  ::debug::qualified repository = 'tschaffter/challenge-registry'
[CI/pr]   💬  ::debug::ref = 'refs/pull/%25!f(<nil>)/merge'
[CI/pr]   💬  ::debug::commit = '6bfdac5663a60f40724fa92da6bd4a97ada430a1'
[CI/pr]   💬  ::debug::clean = true
[CI/pr]   💬  ::debug::fetch depth = 0
[CI/pr]   💬  ::debug::lfs = false
[CI/pr]   💬  ::debug::submodules = false
[CI/pr]   💬  ::debug::recursive submodules = false
[CI/pr]   ❗  ::error::Input required and not supplied: token
[CI/pr]   ❌  Failure - Main actions/checkout@v2
[CI/pr] exitcode '1': failure
[CI/pr] ⭐ Run Post actions/checkout@v2
[CI/pr]   🐳  docker exec cmd=[node /var/run/act/actions/actions-checkout@v2/dist/index.js] user= workdir=
[CI/pr]   ✅  Success - Post actions/checkout@v2
[CI/pr] 🏁  Job failed
Error: Job 'pr' failed

I thought that the issue was because I'm running act on my fork and because GH does not provide the GH token. But trying using the remote upstream does not solve the issue:

$ ~/bin/act pull_request -j pr --remote-name upstream
[CI/pr] Cannot parse container options: --user root
[CI/pr] 🚀  Start image=sagebionetworks/challenge-devcontainer:bf457d84
[CI/pr]   🐳  docker pull image=sagebionetworks/challenge-devcontainer:bf457d84 platform= username= forcePull=false
[CI/pr]   🐳  docker create image=sagebionetworks/challenge-devcontainer:bf457d84 platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[CI/pr]   🐳  docker run image=sagebionetworks/challenge-devcontainer:bf457d84 platform= entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[CI/pr]   🐳  docker exec cmd=[chown -R 1000:1001 /workspaces/challenge-registry] user=0 workdir=
[CI/pr]   ☁  git clone 'https://github.com/actions/checkout' # ref=v2
[CI/pr]   ☁  git clone 'https://github.com/nrwl/nx-set-shas' # ref=v2
[CI/pr] ⭐ Run Pre Derive appropriate SHAs for base and head for `nx affected` commands
[CI/pr]   ✅  Success - Pre Derive appropriate SHAs for base and head for `nx affected` commands
[CI/pr]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/pr]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/pr]   ☁  git clone 'https://github.com/actions/cache' # ref=v3
[CI/pr] ⭐ Run Main actions/checkout@v2
[CI/pr]   🐳  docker cp src=/home/vscode/.cache/act/actions-checkout@v2/ dst=/var/run/act/actions/actions-checkout@v2/
[CI/pr] close /tmp/act2677468349: file already closed
[CI/pr]   🐳  docker exec cmd=[chown -R 1000:1001 /var/run/act/actions/actions-checkout@v2/] user=0 workdir=
[CI/pr]   🐳  docker exec cmd=[node /var/run/act/actions/actions-checkout@v2/dist/index.js] user= workdir=
[CI/pr]   💾  ::save-state name=isPost::true
[CI/pr]   💬  ::debug::GITHUB_WORKSPACE = '/workspaces/challenge-registry'
[CI/pr]   💬  ::debug::qualified repository = 'Sage-Bionetworks/challenge-registry'
[CI/pr]   💬  ::debug::ref = 'refs/pull/%25!f(<nil>)/merge'
[CI/pr]   💬  ::debug::commit = '6bfdac5663a60f40724fa92da6bd4a97ada430a1'
[CI/pr]   💬  ::debug::clean = true
[CI/pr]   💬  ::debug::fetch depth = 0
[CI/pr]   💬  ::debug::lfs = false
[CI/pr]   💬  ::debug::submodules = false
[CI/pr]   💬  ::debug::recursive submodules = false
[CI/pr]   ❗  ::error::Input required and not supplied: token
[CI/pr]   ❌  Failure - Main actions/checkout@v2
[CI/pr] exitcode '1': failure
[CI/pr] ⭐ Run Post actions/checkout@v2
[CI/pr]   🐳  docker exec cmd=[node /var/run/act/actions/actions-checkout@v2/dist/index.js] user= workdir=
[CI/pr]   ✅  Success - Post actions/checkout@v2
[CI/pr] 🏁  Job failed
Error: Job 'pr' failed

See:

tschaffter · 2022-08-05T16:47:48Z

I got past this issue with (see act doc):

export GITHUB_TOKEN=changeme
~/bin/act pull_request -j pr --remote-name upstream -s GITHUB_TOKEN=$GITHUB_TOKEN

tschaffter · 2022-08-05T16:52:20Z

A glimpse into the containers started by act:

$ docker ps
CONTAINER ID   IMAGE                                             COMMAND                  CREATED          STATUS          PORTS     NAMES
3ad4c28b5032   sagebionetworks/challenge-devcontainer:bf457d84   "/usr/bin/tail -f /d…"   2 minutes ago    Up 2 minutes              act-CI-pr
404e9a1689e1   sagebionetworks/challenge-devcontainer:bf457d84   "/usr/bin/tail -f /d…"   35 minutes ago   Up 35 minutes             act-CI-main
07ff18172ba8   catthehacker/ubuntu:act-latest                    "/usr/bin/tail -f /d…"   48 minutes ago   Up 48 minutes             act-Optimize-SVGs-SVGs

tschaffter · 2022-08-05T16:54:41Z

The CI workflow needs to be updated because affected is not listing any projects since none have changed in my test branch.

      # - run: yarn nx affected --target=build --parallel --max-parallel=3
      # - run: yarn nx affected --target=test --parallel --max-parallel=2
      - run: yarn nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=build --parallel --max-parallel=3 --skip-nx-cache
      - run: yarn nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=test --parallel --max-parallel=2 --skip-nx-cache

tschaffter · 2022-08-05T16:55:28Z

I was able to reproduce the error locally with act! 🎉 (I guess :))

$ /workspaces/challenge-registry/node_modules/.bin/nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=build --parallel --max-parallel=3 --skip-nx-cache
| 
|  >  NX   Running target build for 6 project(s) and 2 task(s) they depend on:
| 
|     - challenge-api-gateway
|     - challenge-auth-service
|     - challenge-config-service
|     - challenge-core-service
|     - challenge-service-registry
|     - challenge-user-service
| 
|  
| Warning: JAVA_HOME environment variable is not set.
| Warning: JAVA_HOME environment variable is not set.
| Warning: JAVA_HOME environment variable is not set.
| Exception in thread "main" java.lang.reflect.InvocationTargetException
|       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
|       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
|       at java.base/java.lang.reflect.Method.invoke(Method.java:568)
|       at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:47)
|       at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:156)
|       at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:72)
| Caused by: java.lang.NoClassDefFoundError: org/apache/maven/exception/ExceptionHandler
|       at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
|       at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3402)
|       at java.base/java.lang.Class.getMethodsRecursive(Class.java:3543)
|       at java.base/java.lang.Class.getMethod0(Class.java:3529)
|       at java.base/java.lang.Class.getMethod(Class.java:2225)
|       at org.codehaus.plexus.classworlds.launcher.Launcher.getEnhancedMainMethod(Launcher.java:168)
|       at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:261)
|       at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
|       at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
|       at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
|       ... 7 more
| Caused by: java.lang.ClassNotFoundException: org.apache.maven.exception.ExceptionHandler
|       at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
|       at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
|       at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:247)
|       at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
|       ... 17 more
| 
| > nx run challenge-config-service:build
| 
| Executing command: ./mvnw package 
| Failed to execute command: ./mvnw package 
| Error: Command failed: ./mvnw package 
|     at checkExecSyncError (node:child_process:828:11)
|     at execSync (node:child_process:899:15)
|     at runBuilderCommand (/workspaces/challenge-registry/node_modules/@nxrocks/common/src/lib/core/jvm/utils.js:19:38)
|     at runBootPluginCommand (/workspaces/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/utils/boot-utils.js:15:43)
|     at /workspaces/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:10:62
|     at Generator.next (<anonymous>)
|     at /workspaces/challenge-registry/node_modules/tslib/tslib.js:118:75
|     at new Promise (<anonymous>)
|     at Object.__awaiter (/workspaces/challenge-registry/node_modules/tslib/tslib.js:114:16)
|     at buildExecutor (/workspaces/challenge-registry/node_modules/@nxrocks/nx-spring-boot/src/executors/build/executor.js:8:20)
|     at /workspaces/challenge-registry/node_modules/nx/src/command-line/run.js:101:23
|     at Generator.next (<anonymous>)
|     at /workspaces/challenge-registry/node_modules/tslib/tslib.js:118:75
|     at new Promise (<anonymous>)
|     at Object.__awaiter (/workspaces/challenge-registry/node_modules/tslib/tslib.js:114:16)
| 
|  >  NX   Error
| 
|    Pass --verbose to see the stacktrace.

tschaffter · 2022-08-05T16:58:26Z

It does not seem possible to run act concurrently locally. The following commands runs the modified CI workflow that uses run-many 20 times. The workflow failed 6 times out of 20 runs.

hyperfine --runs 20 --ignore-failure '~/bin/act pull_request -j pr -s GITHUB_TOKEN=$GITHUB_TOKEN || echo "x" >> error.txt'

# CI
- run: yarn nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=build --parallel --max-parallel=3 --skip-nx-cache

Note
The inspection of the container started by act shows that no limitations are set for the compute resources (CPU, memory, etc.), which act could have done to mirror GH configuration. Anyway, this means that the issue is probably not due to limited compute resources.

The above experiments run 20 * 6 Java projects with build = 120 build commands.

Let's run the following command that has previously failed (challenge-config-service) 120 times. First, let's do it without act for the sake of simplicity.

hyperfine --runs 120 --ignore-failure 'yarn nx run-many --projects challenge-config-service --target=build --parallel --max-parallel=3 --skip-nx-cache || echo "x" >> error.txt'

This task failed 0 times.

Let's run the same commands that failed above but without act:

hyperfine --runs 20 --ignore-failure 'yarn nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=build --parallel --max-parallel=3 --skip-nx-cache || echo "x" >> error.txt'

None of the runs failed. This seems to indicate that the issue is due to a difference between between my dev environment and the CI environment. While the CI workflow is using our dev container, it's using an older version (bf457d84, current version is f0afc028).

Trying again the command with act using the latest dev container (f0afc028):

hyperfine --runs 20 --ignore-failure '~/bin/act pull_request -j pr -s GITHUB_TOKEN=$GITHUB_TOKEN || echo "x" >> error.txt'

4 out of 20 runs failed. Same container used to run the task, but it fails when run using the GH workflow.

tschaffter · 2022-08-05T22:41:11Z

Narrowing down the issue

Here are the programs involved with running the task:

nx > @nxrocks/nx-spring-boot:build > ./mvnw

Skipping `@nxrocks/nx-spring-boot:build` and using `./mvnw` directly

In the Java projects, replacing:

    "build": {
      "executor": "@nxrocks/nx-spring-boot:build",
      "options": {
        "root": "apps/challenge-user-service"
      },
      "dependsOn": [
        {
          "target": "install",
          "projects": "dependencies"
        }
      ]
    },

by

    "build": {
      "executor": "@nrwl/workspace:run-commands",
      "options": {
        "command": "./mvnw package",
        "cwd": "apps/challenge-user-service"
      }
      "dependsOn": [
        {
          "target": "install",
          "projects": "dependencies"
        }
      ]
    },

Running the experiment:

hyperfine --runs 20 --ignore-failure '~/bin/act pull_request -j pr -s GITHUB_TOKEN=$GITHUB_TOKEN || echo "x" >> error.txt'

# CI
- run: yarn nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=build --parallel --max-parallel=3 --skip-nx-cache

4 out of 20 runs failed. This means that the Nx plugin @nxrocks/nx-spring-boot is not responsible for the issue.

tschaffter · 2022-08-06T02:57:41Z

Running system `mvn` instead of the `mvnw`

Before installing mvn, here is the version of maven and maven wrapper used:

Content of a Java project .mvn/wrapper/maven-wrapper.properties;

distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.8.4/apache-maven-3.8.4-bin.zip
wrapperUrl=https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.1.0/maven-wrapper-3.1.0.jar

Build a new version of the dev container where Maven is installed called sagebionetworks/challenge-devcontainer:maven.

$ mvn --version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 17.0.3, vendor: Debian, runtime: /usr/lib/jvm/java-17-openjdk-amd64
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "4.19.104-microsoft-standard", arch: "amd64", family: "unix"

Use this image in the GH workflow.

Note
To keep track of new changes, I created the first commit in my exploration branch. At that point, act failed with an issue about not being able to checkout the code. Pushing the exploration branch to my origin solved the issue.

Replacing the maven wrapper by the system-wide mvn:

    "build": {
      "executor": "@nrwl/workspace:run-commands",
      "options": {
        "command": "mvn package",
        "cwd": "apps/challenge-user-service"
      }
      "dependsOn": [
        {
          "target": "install",
          "projects": "dependencies"
        }
      ]
    },

Running the experiment:

hyperfine --runs 20 --ignore-failure '~/bin/act pull_request -j pr -s GITHUB_TOKEN=$GITHUB_TOKEN || echo "x" >> error.txt'

# CI
- run: yarn nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=build --parallel --max-parallel=3 --skip-nx-cache

~~3 runs out of 20 failed. This means that the maven wrapper is not the cause of the issue.~~

Warning
It seems that I've been using act incorrectly as what is run must have been pushed to a remote. Making a commit without pushing will make act generate an error. Below are the results where I really use mvw everywhere, including in the shared Java library.

1 run failed over 20 runs when using mvn (maven v3.6.3) instead of ./mvnw (maven v3.8.4).
2 runs failed over 40 runs when using mvn (maven v3.6.3) instead of ./mvnw (maven v3.8.4).

Observation: Using a globally-installed mvn binary contributes to reduce the frequency of the issue and suppress the errors of the type:

Exception in thread "main" java.util.zip.ZipException: zip END header not found
	at java.base/java.util.zip.ZipFile$Source.findEND(ZipFile.java:1469)
	at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1477)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1315)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1277)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:709)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:243)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:172)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:186)
Exception in thread "main" java.nio.file.NoSuchFileException: /root/.m2/wrapper/dists/apache-maven-3.8.6-bin/67568434/apache-maven-3.8.6-bin.zip.part
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at org.apache.maven.wrapper.Installer.unzip(Installer.java:207)
	at org.apache.maven.wrapper.Installer.createDist(Installer.java:110)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:151)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:76)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:429)
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266)
	at java.base/java.nio.file.Files.move(Files.java:1432)
	at org.apache.maven.wrapper.Installer.createDist(Installer.java:95)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:151)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:76)
Exception in thread "main" java.nio.file.NoSuchFileException: /root/.m2/wrapper/dists/apache-maven-3.8.6-bin/67568434/apache-maven-3.8.6-bin.zip.part
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:429)
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266)
	at java.base/java.nio.file.Files.move(Files.java:1432)
	at org.apache.maven.wrapper.Installer.createDist(Installer.java:95)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:151)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:76)

tschaffter · 2022-08-06T03:31:14Z

Issue when concurrent maven executions update `.m2`?

Just remembered that this could be the issue. This post may confirm this hypothesis.

One way to test this is to copy my local .m2 into the dev container used by the GH workflow.

Here I try again but without act and with removing the content of my local ~/.m2/repository/* before each run.

hyperfine --runs 20 --ignore-failure 'rm -fr ~/.m2/repository/*; yarn nx run-many --projects challenge-api-gateway,challenge-auth-service,challenge-config-service,challenge-core-service,challenge-service-registry,challenge-user-service --target=build --parallel --max-parallel=3 --skip-nx-cache || echo "x" >> error.txt'

0 runs failed over 20 and even 40 runs. This tends to show that the issue is not due to corruption of the ~/.m2 by concurrent maven executions, unless there is a difference between my local environment and the GH workflow that would make corruption more likely when run with the GH workflow.

tschaffter · 2022-08-06T13:45:49Z

Additional experiments

Configure the maven wrapper to use the latest version of Maven (3.8.6).
- The changelog of the latest release mentions concurrent build improvements.

Potential solutions

Use project-specific .m2 with -Dmaven.repo.local=$MAVEN_REPO or settings.xml (see this post).
- This approach may make caching in the CI workflow more complicated.
- I have two projects that relies on a shared library that is part of this monorepo. The targets build and build-image of these two projects rely on the fact that the shared library is installed and accessible to the two projects. One solution would be to install the shared library several times, each time with a different .m2 target. This complicated the setup.
Disable build tasks concurrency in CI workflow.
- Not a viable solution as the monorepo grows.
Disable build tasks concurrency in CI workflow for Java project only.
- Not a viable solution as the monorepo grows.
- Increase the complexity of the monorepo management.
Use Nx Distributed Task Execution?

tschaffter · 2022-08-06T17:49:26Z

After #500 , the CI workflow fails probably because of concurrent download of the maven binary by the maven wrapper.

Exception in thread "main" java.util.zip.ZipException: zip END header not found
	at java.base/java.util.zip.ZipFile$Source.findEND(ZipFile.java:1469)
	at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1477)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1315)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1277)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:709)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:243)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:172)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:186)
Exception in thread "main" java.nio.file.NoSuchFileException: /root/.m2/wrapper/dists/apache-maven-3.8.6-bin/67568434/apache-maven-3.8.6-bin.zip.part
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at org.apache.maven.wrapper.Installer.unzip(Installer.java:207)
	at org.apache.maven.wrapper.Installer.createDist(Installer.java:110)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:151)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:76)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:429)
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266)
	at java.base/java.nio.file.Files.move(Files.java:1432)
	at org.apache.maven.wrapper.Installer.createDist(Installer.java:95)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:151)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:76)
Exception in thread "main" java.nio.file.NoSuchFileException: /root/.m2/wrapper/dists/apache-maven-3.8.6-bin/67568434/apache-maven-3.8.6-bin.zip.part
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:429)
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266)
	at java.base/java.nio.file.Files.move(Files.java:1432)
	at org.apache.maven.wrapper.Installer.createDist(Installer.java:95)
	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:151)
	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:76)

tschaffter added bug Something isn't working java ci Continuous integration labels Jul 10, 2022

tschaffter self-assigned this Jul 10, 2022

tschaffter added the monorepo Monorepo management label Aug 5, 2022

tschaffter mentioned this issue Aug 6, 2022

Add Maven to the dev container #498

Closed

tschaffter mentioned this issue Aug 6, 2022

Explore mvn concurrency issue #499

Closed

This was referenced Aug 6, 2022

Install Maven and Java project dependencies sequentially #503

Closed

Install Maven and Java project dependencies sequentially #504

Merged

tschaffter closed this as completed in #504 Aug 7, 2022

tschaffter mentioned this issue Oct 4, 2022

Migrate remaining Maven projects to Gradle #763

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Maven commands fail randomly in CI/CD workflow #422

Maven commands fail randomly in CI/CD workflow #422

tschaffter commented Jul 10, 2022

tschaffter commented Jul 10, 2022

tschaffter commented Aug 1, 2022

tschaffter commented Aug 1, 2022

tschaffter commented Aug 5, 2022

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022

tschaffter commented Aug 5, 2022

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 6, 2022 •

edited

Loading

tschaffter commented Aug 6, 2022 •

edited

Loading

tschaffter commented Aug 6, 2022 •

edited

Loading

tschaffter commented Aug 6, 2022

Maven commands fail randomly in CI/CD workflow #422

Maven commands fail randomly in CI/CD workflow #422

Comments

tschaffter commented Jul 10, 2022

tschaffter commented Jul 10, 2022

tschaffter commented Aug 1, 2022

tschaffter commented Aug 1, 2022

tschaffter commented Aug 5, 2022

tschaffter commented Aug 5, 2022 • edited Loading

tschaffter commented Aug 5, 2022 • edited Loading

Troubleshooting

tschaffter commented Aug 5, 2022 • edited Loading

tschaffter commented Aug 5, 2022

tschaffter commented Aug 5, 2022

tschaffter commented Aug 5, 2022 • edited Loading

tschaffter commented Aug 5, 2022 • edited Loading

tschaffter commented Aug 5, 2022 • edited Loading

Narrowing down the issue

Skipping @nxrocks/nx-spring-boot:build and using ./mvnw directly

tschaffter commented Aug 6, 2022 • edited Loading

Running system mvn instead of the mvnw

tschaffter commented Aug 6, 2022 • edited Loading

Issue when concurrent maven executions update .m2?

tschaffter commented Aug 6, 2022 • edited Loading

Additional experiments

Potential solutions

tschaffter commented Aug 6, 2022

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

tschaffter commented Aug 5, 2022 •

edited

Loading

Skipping `@nxrocks/nx-spring-boot:build` and using `./mvnw` directly

tschaffter commented Aug 6, 2022 •

edited

Loading

Running system `mvn` instead of the `mvnw`

tschaffter commented Aug 6, 2022 •

edited

Loading

Issue when concurrent maven executions update `.m2`?

tschaffter commented Aug 6, 2022 •

edited

Loading