Docker error for multi-arch build

[Jaskaran-Singh07]

Description

Info

• docker-maven-plugin version : 0.43.0
• Maven version (mvn -v) : 3.8.6
• Docker version : 20.10.13

I am using the docker-maven-plugin for a multi-architecture linux/amd64,linux/arm64 build using an external Dockerfile.

FROM amazoncorretto:8-alpine3.17
RUN apk add --update curl && rm -rf /var/cache/apk/*
ADD kcheque-api*.jar app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

I have defined the plugin in the pom.xml file in the following manner:

<plugin>
        <groupId>io.fabric8</groupId>
        <artifactId>docker-maven-plugin</artifactId>
        <version>0.43.0</version>
        <configuration>
	        <images>
		        <image>
			        <name>${docker.image.prefix}/kcheque-api</name>
			        <build>
				        <dockerFile>${project.basedir}/src/main/docker/Dockerfile</dockerFile>
				        <buildx>
					        <platforms>
						        <platform>${docker.platforms}</platform>
					        </platforms>
				        </buildx>
			        </build>
		        </image>
	        </images>
	        <resources>
		        <resource>
			        <targetPath>/</targetPath>
			        <directory>${project.build.directory}</directory>
			        <include>${project.build.finalName}.jar</include>
		        </resource>
	        </resources>
        </configuration>
        <executions>
	        <execution>
		        <id>docker-build</id>
		        <goals>
			        <goal>build</goal>
		        </goals>
	        </execution>
        </executions>
</plugin>

My workflow is in the following manner:
I start the build on an x86 server. The application is built and deployed using jenkins with the command mvn clean package docker:build -Ddocker.platforms=linux/amd64,linux/arm64.
I get the following error while building: [ERROR] DOCKER> Error status (125) while creating builder maven

Error

I think the error is due to this:
docker --config /var/lib/jenkins/workspace/Application/target/docker//Application/docker buildx create --driver docker-container --name maven
[INFO] DOCKER> unknown flag: --driver

It's trying to create a builder maven using the config file defined in the path /var/lib/jenkins/workspace/Application/target/docker/<ECR REPO URL>/Application/docker. I wish to know if there is a way to disable using the config file for creating builder, or how I can change the config file path for the builder.

I also tried creating a test builder using <builderName>test_builder</builderName> in the buildx tag, as given in the docs, however, I am unable to change the config file for the same. It's not necessary to provide a config file while creating a builder and I wish to do that.

Also, the error is unknown flag --driver which is strange, considering I was able to create the test_builder using docker buildx create --name test_builder --driver docker-container --bootstrap.

[jdelker]

There are quite a few issues (see #1583, #1643 for example), which all seem to boil down to the same problem.
That is:
Presence of the --config flag, causes the build to fail with unknown flag: --driver.

Some propose to simply remove the --config ... parameter - and in fact, that solves the problem.
However, that "fix" totally neglects, that this flag has a purpose. It effectively scopes that execution to a dedicated directory, which prevents your "default" docker environment from being cluttered with project leftovers.
By giving docker an explicit config directory like docker --config /my/tmp/config/dir <command> ... you're telling Docker to use this non-standard config directory, instead of the default (which is ~/.docker on Unix/macos).

I figured, that the reason why this fails on some (!) Docker setups, seems to be the following:
By pointing Docker to a (mostly empty) config directory, it basically looses it's links to the installed plugins.
At least with Docker Desktop on macos (4.19 currently), all plugin executables are linked within ~/.docker/cli-plugins:

$HOME/.docker/cli-plugins
├── docker-buildx -> /Applications/Docker.app/Contents/Resources/cli-plugins/docker-buildx
├── docker-compose -> /Applications/Docker.app/Contents/Resources/cli-plugins/docker-compose
├── docker-dev -> /Applications/Docker.app/Contents/Resources/cli-plugins/docker-dev
├── docker-extension -> /Applications/Docker.app/Contents/Resources/cli-plugins/docker-extension
├── docker-init -> /Applications/Docker.app/Contents/Resources/cli-plugins/docker-init
├── docker-sbom -> /Applications/Docker.app/Contents/Resources/cli-plugins/docker-sbom
├── docker-scan -> /Applications/Docker.app/Contents/Resources/cli-plugins/docker-scan
└── docker-scout -> /Applications/Docker.app/Contents/Resources/cli-plugins/docker-scout

As there is no cli-plugins subdir within the config directory generated by the plugin, the docker execution fails.
Proof: If you copy (or link) this directory into the new config directory, the plugin builds the images without any error.

Unfortunately, this behavior is non consistent with different Docker installations.
I was unable to perceive this problem with Docker Desktop on macos/intel, but Docker Desktop on macos/arm, as well as Docker Desktop on Windows, fails with the "unknown flag: --driver" problem on the very same project.

Conclusion:
I can only assume, that there is some kind of fallback mechanism implemented on Docker Desktop for macos/intel, that allows it to still find it's plugins (primarily buildx here), despite the non-standard config directory.
Other setups, don't have that feature obviously.

[jdelker]

Problem resolved!
Some comparisons with docker --config ... info has finally brought the desired insight.

Beware: The following may or may not apply to Docker Desktop on macos only.

I discovered, that Docker Desktop is using an additional path to resolve plugins: /usr/local/lib/docker/cli-plugins.
In my setup, this is a symbolic link: cli-plugins -> /Applications/Docker.app/Contents/Resources/cli-plugins.
On the other macos/arm host, this link is missing! Whether this is an issue with Docker Desktop Installer, I can't say - but it looks like.

So finally:
If you get the error "unknown flag: --driver" during the "docker --config ... buildx ..." actions, it is most probably caused by the fact, that access to your plugins is missing in that context.
Solution: Find a way that your docker is using a fallback path for plugin lookup.

This might help: docker/cli#1534

[janpauldahlke]

i just discovered with a collegue that it appears to be a problem on apple m2, since he has the m1 model. We checked out and build https://github.com/fabric8io/docker-maven-plugin/tree/master/samples/multi-architecture and for him it works, for me doesn't. uncertain if this helps, tho.
i commented out the <platforms> basically to be able to work at least.

cheers

[Jaskaran-Singh07]

Thanks for your insightful comments @jdelker @janpauldahlke

Update: After facing this issue and being unable to resolve it, I moved on to trying another plugin. However, due to facing some issues with that plugin too, I moved back to testing this one and it worked this time.

Simply updating docker with sudo yum upgrade docker seems to have done the trick. It increases the docker version to 20.10.23, which while being just a patch update, resolved the issues within docker. The config file is now picked up at this path /var/lib/jenkins/workspace/Application/target/docker/<ECR REPO URL>/Application/docker which was earlier completely empty. The image is now being built and pushed to remote repo as well as being stored locally.

Although, when I try to run it in an interactive env using docker exec -it <image> sh, it gives the error Unable to access jarfile /app.jar. I am unsure whether this is due to the plugin, or the base image amazoncorretto I am using for the multi-arch build. I was not facing this issue when the application was only on x86, using openjdk:8-alpine image and I was able to interact with the shell env and access the app.jar

---

Edit: setting the tag contextdDir as ${project.basedir} in the pom file works for the above problem.

[tdolphin]

The solution for me with homebrew used to install docker on M1

brew install docker docker-compose docker-buildx docker-credential-helper

sudo mkdir -p /usr/local/lib/docker
sudo ln -sfn /opt/homebrew/lib/docker/cli-plugins /usr/local/lib/docker/cli-plugins