Adding ppc64le support back into the Jenkins CI SSH agent Docker build.

[ksalerno99]

The following are the steps I have automated to build Jenkins CI controller and agents for a foreign architecture. My build workstation architecture is amd64.

I use docker.io/multiarch/qemu-user-static to emulate the build processes only, I do not use full qemu-system-ppc64le emulation of a complete Operating System. It is unecessary and slow to attempt full emuluation of kernel etc. to build software, a container is much more efficient and you are always using the latest code, versus maintaining a full Operating System.

If the Jenkins infrastructure team is not already doing so, I suggest switching to this containerized method of building foreign target architecures, as these steps work well for s390x and aarch64 (just substitute ARCH=ppc64le below with ARCH=s390x or ARCH=arm64v8).

Build times on a modest 2-core 4-thread Intel i5 M520 @2.40GHz system:

The build time for two controller images (debian_jdk11 and rhel_ubi9_jdk17 simultaneously) was 11 minutes and 23.9 seconds.

The build time for the docker-agent was 4 minutes and 30.8 seconds.

The build time for the docker-ssh-agent was 3 minutes and 57.1 seconds.

Step 1: set ARCH environment variable to desired target architecture

ARCH=ppc64le

Step 2: run privileged qemu-user-static container to set host's kernel binfmt_misc magic to execute foreign arch binaries with appropriate /usr/bin/qemu-ARCH-static interpreter. IMPORTANT: must use --credential yes parameter.

sudo docker run --rm --privileged \
	docker.io/multiarch/qemu-user-static --reset -p yes --credential yes

Step 3: create Docker buildx container with target ARCH. Important: you must mount your host's Docker socket /var/run/docker.sock in the container, so that we can eliminate the need to run the Docker service inside the emulated container.

sudo docker run -dt -h docker-buildx-$ARCH --name docker-buildx-$ARCH \
	-v/var/run/docker.sock:/var/run/docker.sock $ARCH/ubuntu

Step 4: build Jenkins CI inside target ARCH Docker container with buildx

sudo docker exec -ti docker-buildx-$ARCH /bin/bash -c "\
	apt-get update; \
	apt-get install -y --no-install-recommends \
		ca-certificates curl gnupg lsb-release jq make git vim patch; \
	mkdir -m 0755 -p /etc/apt/keyrings; \
	curl -fsSL \
		https://download.docker.com/linux/ubuntu/gpg | \
		gpg --dearmor -o /etc/apt/keyrings/docker.gpg; \
	echo \"deb [arch=\$(dpkg --print-architecture) \
		signed-by=/etc/apt/keyrings/docker.gpg] \
		https://download.docker.com/linux/ubuntu \
		\$(lsb_release -cs) stable\" | \
		tee /etc/apt/sources.list.d/docker.list >/dev/null; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
		docker-ce-cli docker-buildx-plugin docker-compose-plugin"
sudo docker exec -ti docker-buildx-$ARCH /bin/bash -c "\
	cd; \
	git clone https://github.com/jenkinsci/docker.git; \
	git clone https://github.com/jenkinsci/docker-agent.git; \
	git clone https://github.com/jenkinsci/docker-ssh-agent.git; \
	docker buildx version; docker version; \
	cd docker && time make build && \
	cd ../docker-agent && time make build && \
	cd ../docker-ssh-agent && time make build"

Step 5 (optional): run Jenkins CI Docker controller and agent containers on Docker host to test (since we shared the /var/run/docker.sock of the host, all images are accessible from the host that were exported from the buildx container)

JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/log.properties"
JAVA_PROP=data/log.properties
sudo docker images | grep jenkins/jenkins
printf "Choose Tag to run: "
read TAG
mkdir data
sudo chown 1000:1000 data
echo "handlers=java.util.logging.ConsoleHandler" >$JAVA_PROP
echo "jenkins.level=FINEST" >>$JAVA_PROP
echo "java.util.logging.ConsoleHandler.level=FINEST" >>$JAVA_PROP
sudo docker run --rm --privileged \
	docker.io/multiarch/qemu-user-static --reset -p yes --credential yes
sudo docker run -d --name kensqa \
	-p 8080:8080 -p 50000:50000 --restart=on-failure \
	--env JAVA_OPTS="$JAVA_OPTS" \
	-v $(pwd)/data:/var/jenkins_home jenkins/jenkins:$TAG
mkdir docker-agent-workdir
sudo chown 1000:1000 docker-agent-workdir
sudo docker run -dt --name docker-agent --init \
	-v $(pwd)/docker-agent-workdir:/home/jenkins/agent \
	jenkins/agent java -jar /usr/share/jenkins/agent.jar \
	-workDir /home/jenkins/agent
echo | ssh-keygen -t rsa -N ''
mkdir docker-ssh-agent-workdir
sudo chown 1000:1000 docker-ssh-agent-workdir
sudo docker run -d --name docker-ssh-agent \
	-v $(pwd)/docker-ssh-agent-workdir:/home/jenkins/agent -p 9022:22 \
	-e "JENKINS_AGENT_SSH_PUBKEY=$(cat .ssh/id_rsa.pub)" \
	jenkins/ssh-agent
cat .ssh/id_rsa .ssh/id_rsa.pub
sudo docker ps

Results:

sudo docker images
REPOSITORY                   TAG                     IMAGE ID       CREATED        SIZE
jenkins/ssh-agent            bullseye-jdk11          a9554b89c3b8   11 hours ago   384MB
jenkins/ssh-agent            jdk11                   a9554b89c3b8   11 hours ago   384MB
jenkins/ssh-agent            latest                  a9554b89c3b8   11 hours ago   384MB
jenkins/ssh-agent            latest-bullseye-jdk11   a9554b89c3b8   11 hours ago   384MB
jenkins/ssh-agent            latest-jdk11            a9554b89c3b8   11 hours ago   384MB
jenkins/agent                bullseye-jdk11          8728cac70f74   11 hours ago   369MB
jenkins/agent                jdk11                   8728cac70f74   11 hours ago   369MB
jenkins/agent                latest                  8728cac70f74   11 hours ago   369MB
jenkins/agent                latest-bullseye-jdk11   8728cac70f74   11 hours ago   369MB
jenkins/agent                latest-jdk11            8728cac70f74   11 hours ago   369MB
jenkins/jenkins              2.356                   a04b8f9b2f1a   11 hours ago   510MB
jenkins/jenkins              2.356-jdk11             a04b8f9b2f1a   11 hours ago   510MB
jenkins/jenkins              2.356-rhel-ubi9-jdk17   1272648dbb50   11 hours ago   542MB
ppc64le/ubuntu               latest                  4220c61b3ab7   12 days ago    118MB
multiarch/qemu-user-static   latest                  3539aaa87393   2 months ago   305MB

sudo docker inspect jenkins/jenkins:2.356 | grep Architecture
        "Architecture": "ppc64le",

sudo docker inspect jenkins/jenkins:2.356-rhel-ubi9-jdk17 | grep Architecture
        "Architecture": "ppc64le",

sudo docker inspect jenkins/agent:jdk11 | grep Architecture
        "Architecture": "ppc64le",

sudo docker inspect image jenkins/ssh-agent:jdk11 | grep Architecture
        "Architecture": "ppc64le",

[dduportal]

Nice, thanks! Let's roll!