qemu-arm-static 2.8 and Java+Maven setup not working

[slamont]

Hi,

We have systems that would stop working if we want to use the qemu-arm-static v2.8.0 that is used in this project.
I've created a project to demonstrate the problem.

I'm not quite sure where the problem is to be honest, is it with Docker, qemu or Java ? I don't know for sure. But what I do know, is that it was working with 2.5, 2.6 and 2.7 and stopped working with 2.8

The demonstration project linked above is a very simplified version of what we use and was created specifically to help the debugging of this issue. So feel free to clone it and let me know if you need more information.

The Dockerfile start from "multiarch/debian-debootstrap:armhf-jessie" and simply add oracle-java and maven. The qemu-arm-static binaries are overridden using volume bind (-v).

The test.sh will take care of everything to demonstrate the problem

[vielmetti]

Similar bug on aarch64 (arm64): https://bugs.linaro.org/show_bug.cgi?id=3259

that bug summary:

qemu-aarch64-static is a extremely valuable that can readily be combined with chroot to create a flexible and convenient environment when actual aarch64 hardware is unavailable or unsuitable. It appears that there is an issue within a recent qemu-aarch64-static that prohibits the reliable use of Java. In particular, javac exits with SIGSEGV on the trivial example I tried.

[vielmetti]

@slamont - are you still seeing this issue?

[slamont]

We still see the problem in our production setup. But we have a workaround and stop the investigation. The workaround for us, is to limit Docker to use only 1 cpu (--cpuset-cpus="0") which could means that this problem is related to Multi-thread in qemu.

I'll update the test repository with the latest 2.8.4 and will add 2.9.1 to see if it change anything.

[slamont]

Test repo is now upgraded. While testing the modifications, the workaround worked for all versions. Without the workaround, 2.8 and 2.9 fails.

The test now run twice by version, first is with the workaround (--cpuset-cpus="0") the second is without it.

[vielmetti]

Thanks @slamont . Stuart Monteith has this report from the Linaro bug tracker at https://bugs.linaro.org/show_bug.cgi?id=3259#c4

When code is being emitted or patched, sometimes the page it is trying to do so is protected from writes. This is something that fundamentally just works on real machines. I'm currently trying to understand the precise mechanism for managing pages.

There's also a bug report on Launchpad at https://bugs.launchpad.net/qemu/+bug/1659901 with this comment from rdicroce:

Additional investigation reveals the problem has something to do with the Azul ARM32 JIT. If I run Java with -Xint to force interpreter-only mode, this problem doesn't occur.

[vielmetti]

The rdicroce report above also identifies when the build broke:

qemu-user-static version 1:2.7+dfsg-3~bpo8+2 works fine. Version 1:2.8+dfsg-1~bpo8+1 always causes Java to crash with a SIGSEGV. The location of the crash appears to be random and hasn't been the same twice.

[vielmetti]

http://lists.nongnu.org/archive/html/qemu-devel/2014-03/msg01665.html gives a perspective from 2014 as to how threading in Java is not going to be easy to support in QEMU. Peter Maydell wrote then, referring to another Java crash in QEMU:

Not sure there's much point looking very deeply into this.
Java programs are threaded, threaded programs don't work
under QEMU => don't try to run Java under QEMU :-)

[ghost]

I'm seeing this manifest in an arm chroot env, on opensuse host

	https://bugs.launchpad.net/raspbian/+bug/1732556

where

rpm -q --whatprovides /usr/bin/qemu-arm
	qemu-linux-user-2.10.1-412.1.x86_64

/usr/bin/qemu-arm --version
	qemu-arm version 2.10.1
	Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

file /usr/bin/qemu-arm
	/usr/bin/qemu-arm: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.0.0, BuildID[sha1]=65c6891466cacd408ef8aa7b8738548daacb9db2, not stripped

Do I understand that this is, in fact, a qemu issue?

Given @vielmetti 's last comment, is there a fix available/planned?

[junaruga]

I experienced the similar issue when installing Java on the QEMU environment.

SIGSEGV (core dump) java -Xmx64m -jar $JAR -storepass "$storepass"
https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1818631

Here is the reproducer script.
https://github.com/junaruga/ubuntu-arm-java-sigsegv

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset
$ docker build --rm -t sample -f Dockerfile-arm64 .

Do you know how to avoid this error?

[slamont]

If you force docker to use only one CPU of should do the trick. That's what we did and we are using it in our build environment without issue. It's slower than I would've liked, but it does the job. I'm currently not at work, so I don't have the exact docker command. Let me know if you need it. Also, I don't know if it changes anything, but since Java 8_191 there seems to be a better support of the containers for the JVM. So, maybe the CPU trick can be avoided, I don't know. Sylvain Lamontagne Le jeu. 7 mars 2019 13:01, Jun Aruga <notifications@github.com> a écrit :

…

I experienced the similar issue when installing Java on the QEMU environment. SIGSEGV (core dump) java -Xmx64m -jar $JAR -storepass "$storepass" https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1818631 Here is the reproducer script. https://github.com/junaruga/ubuntu-arm-java-sigsegv $ docker run --rm --privileged multiarch/qemu-user-static:register --reset $ docker build --rm -t sample -f Dockerfile-arm64 . Do you know how to avoid this error? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABeXXSCSC3OMjlI5mcVO-S2Gcslb-EKHks5vUVPcgaJpZM4MAJF-> .

[junaruga]

Hi @slamont

Thanks for your info and your help!

I'm currently not at work, so I don't have the exact docker command. Let me know if you need it.

I want to know the exact docker commend.

Because I assumed the command is docker build --cpuset-cpus="0" or docker run --cpuset-cpus="0".

$ docker build --help
...
      --cpuset-cpus string         CPUs in which to allow
                                   execution (0-3, 0,1)
...

But below commands still show same error.

 $ docker run --rm --privileged --cpuset-cpus="0" multiarch/qemu-user-static:register --reset
 $ docker build --rm -t sample -f Dockerfile-arm64 --cpuset-cpus="0" .

https://github.com/junaruga/ubuntu-arm-java-sigsegv/tree/feature/one-cpu
https://travis-ci.org/junaruga/ubuntu-arm-java-sigsegv/builds/504096420

[slamont]

Hi @junaruga,

just adding --cpuset-cpus="0"did the trick. But after verification, it's using qemu 2.7 not 2.8.

So, we have a Docker image built from a Debian Jessie Arm filesystem, with qemu-arm-static 2.7 and run our container with --cpuset-cpus="0".

[junaruga]

@slamont thank you for the info.

So, we have a Docker image built from a Debian Jessie Arm filesystem, with qemu-arm-static 2.7 and run our container with --cpuset-cpus="0".

What is the actual commands you executed on the Debian system?
Did you use register/Dockerfile to build it?
The qemu-arm-static is included in multiarch/qemu-user-static:register container?

I tried to see the environment to check the version of qemu with docker run -it bash.

$ docker run --rm --privileged --cpuset-cpus="0" --name qemu-user-static -it multiarch/qemu-user-static:register bash

But it seems the Dockerfile using a entry point to register/register.sh.
https://github.com/multiarch/qemu-user-static/blob/master/register/Dockerfile#L6

ENTRYPOINT ["/register"]

I could not check the version by myself.

[pm215]

QEMU 2.7 and 2.8 are quite old versions -- we think we fixed the java segfaults in upstream QEMU in about the 2.12 release. Is it possible for you to try this with a more recent QEMU release, like 3.1 or the (about to be released) 4.0 ?

[vielmetti]

@pm215 - the place to introduce new versions is in the toolchains used for cross-development on systems like Travis CI. We'd need to dig into the base containers used to do cross-builds (often embedded in Dockerfiles and base layers) and identify how to tip the ecosystem forwards towards a newer release.

It looks like this particular repo has a toolchain path that gets a 3.1 release and pushes it into Alpine via a Fedora RPM and rpm2cpio (a somewhat esoteric path). Has QEMU 3.1 been bundled for Debian distribution?

[junaruga]

I might find a part using the old version QEMU v2.12.0 in the container "ubuntu-debootstrap" that I was using. I sent pull-request to use the latest qemu-user-static here.
multiarch/ubuntu-debootstrap#25

[junaruga]

For debian-debootstrap, here is the pull-request.
multiarch/debian-debootstrap#14

[junaruga]

@slamont I was success to run java on my environment with ubuntu-debootstrap bionic by above pull-request to ubuntu-debootstrap repository to update qemu to 3.1.0!

https://github.com/junaruga/ubuntu-arm-java-sigsegv
https://travis-ci.org/junaruga/ubuntu-arm-java-sigsegv/branches

$ docker run --rm -t multiarch/ubuntu-debootstrap:arm64-bionic /usr/bin/qemu-aarch64-static --version
qemu-aarch64 version 3.1.0 (qemu-3.1.0-6.fc30)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers

I assume you are trying to use multiarch/debian-debootstrap:armhf-jessie.
Unfortunately it is still using qemu 2.12.0 internally.

$ docker run --rm -t multiarch/debian-debootstrap:armhf-jessie /usr/bin/qemu-arm-static --version
qemu-arm version 2.12.0 (qemu-2.12.0-1.fc29)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

Because below trigger file to create the new container images does not include jessie (removed it) by multiarch/debian-debootstrap@184214f .
https://github.com/multiarch/debian-debootstrap/blob/master/.travis.yml

But debian sid using qemu 3.1.0.

$ docker run --rm -t multiarch/debian-debootstrap:armhf-sid /usr/bin/qemu-arm-static --version
qemu-arm version 3.1.0 (qemu-3.1.0-6.fc30)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers

So, if you can use sid, you do not see the java issue on your side, I think we can close this ticket.

[vielmetti]

Looks related to docker/for-mac#3646

[junaruga]

@vielmetti this issue still happens on your environment?

[vielmetti]

Yes @junaruga - the particular circumstance is that Docker has a new feature buildx which uses QEMU to make transparent cross-builds of e.g. arm64 containers on an x86 host. However - for reasons having to do with an Alpine bug - they are currently using a back-rev version of QEMU, and have tickled this bug. This was discovered by @carlosedp .

No need to reopen this ticket, but it provides useful context for people working on the other.

Basically the working strategy is if you find QEMU 2.8 in the wild, or embedded into a container image, you should root it out and replace it with something newer.

[junaruga]

@vielmetti sure, thanks for the explanation.

By the way, for ubuntu-core, here is the pull-request to update qemu.
multiarch/ubuntu-core#16

[vielmetti]

Thanks @junaruga - appreciated. I'll do some searching through github and try to find other images with QEMU 2.8 that should be updated.

[vielmetti]

The ubuntu-core changes for a newer QEMU have been merged.