[native-image] Mongonaut crashes during runtime when compiled as a static native image

[gradinac]

What goes wrong:
Running the docker image that contains the final application works, but after some time, the container dies. Copying the native image from the container using docker cp and running it under GDB shows that it segfaults after running for a bit with the following stacktrace:

Thread 5 "Timer-0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffef7fe700 (LWP 387)]
0x00007ffff71f1448 in internal_getent (stream=stream@entry=0x7ffff6ad3ec0, 
    result=result@entry=0x7fffef7fcf90, buffer=buffer@entry=0x7fffef7fd1e0 "", 
    buflen=buflen@entry=1024, errnop=errnop@entry=0x7fffef7fe6b0, 
    herrnop=herrnop@entry=0x7fffef7fe6e8, af=2, flags=0)
    at nss_files/files-XXX.c:216
216	nss_files/files-XXX.c: No such file or directory.
(gdb) bt
#0  0x00007ffff71f1448 in internal_getent (stream=stream@entry=0x7ffff6ad3ec0, 
    result=result@entry=0x7fffef7fcf90, buffer=buffer@entry=0x7fffef7fd1e0 "", 
    buflen=buflen@entry=1024, errnop=errnop@entry=0x7fffef7fe6b0, 
    herrnop=herrnop@entry=0x7fffef7fe6e8, af=2, flags=0)
    at nss_files/files-XXX.c:216
#1  0x00007ffff71f26af in _nss_files_gethostbyname3_r (
    name=0x7ffff41071c8 "localhost", af=2, result=0x7fffef7fcf90, 
    buffer=<optimized out>, buflen=1024, errnop=0x7fffef7fe6b0, 
    herrnop=0x7fffef7fe6e8, ttlp=0x0, canonp=0x7fffef7fcf80)
    at nss_files/files-hosts.c:352
#2  0x00000000027046af in ?? ()
#3  0x000000000270528b in getaddrinfo ()
#4  0x0000000001abdaa3 in ?? ()
#5  0x00000000027a9000 in ?? ()
#6  0x00007ffff4106028 in ?? ()
#7  0x00007ffff7e6d7a0 in ?? ()
#8  0x0000000001d556ec in ?? ()
#9  0x00000001027a9000 in ?? ()
#10 0x00007ffff4107150 in ?? ()
<The rest of the backtrace has been omitted as the application was not compiled with debug symbols>

Steps to reproduce:
Mongonaut source:https://github.com/dekstroza/mongonaut
Reproducing this bug requires Docker.

• Running this demo locally and not in Kubernetes requires a MongoDB server. The way I went was to install MongoDB locally and then share it with the docker container running the application. However, the bug is still reproducible even without a MongoDB server, but you should not send any requests to the application.
• Clone the repo, checkout the static-linking branch and get the latest GraalVM and Micronaut using SDKMAN.
• Run mvn clean install in the project root directory.

You should now have a new docker image called dekstroza/mongonaut:1.0.0-SNAPSHOT
In order to run it, do docker run --net=host dekstroza/mongonaut:1.0.0-SNAPSHOT

If you have installed MongoDB, you can verify the requests work by running:

$ curl localhost:8080/mongonaut/alarms
[]
$ curl -X POST localhost:8080/mongonaut/alarms -d '{"id": 1,"name": "Beeeeep", "severity": "LOW"}' -H 'Content-Type:application/json'
{"id":1,"name":"Beeeeep","severity":"LOW"}
$ curl localhost:8080/mongonaut/alarms
[{"id":1,"name":"Beeeeep","severity":"LOW"}]

Either way, after waiting for some time, the container will die.
The demo was run on Ubuntu 18.04, with the kernel version 5.0.0-36-generic

[gradinac]

The root cause of this issue is in the function getaddrinfo used by the JDK. Due to a glibc bug (link) this function does not behave properly in statically linked multi-threaded programs. As mongonaut indirectly used this function, it would periodically crash.

In order to fix this, starting from version 20.0 a flag has been added to native-image that allows users to link against musl instead of glibc, which will solve the crash.
Another issue is that by statically linking against the system-provided glibc in most Linux distros, you do not get a truly static binary (anything that uses NSS results in a compiler warning emitted stating that you will need dynamic libraries at runtime). When statically linking against musl, your binary is completely standalone in regards to other libraries and can be used in a FROM scratch docker container.
As soon as the GraalVM 20.0 docker image is released, I will create an example on using it with mongonaut.

For anyone else wanting to use the flag, on 20.0 the usage is as follows: -H:UseMuslC=<path to static library bundle>. The library bundle format is documented in the flag help text. An example of bundle creation and the resulting bundle can be found here: link.

[LeifW]

Not sure how to access the "flag help text", but I did find this commit with a text file describing it: 87fe3d6#diff-0e5fe263fca2f2531fa44856795cb6d0