[GR-48121] Native image compilation fails in Gitlab container environment.

[pompiuses]

Describe the issue
I'm trying to compile a native image in our build pipeline which is running in a container in Gitlab which in turn runs inside our Kubernetes cluster. The native-image tool crashes with the stacktrace below. I've tried using two different container images with both the GraalVM CE JDK and the Oracle GraalVM JDK and get errors in both cases. When running the containers locally in Docker on my Ubuntu machine everything works fine.

Steps to reproduce the issue
Setup a build pipeline in Gitlab. Our Gitlab instance runs in Kubernetes. Then create container images (Dockerfiles) which can be used to build the application inside Gitlab. I'm using these two Dockerfiles:
GraalVM CE JDK Dockerfile
Oracle GraalVM JDK Dockerfile

Then inside the running container in the Gitlab pipeline first build a fat jar of your application and then use the native-image tool to compile it. I'm using the following parameters:

native-image -jar target/my-app-1.0-SNAPSHOT-jar-with-dependencies.jar -H:ConfigurationFileDirectories=graalvm/tracing-agent -o target/app --no-fallback -H:+AddAllCharsets --enable-preview -H:+ReportExceptionStackTraces -H:+StaticExecutableWithDynamicLibC -R:MinHeapSize=128m -R:MaxHeapSize=768m

The native-image tool then immediately crashes with stacktrace below.

Describe GraalVM and your environment:

• A container using the Dockerfiles above which is running in a Gitlab build pipeline which in turn runs inside a Kubernetes cluster which in turn runs on VMware which in turn runs on physical hardware.
• GraalVM version: I've tested both GraalVM CE 20.0.2+9.1 and Oracle GraalVM 20.0.2+9.1
• JDK major version: 20
• OS: See Dockerfiles above. Debian and Oracle Linux.
• Architecture: What the Gitlab container environment is running on I don't know. The container images are built using the Dockerfiles above on my local Ubuntu 18.04 machine using AMD64.

More details
Stacktrace when building with the Oracle GraalVM JDK:

Exception in thread "main" java.lang.InternalError: java.lang.reflect.InvocationTargetException
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:67)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Container.metrics(Container.java:44)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.ContainerInfo.<init>(ContainerInfo.java:34)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.Containers.memoryLimitInBytes(Containers.java:126)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.heap.PhysicalMemory.doInitialize(PhysicalMemory.java:145)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.heap.PhysicalMemory.size(PhysicalMemory.java:88)
	at java.base@20.0.2/java.lang.Runtime.maxMemory(Runtime.java:919)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.DirectMemoryAccessors.initialize(Target_jdk_internal_misc_VM.java:118)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.DirectMemoryAccessors.getPageAlignDirectMemory(Target_jdk_internal_misc_VM.java:100)
	at java.base@20.0.2/jdk.internal.misc.VM.isDirectMemoryPageAligned(VM.java:156)
	at java.base@20.0.2/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:124)
	at java.base@20.0.2/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:343)
	at java.base@20.0.2/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
	at java.base@20.0.2/sun.nio.ch.IOUtil.read(IOUtil.java:303)
	at java.base@20.0.2/sun.nio.ch.IOUtil.read(IOUtil.java:283)
	at java.base@20.0.2/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:234)
	at java.base@20.0.2/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:74)
	at java.base@20.0.2/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
	at java.base@20.0.2/java.io.InputStream.read(InputStream.java:217)
	at java.base@20.0.2/java.util.Properties$LineReader.readLine(Properties.java:504)
	at java.base@20.0.2/java.util.Properties.load0(Properties.java:420)
	at java.base@20.0.2/java.util.Properties.load(Properties.java:409)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.loadProperties(NativeImage.java:2074)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.loadProperties(NativeImage.java:2063)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption.<init>(MacroOption.java:380)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption.create(MacroOption.java:370)
	at java.base@20.0.2/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base@20.0.2/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base@20.0.2/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base@20.0.2/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base@20.0.2/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1921)
	at java.base@20.0.2/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base@20.0.2/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base@20.0.2/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base@20.0.2/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base@20.0.2/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption$Registry.collectMacroOptions(MacroOption.java:184)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption$Registry.addMacroOptionRoot(MacroOption.java:200)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.<init>(NativeImage.java:750)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.build(NativeImage.java:1577)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1557)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.main(NativeImage.java:1531)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base@20.0.2/java.lang.reflect.Method.invoke(Method.java:578)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:63)
	... 41 more
Caused by: java.lang.ExceptionInInitializerError
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:78)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupMetrics.getInstance(CgroupMetrics.java:164)
	... 43 more
Caused by: java.lang.NullPointerException
	at java.base@20.0.2/java.util.Objects.requireNonNull(Objects.java:233)
	at java.base@20.0.2/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:297)
	at java.base@20.0.2/java.nio.file.Path.of(Path.java:148)
	at java.base@20.0.2/java.nio.file.Paths.get(Paths.java:69)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupUtil.lambda$readStringValue$0(CgroupUtil.java:57)
	at java.base@20.0.2/java.security.AccessController.executePrivileged(AccessController.java:147)
	at java.base@20.0.2/java.security.AccessController.doPrivileged(AccessController.java:571)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupUtil.readStringValue(CgroupUtil.java:59)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:66)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemController.getLongValue(CgroupSubsystemController.java:125)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.getLongValue(CgroupV1Subsystem.java:269)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.getHierarchical(CgroupV1Subsystem.java:215)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.setSubSystemControllerPath(CgroupV1Subsystem.java:203)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.initSubSystem(CgroupV1Subsystem.java:111)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.<clinit>(CgroupV1Subsystem.java:47)
	... 45 more

Stacktrace when building with the GraalVM CE JDK:

Fatal error: Failed while initializing the performance data.: java.lang.InternalError: java.lang.reflect.InvocationTargetException
    at com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:67)
    at com.oracle.svm.core.containers.Container.metrics(Container.java:44)
    at com.oracle.svm.core.ContainerInfo.<init>(ContainerInfo.java:34)
    at com.oracle.svm.core.Containers.activeProcessorCount(Containers.java:88)
    at java.lang.Runtime.availableProcessors(Runtime.java:337)
    at com.oracle.svm.core.jvmstat.SystemCounters.getAvailableProcessors(SystemCounters.java:186)
    at com.oracle.svm.core.jvmstat.SystemCounters.allocate(SystemCounters.java:124)
    at com.oracle.svm.core.jvmstat.PerfManager.allocate(PerfManager.java:138)
    at com.oracle.svm.core.jvmstat.PerfManager$PerfDataThread.initializeMemory(PerfManager.java:Exception in thread "main" 212)
    at com.oracle.svm.core.jvmstat.PerfManager$PerfDataThread.run(PerfManager.java:187)
    at com.oracle.svm.core.thread.PlatformThreads.java.lang.InternalError: java.lang.reflect.InvocationTargetException
threadStartRoutine(PlatformThreads.java:807)
    at com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:67)
(PosixPlatformThreads.java:210)	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Container.metrics(Container.java:44)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.ContainerInfo.<init>(ContainerInfo.java:34)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.Containers.memoryLimitInBytes(Containers.java:126)
Current timestamp: 1691678201732	at org.graalvm.nativeimage.builder/com.oracle.svm.core.heap.PhysicalMemory.doInitialize(PhysicalMemory.java:145)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.heap.PhysicalMemory.size(PhysicalMemory.java:88)
Printing Instructions (ip=0x	at java.base@20.0.2/java.lang.Runtime.maxMemory(Runtime.java:919)
000	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.DirectMemoryAccessors.initialize(Target_jdk_internal_misc_VM.java:118)
000	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.DirectMemoryAccessors.getPageAlignDirectMemory(Target_jdk_internal_misc_VM.java:100)
000049ea00):	at java.base@20.0.2/jdk.internal.misc.VM.isDirectMemoryPageAligned(VM.java:156)

  	at java.base@20.0.2/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:124)
0x0000000	at java.base@20.0.2/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:343)
00049e9e0: 	at java.base@20.0.2/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
0x00	at java.base@20.0.2/sun.nio.ch.IOUtil.read(IOUtil.java:303)
 0x	at java.base@20.0.2/sun.nio.ch.IOUtil.read(IOUtil.java:283)
41 0xc7 0x	at java.base@20.0.2/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:234)
87 0x4c 0x01	at java.base@20.0.2/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:74)
 0x00 	at java.base@20.0.2/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
0x00 	at java.base@20.0.2/java.io.InputStream.read(InputStream.java:217)
0xfe 0xfe 0xfe 0x7e 0x48	at java.base@20.0.2/java.util.Properties$LineReader.readLine(Properties.java:504)
 0x8b 0x	at java.base@20.0.2/java.util.Properties.load0(Properties.java:420)
7c 0x	at java.base@20.0.2/java.util.Properties.load(Properties.java:409)
24
  0x0000000	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.loadProperties(NativeImage.java:2074)
00049e9f0	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.loadProperties(NativeImage.java:2063)
: 0x20 	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption.<init>(MacroOption.java:380)
0x48 	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption.create(MacroOption.java:370)
0x8b 0x	at java.base@20.0.2/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
74 0x24 0x	at java.base@20.0.2/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
10 0x48	at java.base@20.0.2/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
 0x8b 0x	at java.base@20.0.2/java.util.Iterator.forEachRemaining(Iterator.java:133)
54 0x24	at java.base@20.0.2/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1921)
 0x18 0xe8 0x70 	at java.base@20.0.2/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
0xfc 0xff	at java.base@20.0.2/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
 0xff
  0x000	at java.base@20.0.2/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
00000	at java.base@20.0.2/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
0049ea00	at java.base@20.0.2/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
: 0x90	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption$Registry.collectMacroOptions(MacroOption.java:184)
 0xcc 0x	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption$Registry.addMacroOptionRoot(MacroOption.java:200)
cc 0x	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.<init>(NativeImage.java:750)
cc 0xcc 0x	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.build(NativeImage.java:1577)
cc 0xcc	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1557)
 0xcc 	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.main(NativeImage.java:1531)
0xcc 0xcc 0xcc Caused by: java.lang.reflect.InvocationTargetException
0xcc 0x	at java.base@20.0.2/java.lang.reflect.Method.invoke(Method.java:578)
cc 0xcc	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:63)
 0xcc	... 41 more
 0xcc
  0x0000Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem
00000	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:78)
049ea10: 0x	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupMetrics.getInstance(CgroupMetrics.java:164)
48 0x	... 43 more
83 0xec 0x18 0x48 0xb8 0x18 0xde 0x9f 0x00 0x00 0x00 0x00 0x00 0x49 0x8d

Top of stack (sp=0x00007fdcf16fed20):
  0x00007fdcf16fed20: 0x00007fdcf2658140 0x00007fdcf2a96d30
  0x00007fdcf16fed30: 0x00007fdcf26adbe8 0x00007fdcf0d35188
  0x00007fdcf16fed40: 0x00000000004d8ae9 0x000000000053398a
  0x00007fdcf16fed50: 0x00007fdcf0d03e18 0x00000000004d8ae9
  0x00007fdcf16fed60: 0x0000000000502319 0x00007fdcf2baab08
  0x00007fdcf16fed70: 0x00007fdcf2a96e10 0x00000000004d8b6b
  0x00007fdcf16fed80: 0x00007fdcf16ff640 

[pompiuses]

If it's the case that GraalVM can't find available memory referencing java.lang.Runtime.maxMemory, or cpu cores for that matter with the CE JDK referencing java.lang.Runtime.availableProcessors, there should probably be separate input parameters to manually override these, or make GraalVM more robust with sensible default settings? That would atleast be the quickest and most user friendly way to have something working I think. The alternative is to tweak the server settings of the container environment which in many cases may not be possible.

[pompiuses]

Similar issues also posted for Kubernetes:
micrometer-metrics/micrometer#2800 (comment)
kubernetes/kubernetes#103944

Seems to be related to cgroup access with certain configurations in Kubernetes clusters. This bug has been fixed in OpenJDK as far as I understand, but is obviously still present in GraalVM JDK.

[pompiuses]

Printout of cgroup of the container when running in Gitlab in the Kubernetes cluster.

$ cat /proc/self/cgroup
12:hugetlb:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
11:devices:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
10:perf_event:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
9:net_cls,net_prio:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
8:blkio:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
7:rdma:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
6:pids:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
5:memory:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
4:cpuset:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
3:freezer:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
2:cpu,cpuacct:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
1:name=systemd:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
0::/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee

[oubidar-Abderrahim]

Thank you for reaching out about this issue, we'll take a look into it shortly

[hxtmdev]

I get a similar stacktrace when trying to use a minimal java.net.http.HttpClient example compiled with the current (2023-09-06) versions of native-image 17/20 when running on a system with a /proc/self/cgroup with too many :s.

See babashka/babashka#1621 (comment)

[pompiuses]

I can confirm that this issue has been fixed in Oracle GraalVM JDK 21. I have not tested the CE version.