Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native image compilation fails in Gitlab container environment. #7182

Closed
pompiuses opened this issue Aug 11, 2023 · 8 comments
Closed

Native image compilation fails in Gitlab container environment. #7182

pompiuses opened this issue Aug 11, 2023 · 8 comments
Assignees

Comments

@pompiuses
Copy link

pompiuses commented Aug 11, 2023

Describe the issue
I'm trying to compile a native image in our build pipeline which is running in a container in Gitlab which in turn runs inside our Kubernetes cluster. The native-image tool crashes with the stacktrace below. I've tried using two different container images with both the GraalVM CE JDK and the Oracle GraalVM JDK and get errors in both cases. When running the containers locally in Docker on my Ubuntu machine everything works fine.

Steps to reproduce the issue
Setup a build pipeline in Gitlab. Our Gitlab instance runs in Kubernetes. Then create container images (Dockerfiles) which can be used to build the application inside Gitlab. I'm using these two Dockerfiles:
GraalVM CE JDK Dockerfile
Oracle GraalVM JDK Dockerfile

Then inside the running container in the Gitlab pipeline first build a fat jar of your application and then use the native-image tool to compile it. I'm using the following parameters:

native-image -jar target/my-app-1.0-SNAPSHOT-jar-with-dependencies.jar -H:ConfigurationFileDirectories=graalvm/tracing-agent -o target/app --no-fallback -H:+AddAllCharsets --enable-preview -H:+ReportExceptionStackTraces -H:+StaticExecutableWithDynamicLibC -R:MinHeapSize=128m -R:MaxHeapSize=768m

The native-image tool then immediately crashes with stacktrace below.

Describe GraalVM and your environment:

  • A container using the Dockerfiles above which is running in a Gitlab build pipeline which in turn runs inside a Kubernetes cluster which in turn runs on VMware which in turn runs on physical hardware.
  • GraalVM version: I've tested both GraalVM CE 20.0.2+9.1 and Oracle GraalVM 20.0.2+9.1
  • JDK major version: 20
  • OS: See Dockerfiles above. Debian and Oracle Linux.
  • Architecture: What the Gitlab container environment is running on I don't know. The container images are built using the Dockerfiles above on my local Ubuntu 18.04 machine using AMD64.

More details
Stacktrace when building with the Oracle GraalVM JDK:

Exception in thread "main" java.lang.InternalError: java.lang.reflect.InvocationTargetException
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:67)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Container.metrics(Container.java:44)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.ContainerInfo.<init>(ContainerInfo.java:34)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.Containers.memoryLimitInBytes(Containers.java:126)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.heap.PhysicalMemory.doInitialize(PhysicalMemory.java:145)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.heap.PhysicalMemory.size(PhysicalMemory.java:88)
	at java.base@20.0.2/java.lang.Runtime.maxMemory(Runtime.java:919)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.DirectMemoryAccessors.initialize(Target_jdk_internal_misc_VM.java:118)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.DirectMemoryAccessors.getPageAlignDirectMemory(Target_jdk_internal_misc_VM.java:100)
	at java.base@20.0.2/jdk.internal.misc.VM.isDirectMemoryPageAligned(VM.java:156)
	at java.base@20.0.2/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:124)
	at java.base@20.0.2/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:343)
	at java.base@20.0.2/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
	at java.base@20.0.2/sun.nio.ch.IOUtil.read(IOUtil.java:303)
	at java.base@20.0.2/sun.nio.ch.IOUtil.read(IOUtil.java:283)
	at java.base@20.0.2/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:234)
	at java.base@20.0.2/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:74)
	at java.base@20.0.2/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
	at java.base@20.0.2/java.io.InputStream.read(InputStream.java:217)
	at java.base@20.0.2/java.util.Properties$LineReader.readLine(Properties.java:504)
	at java.base@20.0.2/java.util.Properties.load0(Properties.java:420)
	at java.base@20.0.2/java.util.Properties.load(Properties.java:409)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.loadProperties(NativeImage.java:2074)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.loadProperties(NativeImage.java:2063)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption.<init>(MacroOption.java:380)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption.create(MacroOption.java:370)
	at java.base@20.0.2/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base@20.0.2/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base@20.0.2/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base@20.0.2/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base@20.0.2/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1921)
	at java.base@20.0.2/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base@20.0.2/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base@20.0.2/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base@20.0.2/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base@20.0.2/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption$Registry.collectMacroOptions(MacroOption.java:184)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption$Registry.addMacroOptionRoot(MacroOption.java:200)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.<init>(NativeImage.java:750)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.build(NativeImage.java:1577)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1557)
	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.main(NativeImage.java:1531)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base@20.0.2/java.lang.reflect.Method.invoke(Method.java:578)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:63)
	... 41 more
Caused by: java.lang.ExceptionInInitializerError
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:78)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupMetrics.getInstance(CgroupMetrics.java:164)
	... 43 more
Caused by: java.lang.NullPointerException
	at java.base@20.0.2/java.util.Objects.requireNonNull(Objects.java:233)
	at java.base@20.0.2/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:297)
	at java.base@20.0.2/java.nio.file.Path.of(Path.java:148)
	at java.base@20.0.2/java.nio.file.Paths.get(Paths.java:69)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupUtil.lambda$readStringValue$0(CgroupUtil.java:57)
	at java.base@20.0.2/java.security.AccessController.executePrivileged(AccessController.java:147)
	at java.base@20.0.2/java.security.AccessController.doPrivileged(AccessController.java:571)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupUtil.readStringValue(CgroupUtil.java:59)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:66)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemController.getLongValue(CgroupSubsystemController.java:125)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.getLongValue(CgroupV1Subsystem.java:269)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.getHierarchical(CgroupV1Subsystem.java:215)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.setSubSystemControllerPath(CgroupV1Subsystem.java:203)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.initSubSystem(CgroupV1Subsystem.java:111)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.<clinit>(CgroupV1Subsystem.java:47)
	... 45 more

Stacktrace when building with the GraalVM CE JDK:

Fatal error: Failed while initializing the performance data.: java.lang.InternalError: java.lang.reflect.InvocationTargetException
    at com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:67)
    at com.oracle.svm.core.containers.Container.metrics(Container.java:44)
    at com.oracle.svm.core.ContainerInfo.<init>(ContainerInfo.java:34)
    at com.oracle.svm.core.Containers.activeProcessorCount(Containers.java:88)
    at java.lang.Runtime.availableProcessors(Runtime.java:337)
    at com.oracle.svm.core.jvmstat.SystemCounters.getAvailableProcessors(SystemCounters.java:186)
    at com.oracle.svm.core.jvmstat.SystemCounters.allocate(SystemCounters.java:124)
    at com.oracle.svm.core.jvmstat.PerfManager.allocate(PerfManager.java:138)
    at com.oracle.svm.core.jvmstat.PerfManager$PerfDataThread.initializeMemory(PerfManager.java:Exception in thread "main" 212)
    at com.oracle.svm.core.jvmstat.PerfManager$PerfDataThread.run(PerfManager.java:187)
    at com.oracle.svm.core.thread.PlatformThreads.java.lang.InternalError: java.lang.reflect.InvocationTargetException
threadStartRoutine(PlatformThreads.java:807)
    at com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:67)
(PosixPlatformThreads.java:210)	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Container.metrics(Container.java:44)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.ContainerInfo.<init>(ContainerInfo.java:34)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.Containers.memoryLimitInBytes(Containers.java:126)
Current timestamp: 1691678201732	at org.graalvm.nativeimage.builder/com.oracle.svm.core.heap.PhysicalMemory.doInitialize(PhysicalMemory.java:145)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.heap.PhysicalMemory.size(PhysicalMemory.java:88)
Printing Instructions (ip=0x	at java.base@20.0.2/java.lang.Runtime.maxMemory(Runtime.java:919)
000	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.DirectMemoryAccessors.initialize(Target_jdk_internal_misc_VM.java:118)
000	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.DirectMemoryAccessors.getPageAlignDirectMemory(Target_jdk_internal_misc_VM.java:100)
000049ea00):	at java.base@20.0.2/jdk.internal.misc.VM.isDirectMemoryPageAligned(VM.java:156)

  	at java.base@20.0.2/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:124)
0x0000000	at java.base@20.0.2/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:343)
00049e9e0: 	at java.base@20.0.2/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
0x00	at java.base@20.0.2/sun.nio.ch.IOUtil.read(IOUtil.java:303)
 0x	at java.base@20.0.2/sun.nio.ch.IOUtil.read(IOUtil.java:283)
41 0xc7 0x	at java.base@20.0.2/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:234)
87 0x4c 0x01	at java.base@20.0.2/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:74)
 0x00 	at java.base@20.0.2/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
0x00 	at java.base@20.0.2/java.io.InputStream.read(InputStream.java:217)
0xfe 0xfe 0xfe 0x7e 0x48	at java.base@20.0.2/java.util.Properties$LineReader.readLine(Properties.java:504)
 0x8b 0x	at java.base@20.0.2/java.util.Properties.load0(Properties.java:420)
7c 0x	at java.base@20.0.2/java.util.Properties.load(Properties.java:409)
24
  0x0000000	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.loadProperties(NativeImage.java:2074)
00049e9f0	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.loadProperties(NativeImage.java:2063)
: 0x20 	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption.<init>(MacroOption.java:380)
0x48 	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption.create(MacroOption.java:370)
0x8b 0x	at java.base@20.0.2/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
74 0x24 0x	at java.base@20.0.2/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
10 0x48	at java.base@20.0.2/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
 0x8b 0x	at java.base@20.0.2/java.util.Iterator.forEachRemaining(Iterator.java:133)
54 0x24	at java.base@20.0.2/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1921)
 0x18 0xe8 0x70 	at java.base@20.0.2/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
0xfc 0xff	at java.base@20.0.2/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
 0xff
  0x000	at java.base@20.0.2/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
00000	at java.base@20.0.2/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
0049ea00	at java.base@20.0.2/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
: 0x90	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption$Registry.collectMacroOptions(MacroOption.java:184)
 0xcc 0x	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.MacroOption$Registry.addMacroOptionRoot(MacroOption.java:200)
cc 0x	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.<init>(NativeImage.java:750)
cc 0xcc 0x	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.build(NativeImage.java:1577)
cc 0xcc	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1557)
 0xcc 	at org.graalvm.nativeimage.driver/com.oracle.svm.driver.NativeImage.main(NativeImage.java:1531)
0xcc 0xcc 0xcc Caused by: java.lang.reflect.InvocationTargetException
0xcc 0x	at java.base@20.0.2/java.lang.reflect.Method.invoke(Method.java:578)
cc 0xcc	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:63)
 0xcc	... 41 more
 0xcc
  0x0000Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem
00000	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:78)
049ea10: 0x	at org.graalvm.nativeimage.builder/com.oracle.svm.core.containers.CgroupMetrics.getInstance(CgroupMetrics.java:164)
48 0x	... 43 more
83 0xec 0x18 0x48 0xb8 0x18 0xde 0x9f 0x00 0x00 0x00 0x00 0x00 0x49 0x8d

Top of stack (sp=0x00007fdcf16fed20):
  0x00007fdcf16fed20: 0x00007fdcf2658140 0x00007fdcf2a96d30
  0x00007fdcf16fed30: 0x00007fdcf26adbe8 0x00007fdcf0d35188
  0x00007fdcf16fed40: 0x00000000004d8ae9 0x000000000053398a
  0x00007fdcf16fed50: 0x00007fdcf0d03e18 0x00000000004d8ae9
  0x00007fdcf16fed60: 0x0000000000502319 0x00007fdcf2baab08
  0x00007fdcf16fed70: 0x00007fdcf2a96e10 0x00000000004d8b6b
  0x00007fdcf16fed80: 0x00007fdcf16ff640 
@pompiuses
Copy link
Author

pompiuses commented Aug 14, 2023

If it's the case that GraalVM can't find available memory referencing java.lang.Runtime.maxMemory, or cpu cores for that matter with the CE JDK referencing java.lang.Runtime.availableProcessors, there should probably be separate input parameters to manually override these, or make GraalVM more robust with sensible default settings? That would atleast be the quickest and most user friendly way to have something working I think. The alternative is to tweak the server settings of the container environment which in many cases may not be possible.

@pompiuses
Copy link
Author

pompiuses commented Aug 17, 2023

Similar issues also posted for Kubernetes:
micrometer-metrics/micrometer#2800 (comment)
kubernetes/kubernetes#103944

Seems to be related to cgroup access with certain configurations in Kubernetes clusters. This bug has been fixed in OpenJDK as far as I understand, but is obviously still present in GraalVM JDK.

@pompiuses
Copy link
Author

Printout of cgroup of the container when running in Gitlab in the Kubernetes cluster.

$ cat /proc/self/cgroup
12:hugetlb:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
11:devices:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
10:perf_event:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
9:net_cls,net_prio:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
8:blkio:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
7:rdma:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
6:pids:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
5:memory:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
4:cpuset:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
3:freezer:/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
2:cpu,cpuacct:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
1:name=systemd:/system.slice/containerd.service/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee
0::/kubepods-pod1c7412bd_cd67_4758_bf39_f299f9090a66.slice:cri-containerd:f5a2c9866c3fc84d23ebe1ad78b50ebe81705442e3d3ca3481f2480398c0dcee

@oubidar-Abderrahim oubidar-Abderrahim self-assigned this Aug 18, 2023
@oubidar-Abderrahim
Copy link
Member

Thank you for reaching out about this issue, we'll take a look into it shortly

@hxtmdev
Copy link

hxtmdev commented Sep 6, 2023

I get a similar stacktrace when trying to use a minimal java.net.http.HttpClient example compiled with the current (2023-09-06) versions of native-image 17/20 when running on a system with a /proc/self/cgroup with too many :s.

See babashka/babashka#1621 (comment)

@pompiuses
Copy link
Author

I can confirm that this issue has been fixed in Oracle GraalVM JDK 21. I have not tested the CE version.

@szeiger
Copy link

szeiger commented Feb 12, 2024

What's the status for GraalVM 17? We still see the problem in the latest release even though the upstream fix was already made in 17.0.1. We didn't have the problem in GraalVM 22.3.1 for JDK 11.

@szeiger
Copy link

szeiger commented Mar 12, 2024

Answering my own question: The bug is still present in GraalVM 17. It was fixed in OpenJDK and the fix is in the labs JDK build used by GraalVM but GraalVM 17 uses its own version of the cgroups code. GraalVM 21 uses the JDK version so it benefits from the upstream fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants