Provide more insights about "Multiple Gradle daemons were used".

[artem-zinnatullin]

Sometimes when I run benchmarks/profiling it fails with

* Stopping daemons
java.lang.RuntimeException: Multiple Gradle daemons were used.
	at org.gradle.profiler.Main.checkPid(Main.java:461)
	at org.gradle.profiler.Main.runGradleScenario(Main.java:188)
	at org.gradle.profiler.Main.run(Main.java:80)
	at org.gradle.profiler.Main.main(Main.java:27)

But nowhere in stdout/stderr or profile.log can I find logs that indicate when and why Gradle daemon was started.

What's more weird is that I'm getting that in very isolated environment: a Docker container gets started for every profile/benchmark session.

(But I've also seen it in non-containerized runs).

Would be great if gradle-profiler could give more info about it, ideally it should be actionable :)

[oehme]

Make sure you're not sharing your Gradle user home between containers. By default the profiler will put it in the project directory. It contains the daemon registry and if that is shared by multiple containers running builds at the same time they will kill each others daemons.

The Gradle profiler shouldn't need more info on this, it should already be in the profile.log, because Gradle prints why it started a new daemon.

[artem-zinnatullin]

I do share Gradle user home to avoid Gradle and dependencies download on every run, but I don't run multiple containers at the same time…

And I've also seen it happening without containerization.

I'll re-investigate logs when I see it again!

[oehme]

Closing since there was no more feedback. Feel free to reopen if you have new insights.

[ZacSweers]

We see this somewhat regularly and the logs yield no information. We kill all daemons on the CI node before and after running the profiler as well.

* Stopping daemons
java.lang.RuntimeException: Multiple Gradle daemons were used.
	at org.gradle.profiler.GradleScenarioInvoker.checkPid(GradleScenarioInvoker.java:188)
	at org.gradle.profiler.GradleScenarioInvoker.doRun(GradleScenarioInvoker.java:117)
	at org.gradle.profiler.GradleScenarioInvoker.doRun(GradleScenarioInvoker.java:21)
	at org.gradle.profiler.ScenarioInvoker.run(ScenarioInvoker.java:17)
	at org.gradle.profiler.Main.invoke(Main.java:110)
	at org.gradle.profiler.Main.run(Main.java:76)
	at org.gradle.profiler.Main.main(Main.java:18)

* Results written to /home/ubuntu/actions-runner/_work/slack-android-ng/slack-android-ng/profile-out
  Scenario incremental_abi_app_kotlin using Gradle 6.8.2
  /home/ubuntu/actions-runner/_work/slack-android-ng/slack-android-ng/profile-out/benchmark.csv
  /home/ubuntu/actions-runner/_work/slack-android-ng/slack-android-ng/profile-out/benchmark.html

It happens after around the second warmup, and any daemon processes are coming from the profiler exec itself. Anything we could try to look for or check?

[ZacSweers]

We just gave up on trying to work around this and believe that Gradle's GC polling is currently flawed. We disable GC polling instead via setting the org.gradle.daemon.gc.polling.disabled system property to true and have been fine since.

[gabrielfeo]

Just ran into this problem myself. I imagine there's a lot of reasons a daemon can be stopped for but in my case it was a lower-than-necessary memory limit for the build.

profile.log usually mentions the cause, as @oehme said:

Daemon will be stopped at the end of the build after running out of JVM memory
Used daemon with pid 5592

The daemon logs may have more info as well: <gradle-user-home>/daemon/<gradle-version>/daemon-<pid-of-dead-daemon>.out.log (remember the default gradle-user-home profiler builds use is <project-dir>/gradle-user-home).

2022-02-04T20:55:32.968+0000 [DEBUG] [org.gradle.launcher.daemon.server.health.DaemonMemoryStatus] non-heap: GC rate: 0.00/s non-heap usage: 82%
2022-02-04T20:55:32.968+0000 [INFO] [org.gradle.launcher.daemon.server.health.LowNonHeapDaemonExpirationStrategy] Expiring Daemon due to JVM Metaspace space being exhausted
2022-02-04T20:55:32.968+0000 [DEBUG] [org.gradle.launcher.daemon.server.DaemonRegistryUpdater] Storing daemon stop event: after running out of JVM memory
2022-02-04T20:55:32.968+0000 [DEBUG] [org.gradle.launcher.daemon.registry.PersistentDaemonRegistry] Storing daemon stop event with timestamp 1644008132968

[juliocbcotta]

I am getting this too

* Stopping daemons
Running command /Users/julio.buenocotta/git/tests/gradle-user-home/wrapper/dists/gradle-7.4.2-bin/48ivgl02cpt2ed3fh9dbalvx8/gradle-7.4.2/bin/gradle --stop --gradle-user-home /Users/julio.buenocotta/git/tests/gradle-user-home
Stopping Daemon(s)
1 Daemon stopped
java.lang.RuntimeException: Multiple Gradle daemons were used.
	at org.gradle.profiler.GradleScenarioInvoker.checkPid(GradleScenarioInvoker.java:191)
	at org.gradle.profiler.GradleScenarioInvoker.run(GradleScenarioInvoker.java:154)
	at org.gradle.profiler.GradleScenarioInvoker.run(GradleScenarioInvoker.java:22)
	at org.gradle.profiler.Main.invoke(Main.java:126)
	at org.gradle.profiler.Main.run(Main.java:86)
	at org.gradle.profiler.Main.main(Main.java:25)

* Results written to /Users/julio.buenocotta/git/tests/baseline
  /Users/julio.buenocotta/git/tests/baseline/benchmark.csv
  /Users/julio.buenocotta/git/tests/baseline/benchmark.html

@ZacSweers How/where should I setup that properties? For my user? For gradle-profiler? For my project?

[kenyee]

"org.gradle.daemon.gc.polling.disabled" is a gradle property...looks like you set it via "-Porg.gradle.daemon.gc.polling.disabled=true" as a gradle param or via the GRADLE_OPTS env var

[kenyee]

I don't see anything in the checkpid code that would be affected by that flag though:
https://github.com/gradle/gradle-profiler/blob/df9ede45b5e4d21822b3bcbb38b58ef66d3deb05/src/main/java/org/gradle/profiler/GradleScenarioInvoker.java

Looks like a fairly simple pid comparison of gradle daemons...no way to opt out in that code...

[ZacSweers]

It's not a gradle property, it's a system property https://github.com/gradle/gradle/blob/32e9a659c465cea3d654512ebae8d6a5d2cc07ed/platforms/core-runtime/launcher/src/main/java/org/gradle/launcher/daemon/server/health/gc/DefaultGarbageCollectionMonitor.java#L30