Kernel out-of-memory + timeouts / related to Gradle wrapper update?

[snazy]

How are you running Renovate?

A Mend.io-hosted app

Which platform you running Renovate on?

GitHub.com

Which version of Renovate are you using?

41.173.1 + 42.16.1

Please tell us more about your question or problem

We see kernel-out-of-memory and timeout issues for a couple of days now.
Nothing has been changed in our renovate config or the dependencies (as renovate didn't run) in the meantime.

Before, Renovate completed within ~2 minutes, also running v41.173.1. The last log message from the kernel-OOM + timeout, trying to bump the Gradle wrapper, is in both cases:

DEBUG: Executing command (branch="renovate/gradle-9.x")
{
  "command": "./gradlew :wrapper --gradle-distribution-url https://services.gradle.org/distributions/gradle-9.2.1-all.zip --gradle-distribution-sha256-sum f86344275d1b194688dd330abf9f6f2344cd02872ffee035f2d1ea2fd60cf7f3"
}

Note that the Gradle setup is special, as we intentionally do not have the gradle-wrapper.jar in Git, following the ASF rules to not commit binary code into source repositories.

The sequence of jobs & timings:

Job ID	Completed (CET)	Duration	Version	Notes
019a900b-59fc-74b9-9872-2a3491489a1c	Nov 17, 2025, 05:36:56	2m 33s	41.173.1	No Gradle wrapper update
019a90e7-6635-7aef-a159-88e46b6cd52a	Nov 17, 2025, 13:14:36	1m 7s	41.173.1	No Gradle wrapper update
019a91c2-c92f-704a-b241-2173e81f3605	Nov 17, 2025, 16:30:21	21m 32s	41.173.1	Gradle wrapper update initiated (branch + PR created)
1d3cc8dd-2186-4d6d-96fc-321f730a683b	Nov 17, 2025, 20:46:16	7m 18s	41.173.1	kernel OOM, last message in the log as above
019aae1e-5a53-7bfa-8601-be38ea83900b	Nov 23, 2025, 08:14:10	30m 2s	42.16.1	timeout, last message in the log as above

Similar setups without the gradle-wrapper.jar work fine (Job IDs afb35fb5-05e0-4d07-a515-50e6828cdb20, 913e7459-a4a6-468e-89c4-ededb6fd15cd, 1eac31f5-71b3-427f-a8aa-fb6033bcfcc2) and got past the Gradle wrapper update, but those projects have way less dependencies.

Logs (if relevant)

Job 019a91c2-c92f-704a-b241-2173e81f3605
Job 1d3cc8dd-2186-4d6d-96fc-321f730a683b
Job 019aae1e-5a53-7bfa-8601-be38ea83900b

[jamietanna]

Apologies that you've been hitting this!

It's possible to request increased resource usage (#33617) - would you like us to apply that to repositories in https://developer.mend.io/github/projectnessie ?

[snazy]

Sure, that would be great! Thank you!

But I wonder what caused the rather sudden increase of renovate job runtime duration from ~2 minutes to 30+ minutes.

[jamietanna]

I think it's largely related to the fact that the JVM spawned for Gradle is getting inconsistent information about how much memory is available, so it's either giving itself far too much (and getting OOM killed) or far too little (and taking very long)

I'm capturing some notes internally to look at what we can do to tune behaviour of the Gradle JVM for these updates, so folks don't need to upgrade resources

(This may require a tweak to the Renovate CLI, and then how the Mend Developer Platform invokes Renovate)

[jamietanna]

You're now upgraded to the Community (OSS) plan, and I've kicked off a new job on https://developer.mend.io/github/projectnessie/nessie, which will take into account the new memory requirements

Now you're on the Community (OSS) plan, you're also entitled to:

• 2 concurrent jobs per organisation (i.e. 2 of your 8 repos on the org will process concurrently)
• double memory (6GB)
• access to Merge Confidence Workflows - you'll be able to use matchConfidence to group PRs based on the Merge Confidence

Please give us a shout if you have any feedback, in particular on Merge Confidence Workflows!

[snazy]

Thank you! I see a successful run (5700d5c8-ded9-4333-9215-37d6fdb99635), finished in 6m 55s.
It reports that it used 5.9 of 6.0GB.

[jamietanna]

We saw that with Backstage too, I wonder if that's because the JVM expands to use all the memory it can do, and if so, should be fixed when we properly tune this 🤞

[snazy]

I think it's not strictly required to even run Gradle.

You can get the checksum from https://services.gradle.org/distributions/gradle-${GRADLE_DIST_VERSION}-wrapper.jar.sha256 and wrapper jar from https://raw.githubusercontent.com/gradle/gradle/v${GRADLE_VERSION}/gradle/wrapper/gradle-wrapper.jar (example code here).

That however wouldn't completely help Renovate, as the gradlew[.bat] scripts are needed as well, and those are generated using some templates.

I don't know about the resources/environment you have, but would it be an option to run Gradle's wrapper task once "somewhere else", stash it and "just":tm: reuse the generated gradlew[.bat] scripts, so you don't have to run Gradle for every Gradle project?

[snazy]

Oh!!!!
Now I realize what's going on!

So, we have Gradle daemon JVM settings in our gradle.properties file set to 2G min + 4G max.
I suspect there are a bunch of other projects doing this.

But such a big heap isn't necessary to just call ./gradlew wrapper.

I think Renovate should call it like this:

./gradlew -Dorg.gradle.jvmargs="-Xms256m -Xmx256m" --no-daemon wrapper

This gives 256m to the Gradle daemon JVM. The Gradle wrapper JVM is (with the standard gradlew scripts) limited to 64m anyway.
I suspect it would also work with 128m instead of 256m.

So, our Gradle config likely blew up Renovate's pods. Just interesting that it starts to happen just now. But that's just life I guess.