Renovate OOM issues on Kubernetes

[faenschi]

How are you running Renovate?

None

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Gitlab and Renovate 41.96.1

Please tell us more about your question or problem

Hello,

we are running Renovate on Kubernetes and for the past 2–3 weeks we have frequently encountered OOM (Out of Memory) errors.

Here is an example of the logs:

<--- Last few GCs --->
[9:0x26121000]    46971 ms: Scavenge (reduce) (interleaved) 1022.2 (1041.2) -> 1021.6 (1041.5) MB, pooled: 0 MB, 3.46 / 0.00 ms  (average mu = 0.221, current mu = 0.227) allocation failure;
[9:0x26121000]    47251 ms: Mark-Compact (reduce) 1022.4 (1041.5) -> 1021.6 (1042.0) MB, pooled: 0 MB, 278.56 / 0.00 ms  (+ 0.2 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 319 ms) (average mu = 0.254, current
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 0xe1803a node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/usr/local/renovate/node]
2: 0x11ec9c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/renovate/node]
3: 0x11ecc97 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/renovate/node]
...
/usr/local/sbin/renovate: line 13:     9 Aborted                 /usr/local/renovate/node --use-openssl-ca "${RENOVATE_NODE_ARGS[@]}" /usr/local/renovate/dist/renovate.js "$@"

The problem occurs most often in repositories that use Helm charts, but not exclusively.

Has anyone experienced similar issues or can provide hints on how to address this?

Any help is greatly appreciated.

Best regards,
Angela

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[hocinehacherouf]

Hi @faenschi

I experienced a similar issue back in April. In my case, increasing the memory limit on the Renovate pod resolved it:

resources:
  limits:
    memory: 2Gi

Since then I haven’t seen any more OOM errors.

Out of curiosity, what memory limit are you currently setting for your Renovate pods?

[faenschi]

hello @hocinehacherouf

at the moment i have the following memory limits set:

              resources:
                limits:
                  memory: "2Gi"
                  ephemeral-storage: "2Gi"

i also tried the following node option:

                - name: RENOVATE_NODE_ARGS
                  value: "--max-old-space-size=4096"

But it didnt help much :-/

[carlos-lehmann]

You define 4GB on node args but limit the pod to 2GB?
In my experience renovate needs at least 8GB, have a try and play around with the limits, see if the behaviour changes.

[faenschi]

@carlos-lehmann I changed that as well (posted the old config :-( ) ...yes i will try and see what happens