Allow setting GOMEMLIMIT and GOGC

[TomBillietKlarrio]

What is missing?
It seems it's not possible to set the GOMEMLIMIT and GOGC environment variables.
I could not find it anywhere the docs or the api from the prometheus CRD. So I don't think it's possible. If the CRD would allow to inject arbitrary env variables, it could be solved like this, but I could not find that either.

Why do we need it?
Avoid prometheus instances (especially long-running instances that consume a lot of memory) from getting OOMKilled. See also https://weaviate.io/blog/gomemlimit-a-game-changer-for-high-memory-applications for some information.

[slashpai]

You should be able to inject env variable using strategic merge patch https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/strategic-merge-patch.md

[slashpai]

Something like:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
spec:
  containers:
  - name: "prometheus"
    env:
    - name: A
      value: B

[TomBillietKlarrio]

Thanks. It was not clear from the docs I could do those things.
It seems to be working, not let's see if it fixes our OOM kills.

[simonpasquier]

This could be a nice addition to https://prometheus-operator.dev/docs/operator/strategic-merge-patch/

[TomBillietKlarrio]

Or even add it as an option to set it automatically. I now do this magic in helm to set it to 80% of the pod limit:

{{ $promMemory := .Values.prometheus.resources.memory }}
    resources:
      requests:
        cpu: {{ .Values.prometheus.resources.cpu }} 
        memory: {{ $promMemory }}
      limits:
        memory: {{ $promMemory }}
    containers:
    - name: "prometheus"
      env:
      - name: GOMEMLIMIT
        # set GOMEMLIMIT: https://weaviate.io/blog/gomemlimit-a-game-changer-for-high-memory-applications
        # this is a complex function to automatically set this to 80% of what the pod limit is.
        value: "{{ floor (mulf (mulf (int (trimSuffix "Gi" $promMemory)) 1024) 0.8) }}MiB"

(caveat: only works if you define memory limit with the unit Gi)