Allow setting an environment variable to some fraction or multiple of a resource field

[dobesv]

What would you like to be added:

It would be nice to pass a process a number that is a bit lower than the container memory limit so that its garbage collection or memory management can use that limit.

For example:

• setting GOMEMLIMIT or setting NODEMEMLIMIT
• setting NODE_OPTS=--max-old-space-size=$(NODEMEMLIMIT) would be useful.
• setting JAVA_MEM_LIMIT and passing -Xmx$(JAVA_MEM_LIMIT)M

Currently you can use envFrom.resourceFieldRef to "copy" a resource request or limit to an environment variable, but you can only multiply the value by some power of 1000 or 1024 by specifying divisor.

Why is this needed:

This would make it possible to pass a memory limit into a process inside the container that is based on the container resource request or limit but less.

For example, when specifying node's --max-old-space-size or Java's -Xmx parameter, setting it equal to the memory limit of the container will be a problem, because the process actually uses more memory than what you provided as a parameter.

Potential Solutions

• If we could specify a divisor like 1.5Mi then it would result in a number of megabytes equal to roughly 2 thirds of the available memory, leaving some extra for the other heaps.
• Add another field like scale or multiplier to enfFrom.resourceFieldRef which is multiplied by the input value to get the env var value
• Allow arithmetic in the existing env var expansion system - where currently you can only do $(SOME_VAR) in the command line, additionally support something like $(SOME_VAR - 100)
• Allow using a go template or some simple templating/expression language to actually fill out the env var with prefix and suffix, like --max-old-space-size={{ value * 800 }}M

Workarounds

• Create an admission webhook that does one of the above without changing the kubernetes core features
• If the container uses bash, you could run a command like bash -c 'NODE_OPTIONS=---max-old-space-size=$(($MEMORY_LIMIT-500)) ... or create your own entrypoint wrapper (maybe a shell script) that calculates the env vars before running the actual command

[dobesv]

/sig apps

[kow3ns]

/remove-sig apps

[kow3ns]

/sig api-machinery

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[JohnRusk]

/reopen

[k8s-ci-robot]

@JohnRusk: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[JohnRusk]

@dobesv I think there's an even stronger reason to support this change, now that GOMEMLIMIT is common (i.e. lots of Go Apps are now built on GO 1.19 or later). I left a comment here https://kubernetes.slack.com/archives/C0EG7JC6T/p1682654871448859 about it.

Mods: For more info on GOMEMLIMIT and why this change is needed, see the end of this page: https://billglover.me/2022/09/14/use-the-kubernetes-downwards-api-to-set-gomemlimit/

[lavalamp]

/remove-sig api-machinery
/sig apps

[JohnRusk]

Hi SIG Apps. Lavalamp pointed out a good point, which is that changing the sematics of an existing field may be undesirable, e.g. may make version rollbacks promblematic. Any thoughts on whether this should be solved by adding a new field, maybe scale which wouhld be a floating point number?

[dobesv]

/reopen

[k8s-ci-robot]

@dobesv: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sylr]

/reopen

[k8s-ci-robot]

@sylr: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sylr]

@dobesv I think there's an even stronger reason to support this change, now that GOMEMLIMIT is common (i.e. lots of Go Apps are now built on GO 1.19 or later). I left a comment here https://kubernetes.slack.com/archives/C0EG7JC6T/p1682654871448859 about it.

Mods: For more info on GOMEMLIMIT and why this change is needed, see the end of this page: https://billglover.me/2022/09/14/use-the-kubernetes-downwards-api-to-set-gomemlimit/

Exactly this.

We need to be able to do something like this:

- name: GOMEMLIMIT
  valueFrom:
    resourceFieldRef:
      resource: limits.memory
      divisor: "1100m"

[sftim]

/reopen
/priority backlog
/sig node
/remove-lifecycle rotten
/lifecycle stale

How about using CEL to perform the transformation? Or a limited subset of CEL.

[k8s-ci-robot]

@sftim: Reopened this issue.

In response to this:

/reopen
/priority backlog
/sig node
/remove-lifecycle rotten
/lifecycle stale

How about using CEL to perform the transformation? Or a limited subset of CEL.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sftim]

/remove-sig apps

[2rs2ts]

I see most of the conversation here has been about memory, but I would like to ask that all resource fields be available for this kind of tweaking (in case y'all were not already thinking that.) For example, CPU throttling issues with languages that don't really see the container's cpu limit and/or just behave poorly when the limits are set. Being able to set GOMAXPROCS without having to rely on external templating logic, or on an entrypoint script to compute what the value should be, would be a huge boon for my company and probably a lot of others.

Another use case I can see (although I don't think people should write cloud-native code this way) is apps that log to multiple files inside the container's emphemeral storage; being able to set app log rotation parameters based on available disk without an entrypoint script or external templating logic would probably be greatly appreciated by people working with code that does that.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[2rs2ts]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[frittentheke]

/remove-lifecycle rotten

[dataviruset]

If this helps anyone: this is what I did in Helm to work-around the missing native functionality in Kubernetes and use 90% of the memory limit configured in values.yaml as the GOMEMLIMIT. Just change mulf $valueMi 0.9 to something other than 0.9 if you want a different percentage. Please note that it only supports Gi and Mi units.

{{- define "goMemLimitEnv" -}}
{{- $memory := .Values.resources.limits.memory -}}
{{- if $memory -}}
- name: GOMEMLIMIT
  {{- $value := regexFind "^\\d*\\.?\\d+" $memory | float64 -}}
  {{- $unit := regexFind "[A-Za-z]+" $memory -}}
  {{- $valueMi := 0.0 -}}
  {{- if eq $unit "Gi" -}}
    {{- $valueMi = mulf $value 1024 -}}
  {{- else if eq $unit "Mi" -}}
    {{- $valueMi = $value -}}
  {{- end -}}
  {{- $percentageValue := int (mulf $valueMi 0.9) }}
  value: {{ printf "%dMiB" $percentageValue -}}
{{- end -}}
{{- end -}}

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: example
spec:
  serviceName: example
  template:
    spec:
      containers:
        - name: example
          image: alpine:latest
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: 100m
              memory: "200Mi"
            limits:
              cpu: 1
              memory: "200Mi"
          env:
            {{- include "goMemLimitEnv" . | nindent 12 }}
            # Will be set to "180MiB"

[frittentheke]

I see most of the conversation here has been about memory, but I would like to ask that all resource fields be available for this kind of tweaking (in case y'all were not already thinking that.) For example, CPU throttling issues with languages that don't really see the container's cpu limit and/or just behave poorly when the limits are set. Being able to set GOMAXPROCS without having to rely on external templating logic, or on an entrypoint script to compute what the value should be, would be a huge boon for my company and probably a lot of others.

Another use case I can see (although I don't think people should write cloud-native code this way) is apps that log to multiple files inside the container's emphemeral storage; being able to set app log rotation parameters based on available disk without an entrypoint script or external templating logic would probably be greatly appreciated by people working with code that does that.

I second the call for generalization of this feature. Isn't there enough templating (gotemplate FTW?) capability that could be leveraged to allow for templating some env variables? This would also align nicely with Helm and what people do about resources there.