KEP-3721: Support for env files. #5384

HirazawaUi · 2025-06-06T14:47:44Z

One-line PR description: Support for env files.

Issue link: Add FileEnvSource and FileKeySelector to add environment generated on the fly #3721

Other comments: This PR replaces KEP-3721: Support for env files. #4524. The previous PR had too many commits, so I didn’t cherry-pick them. I copied and pasted the changes, while also refining some parts that were missed or left pending in the earlier PR.

toVersus · 2025-06-06T15:07:27Z

I copied and pasted the changes, while also refining some parts that were missed or left pending in the earlier PR.

If there are additions from the previous PR, it might be clearer to split the commit into two (one for the initial copy-paste and another for the subsequent updates) so that the diffs are easier to review.

EDIT: Sorry, please ignore this comment as it's been a while since the previous review. It's probably better to review everything from scratch.

SergeyKanzhelev · 2025-06-06T22:45:57Z

keps/sig-node/3721-support-for-env-files/README.md

+   additional ConfigMap or Secret API calls.
+
+2. The user is using a container offered by a vendor that requires configurating
+   environment variables (for eg, license key, one-time secret tokens). They can


I didn't notice the secret token here in scenarios. We need to ensure the security is covered and we will neither write those values in logs nor set them in the Pod status and such

I would prefer to leave it out of the motivation section for now, unless we can offer a reliable way to help users protect their sensitive information.

We will still support this use case, but we will clearly state in the documentation that it is not secure and recommend that users avoid using it.

WDYT?

keps/sig-node/3721-support-for-env-files/README.md

SergeyKanzhelev · 2025-06-06T22:57:59Z

keps/sig-node/3721-support-for-env-files/README.md

+    emptyDir: {}
+```
+
+#### Story 2


Please add User Story 3: Static Pods cannot use secrets as they cannot refer API server objects. However Pod spec is widely available in logs and audit messages. Using init container that copies environment variables from disk, allows to hide those environment variables.

I'm not adding it as a user story for now, for the same reason mentioned in my previous comment, to avoid giving users misleading guidance.
ref: #5384 (comment)

If we believe this use case is important, I’ll update the KEP to include it later.

If you're considering static pods, please add it as a beta requirement to make the final decision if you want to support it or not. If you know you're not planning to support static pods, please make it explicit adding to Non-Goals.

Static pods are an important use case for this feature (though, due to security concerns, we don’t want to document this in the user story). I expect static pods to be supported in the alpha phase. Are there any obstacles to this?

Are there any obstacles to this?

That's perfectly fine, I just want to make sure this is clearly stated in the document one way or the other.

SergeyKanzhelev · 2025-06-06T23:00:59Z

keps/sig-node/3721-support-for-env-files/README.md

+    // successfully created.
+    Key string `json:"key" protobuf:"bytes,3,opt,name=key"`
+    // Specify whether the file or its key must be defined. If the file or key
+    // does not exist, then the env var is not published.


what is the behavior otherwise? If optional == false and key does not present? Fail the container creation forever in the crash loop?

If optional is set to false, we require the user to define the variable in the env file; otherwise, the kubelet will return an error when generating the container configuration. As a result, the container will not be created, and the error will be returned before creation.

If this is optional and the file is not present or the key is not found, do we define the variable in the pod, with value "" or do we not define it at all? Spec it clearly and simply :)

Taking a quick glance at the code, I observed that if a corresponding key doesn't exist in the ConfigMap and optional is set to true, the code will skip processing this env using continue. The env won't be set for the Pod. We will adhere to this existing behavior.

SergeyKanzhelev · 2025-06-06T23:03:05Z

keps/sig-node/3721-support-for-env-files/README.md

+
+3. **Duplicate Names**: If an environment variable is defined multiple times in the file, the last occurrence takes precedence and overrides any previous values.
+
+4. **Size Limit**: To start with, the maximum allowed size for the env file will be 64KiB. Limits for key-value length will be added as a part of implementation after additional investigation.


please define some limits for key and values for the alpha here. Just come up with some values, we can adjust later

Added limits for key and values for the alpha.

How these limits apply to currently existing env vars? Do we have any?

In this KEP, I added restrictions on environment variables that only apply to the environment variables provided by this feature, without affecting the existing environment variable logic (such as those sourced from ConfigMap or Secret).

These restrictions are limited to the alpha phase. When this feature gate progresses to the beta phase, we will relax the environment variable restrictions to align with those sourced from ConfigMap or Secret.

I had some misunderstanding about this comment earlier. I've now added alpha-phase limitations for both keys and values as requested in the comment.

ref: #5384 (comment)

keps/sig-node/3721-support-for-env-files/README.md

SergeyKanzhelev · 2025-06-06T23:27:57Z

keps/sig-node/3721-support-for-env-files/README.md

+|7. The container's UID does not have permission to read the env file. | Pod created | Container fails to start and erorr message is reported in the events.|
+
+
+### Test Plan


Suggested change

### Test Plan

### Security Considerations

These environment variables may be used to store secrets. However the emptyDir will not

provide the same protection as Secrets Objects when mounted. This risk will be documented,

but no built-in protection will be offered.

### Test Plan

@HirazawaUi if you have idea how to secure this file better - please speak up.

I believe a better approach is to state in the documentation that the current best practice is to avoid defining secrets in env files, unless we have a clear mechanism that allows users to explicitly declare these variables as secrets and enables us to provide some level of protection for them.

keps/sig-node/3721-support-for-env-files/README.md

HirazawaUi · 2025-06-07T07:29:35Z

/cc @thockin @jpbetz

thockin · 2025-06-08T21:29:37Z

keps/sig-node/3721-support-for-env-files/README.md

+        fileKeyRef:
+          path: config.env
+          volumeName: config
+          key: CONFIG_VAR


Should the name here should default to the env-var's name if not specified? I suspect it will often be the same. I see that we do NOT do this for other sources, so may be not?

It seems that way, but I'm not sure if it's worth the effort for us to do this.

Taking a quick glance at the code, it doesn't appear we've implemented similar behavior for valueFrom before.

If we tell users here that they can omit defining fileKeyRef.key, and we'll use the environment variable name they defined instead, I'm concerned it might introduce unnecessary cognitive overhead.

I feel like it's a reasonable default but I won't make you be the pioneer.

keps/sig-node/3721-support-for-env-files/README.md

thockin · 2025-06-08T21:40:33Z

keps/sig-node/3721-support-for-env-files/README.md

+    // successfully created.
+    Key string `json:"key" protobuf:"bytes,3,opt,name=key"`
+    // Specify whether the file or its key must be defined. If the file or key
+    // does not exist, then the env var is not published.


If this is optional and the file is not present or the key is not found, do we define the variable in the pod, with value "" or do we not define it at all? Spec it clearly and simply :)

keps/sig-node/3721-support-for-env-files/README.md

HirazawaUi · 2025-06-09T16:44:41Z

@thockin All comments have been addressed.

thockin · 2025-06-09T16:50:56Z

/approve for API - @SergeyKanzhelev are you owning it for node?

SergeyKanzhelev

/lgtm
/approve

for official approval:
/assign @mrunalp

HirazawaUi · 2025-06-11T05:03:22Z

/cc @jpbetz For approval.

thockin · 2025-06-11T05:27:53Z

Joe is out this week. @deads2k @soltysh to cover?

BenTheElder · 2025-06-06T21:56:33Z

keps/sig-node/3721-support-for-env-files/README.md

+spec:
+  initContainers:
+  - name: setup-envfile
+    image: registry.k8s.io/busybox


note that this image is unpatched ...

This isn't a strict example. I'm not sure what our requirements are for this. If a strict example is needed, I can replace it.

soltysh · 2025-06-11T18:33:40Z

Joe is out this week. @deads2k @soltysh to cover?

I can pick it up, but you'll need to opt-in. Currently it seems #3721 is not opted for 1.34, so I can't pick that up.

thockin · 2025-06-11T20:03:27Z

Done

keps/prod-readiness/sig-node/3721.yaml

soltysh · 2025-06-15T16:47:50Z

keps/sig-node/3721-support-for-env-files/README.md

+...
+```
+
+2. **Variable Naming**: We will apply the same variable name [restrictions](https://github.com/kubernetes/kubernetes/blob/a7ca13ea29ba5b3c91fd293cdbaec8fb5b30cee2/pkg/apis/core/validation/validation.go#L2583-L2596) as other API-defined env vars, In the alpha phase, environment variable keys are restricted to ASCII characters matching the pattern `[-._a-zA-Z][-._a-zA-Z0-9]*`. This restriction will be relaxed in the beta phase.


I don't think that's completely correct, the relaxed env validation is related with the work happening in #4369. The fact that it's progressing to GA in 1.34 is orthogonal to this particular feature. So I'm not sure what kind of relaxed validation you're proposing here at beta?

We will enforce strict environment variable name validation (the original restrictions) during the alpha phase.
During the beta phase, we will apply the relaxed environment variable name validation enabled by the RelaxedEnvironmentVariableValidation feature gate.

RelaxedEnvironmentVariableValidation seemed was graduated. So we can start with less restricted version from the beginning. This will minimize changes between graduations

Ah, understood. Upon closer review, I realized I misinterpreted the intent of comment #5384 (comment).

I incorrectly assumed that character restrictions for environment variable names/values were required during alpha. I've now reverted those changes.

I've implemented size limitations (not character rules) for environment variables specifically for the alpha phase.

soltysh · 2025-06-15T16:50:06Z

keps/sig-node/3721-support-for-env-files/README.md

+
+2. **Variable Value**: Similarly, in the alpha phase, environment variable values are also limited to ASCII characters within the [-._a-zA-Z][-._a-zA-Z0-9]* range to prevent injection attacks caused by passing command control characters through environment variables. This restriction will also be lifted in the beta phase.
+
+3. **Duplicate Names**: If an environment variable is defined multiple times in the file, the last occurrence takes precedence and overrides any previous values.


👍 I've also checked our validation and I don't see a place where we verify duplicated env vars, it's semi-related to your proposal, but should we have one? What happens if I define the same env vars multiple times in pod spec, once it's from configmap, other from this file? Or am I missing something?

In-file duplicates:

If multiple environment variables with the same name are defined within a file, the later occurrence overrides the earlier one(s).

We cannot add validation logic for duplicates within ConfigMaps/Secrets/files because our validation logic has no access to the actual environment variable data stored inside these objects.

duplicates in Pod.Spec:

If a user defines duplicate environment variable keys in the Pod.Spec (e.g., one sourced from a ConfigMap and another from a file), the variable defined later in the list will override the one defined earlier. This occurs because the kubelet iterates over container.Env sequentially and assigns values in that order.

I wouldn't mind adding validation logic to detect duplicate environment variable definitions in the Pod.Spec. However, this could potentially break existing workloads that unintentionally rely on this behavior. Perhaps we should instead emit a warning to notify users that if they define duplicates this way, later-defined environment variables will override earlier ones.

Perhaps we should instead emit a warning to notify users that if they define duplicates this way, later-defined environment variables will override earlier ones.

That could be potentially useful. And thank you for clarifying the order of env var loading.

soltysh · 2025-06-15T16:51:26Z

keps/sig-node/3721-support-for-env-files/README.md

+
+3. **Duplicate Names**: If an environment variable is defined multiple times in the file, the last occurrence takes precedence and overrides any previous values.
+
+4. **Size Limit**: To start with, the maximum allowed size for the env file will be 64KiB. Limits for key-value length will be added as a part of implementation after additional investigation.


How these limits apply to currently existing env vars? Do we have any?

soltysh · 2025-06-15T16:54:35Z

keps/sig-node/3721-support-for-env-files/README.md

+    emptyDir: {}
+```
+
+#### Story 2


If you're considering static pods, please add it as a beta requirement to make the final decision if you want to support it or not. If you know you're not planning to support static pods, please make it explicit adding to Non-Goals.

keps/sig-node/3721-support-for-env-files/README.md

keps/sig-node/3721-support-for-env-files/kep.yaml

HirazawaUi · 2025-06-16T14:21:39Z

@soltysh Thank you so much for your review. I've addressed/resolved all the comments. Could you please take another look?

soltysh

This is sufficient for alpha
/approve
the PRR part

soltysh · 2025-06-18T11:15:51Z

keps/sig-node/3721-support-for-env-files/README.md

+    emptyDir: {}
+```
+
+#### Story 2


Are there any obstacles to this?

That's perfectly fine, I just want to make sure this is clearly stated in the document one way or the other.

soltysh · 2025-06-18T11:18:06Z

keps/sig-node/3721-support-for-env-files/README.md

+
+2. **Variable Value**: Similarly, in the alpha phase, environment variable values are also limited to ASCII characters within the [-._a-zA-Z][-._a-zA-Z0-9]* range to prevent injection attacks caused by passing command control characters through environment variables. This restriction will also be lifted in the beta phase.
+
+3. **Duplicate Names**: If an environment variable is defined multiple times in the file, the last occurrence takes precedence and overrides any previous values.


Perhaps we should instead emit a warning to notify users that if they define duplicates this way, later-defined environment variables will override earlier ones.

That could be potentially useful. And thank you for clarifying the order of env var loading.

soltysh · 2025-06-18T11:21:16Z

keps/sig-node/3721-support-for-env-files/README.md

+
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+
+We have added the `fileKeyRef` data structure to `podSpec`, which will undoubtedly increase the size of the Pod API. The increase in size depends on the number of environment variables defined by the user, making it impossible to estimate the exact number of bytes added. However, this should not have a significant impact on the API. For many users, it merely involves migrating environment variables previously defined in `ConfigMap/Secret` to this new structure.


I'll let this go for alpha, but for beta this will need a proper estimation, like all the other features.

k8s-ci-robot · 2025-06-18T11:23:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HirazawaUi, mrunalp, SergeyKanzhelev, soltysh, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [soltysh]
~~keps/sig-node/OWNERS~~ [mrunalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

soltysh · 2025-06-18T11:24:11Z

This was previously tagged by Sergey and Mrunal, so
/lgtm

/label tide/merge-method-squash

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 6, 2025

k8s-ci-robot requested review from dchen1107 and kikisdeliveryservice June 6, 2025 14:47

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/node Categorizes an issue or PR as relevant to SIG Node. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 6, 2025

HirazawaUi force-pushed the kep-3721 branch from bb79f25 to f79b19c Compare June 6, 2025 14:57

SergeyKanzhelev reviewed Jun 6, 2025

View reviewed changes

keps/sig-node/3721-support-for-env-files/README.md Outdated Show resolved Hide resolved

SergeyKanzhelev reviewed Jun 6, 2025

View reviewed changes

keps/sig-node/3721-support-for-env-files/README.md Outdated Show resolved Hide resolved

SergeyKanzhelev reviewed Jun 6, 2025

View reviewed changes

keps/sig-node/3721-support-for-env-files/README.md Show resolved Hide resolved

SergeyKanzhelev reviewed Jun 6, 2025

View reviewed changes

keps/sig-node/3721-support-for-env-files/README.md Show resolved Hide resolved

SergeyKanzhelev reviewed Jun 6, 2025

View reviewed changes

keps/sig-node/3721-support-for-env-files/README.md Show resolved Hide resolved

SergeyKanzhelev reviewed Jun 6, 2025

View reviewed changes

toVersus reviewed Jun 7, 2025

View reviewed changes

keps/sig-node/3721-support-for-env-files/README.md Show resolved Hide resolved

keps/sig-node/3721-support-for-env-files/README.md Outdated Show resolved Hide resolved

HirazawaUi force-pushed the kep-3721 branch 2 times, most recently from c2dcff1 to 5dfc934 Compare June 7, 2025 02:27

k8s-ci-robot requested review from jpbetz and thockin June 7, 2025 07:29

thockin reviewed Jun 8, 2025

View reviewed changes

thockin mentioned this pull request Jun 9, 2025

API can be streamlined: assume same key for some EnvVarSource kubernetes/kubernetes#132195

Open

HirazawaUi force-pushed the kep-3721 branch from 5dfc934 to d61dc5b Compare June 9, 2025 16:36

add support env files

74ce411

HirazawaUi force-pushed the kep-3721 branch from d61dc5b to 74ce411 Compare June 9, 2025 16:42

thockin assigned thockin and SergeyKanzhelev Jun 9, 2025

SergeyKanzhelev approved these changes Jun 9, 2025

View reviewed changes

k8s-ci-robot assigned mrunalp Jun 9, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2025

mrunalp approved these changes Jun 9, 2025

View reviewed changes

BenTheElder reviewed Jun 11, 2025

View reviewed changes

SergeyKanzhelev mentioned this pull request Jun 12, 2025

Add FileEnvSource and FileKeySelector to add environment generated on the fly #3721

Open

4 tasks

soltysh reviewed Jun 15, 2025

View reviewed changes

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 16, 2025

HirazawaUi force-pushed the kep-3721 branch from 6dc5fd1 to ec7a053 Compare June 17, 2025 00:24

Fix prr's comment

845b8af

HirazawaUi force-pushed the kep-3721 branch from ec7a053 to 845b8af Compare June 18, 2025 00:15

soltysh approved these changes Jun 18, 2025

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 18, 2025

k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Jun 18, 2025

k8s-ci-robot assigned soltysh Jun 18, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 18, 2025

k8s-ci-robot merged commit 2f55286 into kubernetes:master Jun 18, 2025
4 checks passed

k8s-ci-robot added this to the v1.34 milestone Jun 18, 2025


		3. Duplicate Names: If an environment variable is defined multiple times in the file, the last occurrence takes precedence and overrides any previous values.

		4. Size Limit: To start with, the maximum allowed size for the env file will be 64KiB. Limits for key-value length will be added as a part of implementation after additional investigation.

		\|7. The container's UID does not have permission to read the env file. \| Pod created \| Container fails to start and erorr message is reported in the events.\|


		### Test Plan

-### Test Plan
+### Security Considerations
+These environment variables may be used to store secrets. However the emptyDir will not
+provide the same protection as Secrets Objects when mounted. This risk will be documented,
+but no built-in protection will be offered.
+### Test Plan


		2. Variable Value: Similarly, in the alpha phase, environment variable values are also limited to ASCII characters within the [-._a-zA-Z][-._a-zA-Z0-9]* range to prevent injection attacks caused by passing command control characters through environment variables. This restriction will also be lifted in the beta phase.

		3. Duplicate Names: If an environment variable is defined multiple times in the file, the last occurrence takes precedence and overrides any previous values.


		###### Will enabling / using this feature result in increasing size or count of the existing API objects?

		We have added the `fileKeyRef` data structure to `podSpec`, which will undoubtedly increase the size of the Pod API. The increase in size depends on the number of environment variables defined by the user, making it impossible to estimate the exact number of bytes added. However, this should not have a significant impact on the API. For many users, it merely involves migrating environment variables previously defined in `ConfigMap/Secret` to this new structure.

KEP-3721: Support for env files. #5384

KEP-3721: Support for env files. #5384

Conversation

HirazawaUi commented Jun 6, 2025

Uh oh!

toVersus commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HirazawaUi Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SergeyKanzhelev Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HirazawaUi commented Jun 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HirazawaUi Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HirazawaUi commented Jun 9, 2025

Uh oh!

thockin commented Jun 9, 2025

Uh oh!

toVersus commented Jun 6, 2025 •

edited

Loading

HirazawaUi Jun 7, 2025 •

edited

Loading

SergeyKanzhelev Jun 6, 2025 •

edited

Loading

HirazawaUi Jun 9, 2025 •

edited

Loading