fix(amazon): EksPodOperator deferrable mode fails on remote triggerers by akhilesharora · Pull Request #63020 · apache/airflow

akhilesharora · 2026-03-07T00:05:04Z

Summary

Fix EksPodOperator with deferrable=True failing with 401 Unauthorized when the triggerer runs on a different host from the worker.

Root Cause: The kubeconfig exec block references a temp file path (/tmp/tmpXYZ) that only exists on the worker. When the trigger is serialized and sent to the triggerer, the exec block tries to source a file that doesn't exist.

Solution: Generate a kubeconfig with an embedded bearer token instead of an exec block with temp file references.

Changes

Added EksHook.generate_config_dict_for_deferral() - generates kubeconfig with embedded token
Override EksPodOperator.invoke_defer_method() to use token-based config for triggerer
Added comprehensive error handling for cluster lookup and token fetch failures
Added 5 new tests covering success and error scenarios

Security Considerations

Token is encrypted at rest (Fernet encryption in trigger serialization)
Token has short lifespan (~14 minutes for EKS)
Token is never logged
Robust error handling with actionable messages

Test Plan

test_generate_config_dict_for_deferral - verifies embedded token config
test_generate_config_dict_for_deferral_cluster_not_found - error handling
test_generate_config_dict_for_deferral_empty_token - security validation
test_generate_config_dict_for_deferral_token_fetch_failure - error handling
test_invoke_defer_method_generates_token_based_config - operator integration
All existing EKS tests pass

Closes #61736

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.5)

Generated-by: Claude Code (Opus 4.5) following the guidelines

boring-cyborg · 2026-03-07T00:05:08Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

SameerMesiah97

Looks fine. But there was one critical issue that I have commented on already. And I do have one other concern:

Since the EKS authentication token generated by fetch_access_token_for_cluster typically expires after approx. 15 minutes, I’m wondering how this behaves for longer-running triggers. If the trigger polls the Kubernetes API for longer than the token lifetime, could the embedded token expire and cause authentication failures?

CI needs to be run to see if there any other issues.

SameerMesiah97 · 2026-03-08T01:00:15Z

providers/amazon/src/airflow/providers/amazon/aws/hooks/eks.py

+
+        :param eks_cluster_name: The name of the cluster to generate kubeconfig for.
+        :param pod_namespace: The namespace to run within kubernetes.
+        :return: A kubeconfig dict with embedded bearer token.


This docstring is too long. The emphasis should be on description not justification. Something like this would be better:

""" Generate a kubeconfig dict with an embedded bearer token for deferrable execution. The token-based config avoids the exec credential plugin so it can be safely serialized and used by the triggerer process. :param eks_cluster_name: The name of the EKS cluster. :param pod_namespace: The Kubernetes namespace. :return: Kubeconfig dictionary with embedded bearer token. """

The additional content where you explain why this function exists might be better as a comment.

SameerMesiah97 · 2026-03-08T01:09:16Z

providers/amazon/src/airflow/providers/amazon/aws/hooks/eks.py

+            sts_url = f"{StsHook(region_name=session.region_name).conn_client_meta.endpoint_url}/?Action=GetCallerIdentity&Version=2011-06-15"
+        finally:
+            del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
+


Modifying environment variables is not acceptable here. os.environ is process-global so setting and deleting AWS_STS_REGIONAL_ENDPOINTS like this could interfere with other tasks running in the same worker process. It could also remove a value that was already set by the environment.

I am not sure why you need to manipulate environment variables because my understanding is that the url construction would default to 'regional' without explicitly setting AWS_STS_REGIONAL_ENDPOINTS to regional.

You're right, I've removed the env var manipulation entirely. Now constructing the regional STS URL directly: https://sts.{region}.amazonaws.com/.... Also applied this fix to the existing generate_config_file method for consistency.

SameerMesiah97 · 2026-03-08T01:11:07Z

providers/amazon/src/airflow/providers/amazon/aws/hooks/eks.py

+                sts_url=sts_url,
+                region_name=session.region_name,
+            )
+        except Exception as e:


The Exception here is too broad. Can you perhaps narrow it to the errors you would expect when invoking fetch_access_token_for_cluster ?

SameerMesiah97 · 2026-03-08T01:17:28Z

providers/amazon/src/airflow/providers/amazon/aws/operators/eks.py

+
+        This override generates a kubeconfig with an embedded bearer token instead of an exec
+        block, allowing the config to work on the triggerer without requiring local temp files.
+        """


This docstring is too long as well. I would suggest the below:

""" Override to generate a token-based kubeconfig for the triggerer. EKS kubeconfigs use an exec credential plugin that references temporary files created on the worker. These files are not available on the triggerer, so this override embeds a bearer token instead. """

Same as above, I would leave the truncated content to be included in a comment instead.

SameerMesiah97 · 2026-03-08T01:24:53Z

providers/amazon/src/airflow/providers/amazon/aws/operators/eks.py

+        import datetime
+
+        from airflow.providers.cncf.kubernetes.triggers.pod import ContainerState, KubernetesPodTrigger
+        from airflow.providers.common.compat.sdk import AirflowNotFoundException, BaseHook


Why are these imports within the function? Not necessarily an issue but can you explain why?

I placed them locally but they could be moved to module level to match the parent class pattern

SameerMesiah97 · 2026-03-08T01:27:42Z

providers/amazon/tests/unit/amazon/aws/hooks/test_eks.py

+
+        This test verifies that the method generates a kubeconfig dict with a bearer token
+        embedded directly (instead of an exec block that references temp files), allowing
+        the config to be serialized and used on the triggerer.


Remove everything from this docstring except the first line.

SameerMesiah97 · 2026-03-08T01:35:21Z

providers/amazon/tests/unit/amazon/aws/operators/test_eks.py

+        This test verifies that EksPodOperator.invoke_defer_method() generates a kubeconfig
+        with an embedded bearer token (instead of an exec block with temp file references)
+        so that the triggerer can authenticate without requiring files that only exist on the worker.
+        """


Remove everything from this docstring except the first line.

SameerMesiah97 · 2026-03-08T01:39:49Z

providers/amazon/newsfragments/63020.bugfix.rst

@@ -0,0 +1 @@
+Fix EksPodOperator deferrable mode failing on remote triggerers with 401 Unauthorized by embedding bearer token in kubeconfig instead of using exec block with temp file references


I am not sure if a news fragment is necessary for this but let's see what a committer/maintainer has to say.

Will defer to committer guidance on this. Included it as it's a user-facing bugfix

When using EksPodOperator with deferrable=True, the triggerer fails with 401 Unauthorized because credential temp files created on the worker don't exist on the triggerer host. The root cause is that the kubeconfig exec block references a temp file path that only exists on the worker. When the trigger is serialized and sent to the triggerer (on a different host), the exec block tries to source a file that doesn't exist. This fix adds a new method `generate_config_dict_for_deferral()` that creates a kubeconfig with an embedded bearer token instead of an exec block. The EksPodOperator.invoke_defer_method() is overridden to use this token-based config for the triggerer. Security considerations: - Token is encrypted at rest (Fernet encryption in trigger serialization) - Token has short lifespan (~14 minutes) - Token is never logged - Robust error handling with clear messages Closes apache#61736

- Remove os.environ manipulation for AWS_STS_REGIONAL_ENDPOINTS, construct regional STS URL directly to avoid process-global side effects - Trim verbose docstrings to focus on description rather than justification - Narrow exception handling from broad Exception to specific (BotoCoreError, ClientError, ValueError) - Add comment explaining why imports are inside invoke_defer_method - Apply same env var fix to existing generate_config_file method

akhilesharora · 2026-03-08T09:03:47Z

Looks fine. But there was one critical issue that I have commented on already. And I do have one other concern:

Since the EKS authentication token generated by fetch_access_token_for_cluster typically expires after approx. 15 minutes, I’m wondering how this behaves for longer-running triggers. If the trigger polls the Kubernetes API for longer than the token lifetime, could the embedded token expire and cause authentication failures?

CI needs to be run to see if there any other issues.

Yes, if the trigger polls for longer than ~14 minutes, the token could expire. Anticipating most pod operations (startup, completion monitoring) to finish well within this window, and trigger_reentry() generates fresh credentials when the trigger completes.

For very long-running pods, token refresh in the trigger could be a future enhancement.

akhilesharora requested a review from o-nikolas as a code owner March 7, 2026 00:05

boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Mar 7, 2026

akhilesharora force-pushed the fix/issue-61736-eks-deferrable-credentials branch from fdad29e to dc9c61f Compare March 7, 2026 00:08

SameerMesiah97 reviewed Mar 8, 2026

View reviewed changes

akhilesharora added 2 commits March 8, 2026 14:07

akhilesharora force-pushed the fix/issue-61736-eks-deferrable-credentials branch from dc9c61f to cfcd906 Compare March 8, 2026 08:39

Rename newsfragment to match PR number

9bd5807

		@@ -0,0 +1 @@
		Fix EksPodOperator deferrable mode failing on remote triggerers with 401 Unauthorized by embedding bearer token in kubeconfig instead of using exec block with temp file references

Conversation

akhilesharora commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Security Considerations

Test Plan

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg bot commented Mar 7, 2026

Uh oh!

SameerMesiah97 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akhilesharora commented Mar 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

akhilesharora commented Mar 7, 2026 •

edited

Loading