Getting 403 forbidden while creating namespaced pod #48024
Description
Apache Airflow version
3.0.0
If "Other Airflow 2 version" selected, which one?
No response
What happened?
ERROR - Task failed with exception source="task" error_detail=[{"exc_type":"ApiException","exc_value":"(403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'a20f9afa-d32d-44c6-93a7-5dd213f2ea29', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '7ea62e44-19c5-48a2-a560-ba8ca979b02c', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'de91312c-33da-49cb-8500-93b259a7b0a1', 'Date': 'Thu, 20 Mar 2025 16:21:27 GMT', 'Content-Length': '346'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"cowsay-statc-79n0g2sj\" is forbidden: exceeded quota: primitive-aurora-5047-default, requested: cpu=250m,memory=512Mi, used: cpu=1,memory=2Gi, limited: cpu=1,memory=2Gi","reason":"Forbidden","details":{"name":"cowsay-statc-79n0g2sj","kind":"pods"},"code":403}\n\n","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":582,"name":"run"},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":718,"name":"_execute_task"},{"filename":"/usr/local/lib/python3.12/site-packages/airflow/sdk/definitions/baseoperator.py","lineno":373,"name":"wrapper"},{"filename":"/home/astro/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py","lineno":583,"name":"execute"},{"filename":"/home/astro/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py","lineno":593,"name":"execute_sync"},{"filename":"/home/astro/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py","lineno":555,"name":"get_or_create_pod"},{"filename":"/usr/local/lib/python3.12/site-packages/tenacity/init.py","lineno":336,"name":"wrapped_f"},{"filename":"/usr/local/lib/python3.12/site-packages/tenacity/init.py","lineno":475,"name":"call"},{"filename":"/usr/local/lib/python3.12/site-packages/tenacity/init.py","lineno":376,"name":"iter"},{"filename":"/usr/local/lib/python3.12/site-packages/tenacity/init.py","lineno":398,"name":""},{"filename":"/usr/local/lib/python3.12/concurrent/futures/_base.py","lineno":449,"name":"result"},{"filename":"/usr/local/lib/python3.12/concurrent/futures/_base.py","lineno":401,"name":"__get_result"},{"filename":"/usr/local/lib/python3.12/site-packages/tenacity/init.py","lineno":478,"name":"call"},{"filename":"/home/astro/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py","lineno":373,"name":"create_pod"},{"filename":"/home/astro/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py","lineno":351,"name":"run_pod_async"},{"filename":"/home/astro/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py","lineno":343,"name":"run_pod_async"},{"filename":"/usr/local/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py","lineno":7356,"name":"create_namespaced_pod"},{"filename":"/usr/local/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py","lineno":7455,"name":"create_namespaced_pod_with_http_info"},{"filename":"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py","lineno":348,"name":"call_api"},{"filename":"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py","lineno":180,"name":"__call_api"},{"filename":"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py","lineno":391,"name":"request"},{"filename":"/usr/local/lib/python3.12/site-packages/kubernetes/client/rest.py","lineno":279,"name":"POST"},{"filename":"/usr/local/lib/python3.12/site-packages/kubernetes/client/rest.py","lineno":238,"name":"request"}]}]
What you think should happen instead?
No response
How to reproduce
Run the below Dag in k8s executor:
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import (
KubernetesPodOperator,
)
from airflow.configuration import conf
namespace = conf.get("kubernetes_executor", "NAMESPACE")
with DAG(
dag_id="kpo_mapped",
start_date=datetime(1970, 1, 1),
schedule=None,
tags=["taskmap"]
# render_template_as_native_obj=True,
) as dag:
KubernetesPodOperator(
task_id="cowsay_static",
name="cowsay_statc",
namespace=namespace,
image="docker.io/rancher/cowsay",
cmds=["cowsay"],
arguments=["moo"],
log_events_on_failure=True,
)
KubernetesPodOperator.partial(
task_id="cowsay_mapped",
name="cowsay_mapped",
namespace=namespace,
image="docker.io/rancher/cowsay",
cmds=["cowsay"],
log_events_on_failure=True,
).expand(arguments=[["mooooove"], ["cow"], ["get out the way"]])Operating System
Linux
Versions of Apache Airflow Providers
No response
Deployment
Other
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct