Unable to run EksPodOperator since updating mwaa from 2.4.3 to 2.7.1

[0x26res]

While upgrading aws MWAA from 2.4.3 to 2.7.1, my EksPodOperator stopped working with this error:

HTTP response headers: HTTPHeaderDict({'Audit-Id': '7d78edc7-c595-4baa-8a54-', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '5c0c1226-e427-42a3-ab4c-', 'X-Kubernetes-Pf-Prioritylevel-Uid': '5c640447-3651-46ec-8ad2-', 'Date': 'Mon, 05 Feb 2024 00:15:24 GMT', 'Content-Length': '270'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"shared\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

I haven't changed any of the permissions, so I believe some changes in the EksPodOperator, KubernetesPodOperator or any hook they use is causing the issue.

I've been following these instructions though they are not completely relevant since they use the KubernetesPodOperator instead of the EksPodOperator

This upgrade took me:

• from apache-airflow-providers-cncf-kubernetes==4.4.0 to apache-airflow-providers-cncf-kubernetes==7.6.0.
• from apache-airflow-providers-amazon==6.0.0 to apache-airflow-providers-amazon==8.7.1

[kl4072]

were you able to resolve this?
I have similar issue with AKS and running 2.8.1 the service account has access to the namespace its deployed in, but it is trying to list default namespace which it wouldnt have access on in my case.

[meshok0]

We had a similar issue(message":"pods is forbidden: User "system:anonymous" cannot list resource "pods" in tasks logs) on MWAA + EKS on Fargate.
AWS support suggested to change the verbosity of the worker log group of the MWAA environment to WARNING or ERROR. Unbelievable, but this fixed the issue.

[0x26res]

Hi @meshok0, just to confirm, did you change the verbosity from the default (INFO) to WARNING and it worked?

[meshok0]

Yes

…

On Mon, Mar 18, 2024, 15:22 0x26res ***@***.***> wrote: Hi @meshok0 <https://github.com/meshok0>, just to confirm, did you change the verbosity from the default (INFO) to WARNING and it worked? — Reply to this email directly, view it on GitHub <#37178 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWJGWGGT6KHU7BQ7BAE5HLYY3L65AVCNFSM6AAAAABCZ6JALCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQMRXGE4DM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[0x26res]

Ok, I gave it a try, it works. Thanks for pointing it out.

There's a big caveat: you can't submit deferrable task. I'll create an issue.

[0x26res]

I've created an issue for the new problem I have encountered: #38280

[andrewhharmon]

Im facing this problem as well but I don't understand the fix. Why would changing the verbosity of the worker logs fix this.

[0x26res]

I don't have a good explanation either. It could be that the debug log tries to query Eks for more information and that's when the failure happens.

But I can confirm that changing the log level fixes the issue.

[evredinka]

I faced the same issue upgrading from version 2.0.2→2.7.2 and 2.0.2→2.8.1 and the issue still remained even after logging level adjustment.
The root cause in my case was outdated kube_config.yaml:

[andrewhharmon]

mine ended up being network related. Even though the EKS API Server is public facing, it was still taking a private route. So I had to open access in the security group. https://stackoverflow.com/questions/78638015/aws-mwaa-times-out-trying-to-connect-to-eks-api-server/78638114#78638114