Trigger logs unavailable using KubernetesPodOperator in defferable mode #33085
Description
Apache Airflow version
2.6.3
What happened
locally (astro dev start):
*** Could not read served logs: Client error '404 NOT FOUND' for url 'http://d19d312ee147:8794/log/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:08:36.472735+00:00/task_id=kpo_task/map_index=0/attempt=1.log.trigger.4.log'
then after the KPO completes, that line changes to
*** Found local files:
*** * /usr/local/airflow/logs/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:08:36.472735+00:00/task_id=kpo_task/map_index=0/attempt=1.log
*** * /usr/local/airflow/logs/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:08:36.472735+00:00/task_id=kpo_task/map_index=0/attempt=1.log.trigger.4.log
and I can see the one line from the triggerer.4.log in the UI:
[2023-08-02T15:09:41.104+0000] {triggerer_job_runner.py:608} INFO - Trigger example_dynamic_task_mapping_async/manual__2023-08-02T15:08:36.472735+00:00/kpo_task/0/1 (ID 17) fired: TriggerEvent<{'status': 'done', 'namespace': 'default', 'pod_name': 'kpo-task-74ru0c3a'}>
on AWS:
*** Could not read served logs: Client error '404 NOT FOUND' for url 'http://172.21.210.239:8794/log/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:23:10.766353+00:00/task_id=kpo_task/map_index=1/attempt=1.log.trigger.122.log'
172.21.210.239 is the IP of the triggerer pod. exec’d onto it and there is indeed no file at that path, there’s not even a dag_id... dir
but no mention of the trigger log line upon task completion:
172.21.153.159
*** Found logs in s3:
*** * s3://airflow-logs-clj7iifjn00i301muxstkk3qt/clklbvzr17992245s0cxefxljjr/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:23:10.766353+00:00/task_id=kpo_task/map_index=1/attempt=1.log
On GCP, during task execution I get the same *** Could not read served logs: Client error '404 NOT FOUND' error, but once the task completes I can then see
172.21.4.107
*** Found remote logs:
*** * gs://airflow-logs-cliuf5mwb01o901l4vf3lcihz/clktwflrz4100034zvyen3adssu/dag_id=example_dynamic_task_mapping_async/run_id=scheduled__2023-08-01T15:50:58.184208+00:00/task_id=kpo_task/map_index=0/attempt=1.log
*** * gs://airflow-logs-cliuf5mwb01o901l4vf3lcihz/clktwflrz4100034zvyen3adssu/dag_id=example_dynamic_task_mapping_async/run_id=scheduled__2023-08-01T15:50:58.184208+00:00/task_id=kpo_task/map_index=0/attempt=1.log.trigger.4.log
with the single trigger message:
[2023-08-02, 15:52:35 UTC] {triggerer_job_runner.py:616} INFO - Trigger example_dynamic_task_mapping_async/scheduled__2023-08-01T15:50:58.184208+00:00/kpo_task/0/1 (ID 9) fired: TriggerEvent<{'status': 'done', 'namespace': 'traditional-waxing-1670', 'pod_name': 'kpo-task-7o06jv52'}>
What you think should happen instead
Per PR #27758 the task logs view in the UI should show the trigger log output both while the logs are local and the task is in-flight, and after completion, once logs are uploaded to whatever remote_logging provider is being used.
How to reproduce
Attached an example dag with mapped and deferred KPOA
example_dynamic_mapping_dag_async.py.txt
Operating System
debian
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.3.1
apache-airflow-providers-celery==3.2.1
apache-airflow-providers-cncf-kubernetes==7.3.0
apache-airflow-providers-common-sql==1.6.0
apache-airflow-providers-datadog==3.3.1
apache-airflow-providers-elasticsearch==4.5.1
apache-airflow-providers-ftp==3.4.2
apache-airflow-providers-google==10.0.0
apache-airflow-providers-http==4.5.0
apache-airflow-providers-imap==3.2.2
apache-airflow-providers-microsoft-azure==6.2.1
apache-airflow-providers-postgres==5.5.2
apache-airflow-providers-redis==3.2.1
apache-airflow-providers-sqlite==3.4.2
Deployment
Astronomer
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct