AIrflow Webserver only reading last 100 lines from the pod logs #21452
Description
Apache Airflow version
2.2.3 (latest released)
What happened
Hi I see when task is in running state, airflow reads pod logs but only last 100 lines.
*** Falling back to local log
*** Trying to get logs (last 100 lines) from worker pod XXX ***
What you expected to happen
Is there any configuration variable to increase this number. Why are we just limiting to 100, is there any specific reason to do that?
How to reproduce
It is a default behaviour.
I am using kubernetes executor.
https://github.com/apache/airflow/blob/main/airflow/utils/log/file_task_handler.py#L145-L154 these are the code lines
Operating System
Mac OS
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct