KPO - get_logs - logs missing

[raphaelauv]

Apache Airflow version

2.9.1

What happened?

I tried with apache-airflow-providers-cncf-kubernetes 8.2.0 or 8.1.1

the KPO is missing some logs ( in defer or not )

example put this script in file src/job.py

import time

if __name__ == '__main__':
    for i in range(10):
        time.sleep(2)
        print("RUNNING")

FROM python:3.12-slim

RUN mkdir -p /opt/program
WORKDIR /opt/program

COPY src/job.py /opt/program/job.py

CMD ["python3","job.py"]

docker build -t job_example:0.1 .

from airflow.providers.cncf.kubernetes.utils.pod_manager import OnFinishAction
from pendulum import today
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

dag = DAG(
    dag_id="kpo_async_get_logs",
    schedule_interval="0 0 * * *",
    start_date=today("UTC").add(days=-1)
)

with dag:
    KubernetesPodOperator(
        task_id="task-two",
        namespace="default",
        kubernetes_conn_id="kubernetes_default",
        name="airflow-toto-pod",
        image="job_example:0.1",
        # deferrable=True,
        poll_interval=2,
        on_finish_action=OnFinishAction.DELETE_POD,
        get_logs=True
    )

the logs in airflow give

{pod.py:536} INFO - `try_number` of task_instance: 1
{pod.py:537} INFO - `try_number` of pod: 1
{pod_manager.py:374} WARNING - Pod not yet started: airflow-toto-pod-l3mfagyt
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:468} INFO - [base] RUNNING
{pod_manager.py:486} INFO - [base] RUNNING
{pod_manager.py:619} INFO - Pod airflow-toto-pod-l3mfagyt has phase Running
{pod.py:931} INFO - Deleting pod: airflow-toto-pod-l3mfagyt
{taskinstance.py:441} ▼ Post task execution logs
{taskinstance.py:1206} INFO - Marking task as SUCCESS. dag_id=kpo_async_get_logs, task_id=task-two, run_id=manual__2024-05-17T12:00:38.699092+00:00, execution_date=20240517T120038, start_date=20240517T120039, end_date=20240517T120103
{local_task_job_runner.py:240} INFO - Task exited with return code 0
{taskinstance.py:3498} INFO - 0 downstream tasks scheduled from follow-on schedule check

if I check the logs in the pod directly it give

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Taragolis]

What happen if you change your script to?

import time

if __name__ == '__main__':
    for i in range(10):
        time.sleep(2)
        print(f"RUNNING {i}")

[raphaelauv]

@Taragolis

in that case there is no missing logs

{pod_manager.py:374} WARNING - Pod not yet started: airflow-toto-pod-s0p6ivyu
{pod_manager.py:468} INFO - [base] RUNNING_0
{pod_manager.py:468} INFO - [base] RUNNING_1
{pod_manager.py:468} INFO - [base] RUNNING_2
{pod_manager.py:468} INFO - [base] RUNNING_3
{pod_manager.py:468} INFO - [base] RUNNING_4
{pod_manager.py:468} INFO - [base] RUNNING_5
{pod_manager.py:468} INFO - [base] RUNNING_6
{pod_manager.py:468} INFO - [base] RUNNING_7
{pod_manager.py:468} INFO - [base] RUNNING_8
{pod_manager.py:486} INFO - [base] RUNNING_9
{pod_manager.py:619} INFO - Pod airflow-toto-pod-s0p6ivyu has phase Running
{pod.py:931} INFO - Deleting pod: airflow-toto-pod-s0p6ivyu

[Taragolis]

I guess it similar to #39686

[Lee2532]

What happen if you change your script to? print to logging

import time
import logging

if name == 'main':
for i in range(10):
time.sleep(2)
logging.info("RUNNING")

[github-actions]

This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[raphaelauv]

No stale