KubernetesPodOperator prints empty log lines without respecting the log format #36571
Comments
karakanb
added
area:providers
kind:bug
|
This log is very strange; for example, for the pod starting wait, the check is running in a loop, and there are no other log statements: airflow/airflow/providers/cncf/kubernetes/utils/pod_manager.py Lines 355 to 366 in 2c15dc9 These log messages come from something else, probably from the |
|
Agree log is very strange, @karakanb I am not able to replicate this with 2.8.0 & cncf-kubernetes==7.12.0, kindly do update us if you do not see this issue after disabling
|
|
I can confirm that the issue is still there even after I disabled the context logger: [logging]
colored_console_log = False
enable_task_context_logger = False
log_format = %%(asctime)s | %%(levelname)s | %%(message)s
remote_logging = False |
|
@karakanb, since you've upgraded the CNCF Kubernetes provider from version |
|
Exactly, the issue wasn't happening in 7.4.2, it has started happening in 7.12.0, although I have updated pretty much all the other providers as well as the airflow version at the same time. Let me test it by downgrading the cncf provider. |
|
The issue disappeared when I went back to 7.4.2, which means it has nothing to do with Airflow 2.8.0 or my environment as far as I can tell. I have upgraded to v7.13.0 and the issue persists there as well. |
|
cc: @dstandish - likely #35416 was not THAT inconsequential |
|
@potiuk the empty log messages are related to this loop: airflow/airflow/providers/cncf/kubernetes/utils/pod_manager.py Lines 355 to 366 in 2c15dc9 and most probably logged by a 3rd party lib ( |
|
Yeah. It's just strange changing provider version fixes it - but yeah it could be caused also by some changes to provider dependencies between those two versions (but it does not look like it could be the reason): git diff providers-cncf-kubernetes/7.4.2..providers-cncf-kubernetes/7.12.0 -- airflow/providers/cncf/kubernetes/provider.yamldiff --git a/airflow/providers/cncf/kubernetes/provider.yaml b/airflow/providers/cncf/kubernetes/provider.yaml
index 11a661e63e..f0b35ca392 100644
--- a/airflow/providers/cncf/kubernetes/provider.yaml
+++ b/airflow/providers/cncf/kubernetes/provider.yaml
@@ -22,7 +22,17 @@ description: |
`Kubernetes <https://kubernetes.io/>`__
suspended: false
+source-date-epoch: 1703288121
versions:
+ - 7.12.0
+ - 7.11.0
+ - 7.10.0
+ - 7.9.0
+ - 7.8.0
+ - 7.7.0
+ - 7.6.0
+ - 7.5.1
+ - 7.5.0
- 7.4.2
- 7.4.1
- 7.4.0
@@ -65,7 +75,8 @@ versions:
- 1.0.0
dependencies:
- - apache-airflow>=2.4.0
+ - aiofiles>=23.2.0
+ - apache-airflow>=2.6.0
- asgiref>=3.5.2
- cryptography>=2.0.0
# The Kubernetes API is known to introduce problems when upgraded to a MAJOR version. Airflow Core
@@ -100,7 +111,6 @@ integrations:
operators:
- integration-name: Kubernetes
python-modules:
- - airflow.providers.cncf.kubernetes.operators.kubernetes_pod
- airflow.providers.cncf.kubernetes.operators.pod
- airflow.providers.cncf.kubernetes.operators.spark_kubernetes
- airflow.providers.cncf.kubernetes.operators.resource
@@ -337,3 +347,6 @@ config:
type: string
example: ~
default: ""
+
+executors:
+ - airflow.providers.cncf.kubernetes.kubernetes_executor.KubernetesExecutor |
|
Here's the dockerfile I use for this deployment: FROM apache/airflow:2.8.0-python3.11
ENV PYTHONPATH "${PYTHONPATH}:${AIRFLOW_HOME}"
ENV SQLALCHEMY_SILENCE_UBER_WARNING=1
USER root
RUN apt-get update && apt-get install -y git gcc g++ unixodbc unixodbc-dev make nano curl
USER airflow
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt
COPY . .The The full set of dependencies, in case it helps: |
|
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. |
github-actions
bot
added
the
stale
hussein-awala
removed
the
stale
|
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. |
github-actions
bot
added
the
stale
|
Please keep this open. |
github-actions
bot
removed
the
stale
|
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. |
github-actions
bot
added
the
stale
|
ah come on now, please stay open. |
potiuk
removed
the
stale
|
I added |
|
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. |
github-actions
bot
added
the
stale
|
keep it open please |
potiuk
removed
the
stale
|
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. |
github-actions
bot
added
the
stale
|
This issue has been closed because it has not received response from the issue author. |
|
please reopen this |
potiuk
removed
the
stale
|
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. |
github-actions
bot
added
the
stale
|
There's not much I can report unfortunately. I am running v2.9.0 and the issue is still there. |
eladkal
removed
stale
Apache Airflow Provider(s)
cncf-kubernetes
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.14.0
apache-airflow-providers-celery==3.5.1
apache-airflow-providers-cncf-kubernetes==7.12.0
apache-airflow-providers-common-io==1.1.0
apache-airflow-providers-common-sql==1.10.0
apache-airflow-providers-discord==3.5.0
apache-airflow-providers-docker==3.8.2
apache-airflow-providers-elasticsearch==5.3.0
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.13.0
apache-airflow-providers-grpc==3.4.0
apache-airflow-providers-hashicorp==3.6.0
apache-airflow-providers-http==4.8.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-microsoft-azure==8.4.0
apache-airflow-providers-mysql==5.5.0
apache-airflow-providers-odbc==4.2.0
apache-airflow-providers-openlineage==1.3.0
apache-airflow-providers-postgres==5.9.0
apache-airflow-providers-redis==3.5.0
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.8.0
apache-airflow-providers-slack==8.5.1
apache-airflow-providers-snowflake==5.2.1
apache-airflow-providers-sqlite==3.6.0
apache-airflow-providers-ssh==3.9.0
Apache Airflow version
2.8.0
Operating System
Debian 12 (bookworm)
Deployment
Official Apache Airflow Helm Chart
Deployment details
deployed on digitalocean k8s. I have a custom log format as
'%%(asctime)s | %%(levelname)s | %%(message)s'What happened
I have recently upgraded the cncf-kubernetes provider from
7.4.2to7.12.0and started seeing weird log lines being printed out in-between application logs:this also happens while the application is running:
As you can see the log lines have a different format than the actual airflow log format, and they seem to contain no messages. This causes confusion, and for a few cases where I parse the logs it broke my existing code.
I have checked through external log collectors as well as via tailing the pod logs and I can confirm that the logs do not come from the application/pod side, and being logged somewhere on the airflow side.
What you think should happen instead
there should be no log lines printed that doesn't respect the logging format.
How to reproduce
Create a pod with KubernetesPodOperator, it happens to me consistently.
Anything else
Every time I run a pod it happens.
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: