Stackdriver Remote Logging - Only partial log content captured on Google Cloud Logging #21837
Replies: 3 comments 2 replies
-
Thanks for opening your first issue here! Be sure to follow the issue template! |
Beta Was this translation helpful? Give feedback.
-
I think you are mixing logs vs. remote logs here. When stackdriver is enabled, your task logs should be sent to stackdriver and what you are showing as "Here's the StackDriver output (unfiltered) from executing a sample DAG" is the file log of task instance execution in the log file, not stackdriver. You likely will find the missing logs in ... stackdriver. Or maybe your explanation was not clear enough. Just one comment: You need ot make sure that you configure your docker-compose properly (and we do not guarantee the docker compose we have is good for this kind of modifications. Our docker compose is "quick-start" only and you have to make sure you know what you are doing there if you want to productionize it and warn against it: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html
It is likely that you have misconfigured your docker compose. It's very easy to overlook something and make mistakes when you configure the docker compose, so that's why we do not recommend to use it and it will be hard for you to get any help here even if you show your full docker compose (As it is likely to be too complex to analyse and look for bugs and typos). I stronly recommend to switch to K8S and Helm chart as there are ready-to-use recipes on how to reconfigure it and much easier to get help. https://airflow.apache.org/docs/helm-chart/stable/index.html I converted it into discussion, as this is likely misconfiguration rather than Airlfow problem and needs further clarifications. |
Beta Was this translation helpful? Give feedback.
-
Everything from the child process after the fork does not log. |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
2.2.3 (latest released)
What happened
I am trying to configure remote logging with StackDriver on my local Airflow setup (Official Docker Compose). I have updated following config variables in docker-compose file:
The GCP connection has been setup properly on Airflow. The service account has the correct set of permissions to read/write logs.
I am noticing that remote logger is only capturing partial logs. Here's the StackDriver output (unfiltered) from executing a sample DAG:
These logs above are for a simple hello world bash task, code snippet below:
What you expected to happen
I would expect the StackDriver remote logger on Airflow to capture full logs for the DAG.
When I run the same DAG (from above in What happened section) with remote logging turned off, here's the logs I see as expected:
How to reproduce
<name_of_the_dag> resource.type="global"
Operating System
macOS Monterey Version 12.2
Versions of Apache Airflow Providers
apache-airflow-providers-google==6.2.0
apache-airflow-providers-grpc==2.0.1
Deployment
Docker-Compose
Deployment details
I am using official docker compose file from https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml and just adding necessary environment variables
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions