Where can I find errors from the dag-processor?

[jeff-cook]

I'm looking for a way to identify failed to load DAGs by the dag-processor. I can't find a log anywhere for the failure.

I'm running the current helm chart version 1.18.0

[potiuk]

Dag processor containers have log files in "logs" dir - as any other component. Depending on your versio of Airflow, the import erros are also stored in the DB and available in Airflow UI - surfaced via notification in the dag list screen.

Not sure what other errors you might look for - but those are the two places where the logs / errors from dag processing are present

[jeff-cook]

I found #37122 and https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-processor-log-target

I set AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET : stdout in the container.

But doesn't seem to be sending those logs to stdout.

However in apache/airflow:3.0.2 it is not sending to stdout.
I'm running the current helm chart version 1.18.0

[jeff-cook]

I can also find the errors by running airflow dags list-import-errors, but that doesn't get me a way to provide feedback in CI or via an alert in our monitoring system.

https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#list-import-errors

[jeff-cook]

If you are using statsd you can use the airflow_dag_processing_import_errors prometheus metric.