Logs Are Not Written When Using Remote Logging with K8s Executor

[yehoshuadimarsky]

Official Helm Chart version

1.3.0 (latest released)

Apache Airflow version

2.2.1

Kubernetes Version

1.19.14-gke.1900

Helm Chart configuration

executor: "KubernetesExecutor"
postgresql:
  enabled: false
pgbouncer:
  enabled: true
flower:
  enabled: false
config:
  core:
    load_examples: 'False'
    load_default_connections: 'False'
  webserver:
    expose_config: 'False'
  logging:
    remote_logging: 'True'
    remote_log_conn_id: "gcs-conn-dev"
    remote_base_log_folder: "gs://[REDACTED]/airflow_logs"
cleanup:
  enabled: true
dags:
  gitSync:
    enabled: true
    repo: ssh://git@github.com/[REDACTED]/[REDACTED].git
    branch: airflow-dev
    rev: HEAD
    depth: 1
    subPath: "airflow/dags"
    sshKeySecret: airflow-git-ssh-secret
    knownHosts: |
      github.com ssh-rsa [REDACTED]==

Docker Image customisations

Dockerfile

FROM apache/airflow:2.2.1-python3.9

SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
USER root
RUN apt-get update \
  && apt-get upgrade \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*


USER airflow
COPY airflow/requirements.txt .
RUN pip install --upgrade --no-cache-dir -r requirements.txt && rm requirements.txt

requirements.txt

apache-airflow-providers-google==6.0
apache-airflow-providers-cncf-kubernetes==2.0
pandas==1.3
quandl==3.6.1

What happened

I have simple tasks running with simple logs, sent to GCS via remote logging. But they don't show up in GCS.

What you expected to happen

I expect logs to be visible but they are not. I cannot set up a PMV in my k8s cluster, so instead I chose to use remote logging to persist the logs in GCS. I have verified that the permissions are correct and even tested it out. But whenever any task runs, no logs appear in GCS. So of course, when I click "Log" in the task afterwards, I get this, because the worker pod is already deleted, and GCS never got the logs shipped to it.

*** Unable to read remote log from gs://[REDACTED]/airflow_logs/logging_test/list_gcp_bucket_objects_in_dev/2021-12-19T17:44:14.483757+00:00/1.log
*** 404 GET https://storage.googleapis.com/download/storage/v1/b/[REDACTED]/o/airflow_logs%2Flogging_test%2Flist_gcp_bucket_objects_in_dev%2F2021-12-19T17%3A44%3A14.483757%2B00%3A00%2F1.log?alt=media: No such object: [REDACTED]/airflow_logs/logging_test/list_gcp_bucket_objects_in_dev/2021-12-19T17:44:14.483757+00:00/1.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

*** Trying to get logs (last 100 lines) from worker pod loggingtestlistgcpbucketobjectsindev.6c24f1cc7ffe45a88c54afedeb ***

*** Unable to fetch logs from worker pod loggingtestlistgcpbucketobjectsindev.6c24f1cc7ffe45a88c54afedeb ***
(404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'de670d54-6f87-4ce5-90f8-a6c161d70fe2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sun, 19 Dec 2021 18:16:45 GMT', 'Content-Length': '294'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \\"loggingtestlistgcpbucketobjectsindev.6c24f1cc7ffe45a88c54afedeb\\" not found","reason":"NotFound","details":{"name":"loggingtestlistgcpbucketobjectsindev.6c24f1cc7ffe45a88c54afedeb","kind":"pods"},"code":404}\n'

How to reproduce

You will need a GCP account which I obviously cannot provide to reproduce.

Set up the Helm chart with the values.yaml override I provided.

Then run this Dag, I even added a long time.sleep(30) in case it needs some time to ship the logs before the pod is killed, but this didn't work.

import logging
import time

from airflow.models.dag import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.google.cloud.hooks.gcs import GCSHook
from airflow.utils import dates

BUCKET ="some-redacted-gcs-bucket"

def ping_gcp(gcp_conn_id, bucket):
    logging.debug("this is a test debug log")
    logging.info("starting logging task")
    logging.warning("this is a test warning log")
    hook = GCSHook(gcp_conn_id=gcp_conn_id)
    objs = hook.list(bucket)
    logging.info(f"Here are the objects in GCS bucket: {objs}")
    time.sleep(30)


default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "start_date": dates.days_ago(1),
}


dag = DAG("logging_test", default_args=default_args)

task = PythonOperator(
    dag=dag,
    task_id="list_gcp_bucket_objects_in_dev",
    python_callable=ping_gcp,
    op_kwargs=dict(gcp_conn_id="gcs-conn-dev", bucket=BUCKET),
)

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I believe the problem is with understanding the logging authorization:

See: https://airflow.apache.org/docs/apache-airflow-providers-google/stable/logging/gcs.html

By default Application Default Credentials are used to obtain credentials. You can also set google_key_path option in [logging] section, if you want to use your own service account

As I read that - the remote_log_conn_id is used only to read the logs. but in order to write the logs, the PODs executed in K8S executor need to have "Application Default Credentials" with "write" access to the storage (you can also provide google_key_path parameter but you'd have to make sure you have the key available in the POD and is generallly less secure). Since your PODs are likely running in GKE you should have ideallly workload identity configured, so that your PODs can have write access to your bucket: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.

Note that the "gcs-conn-dev" in this case only needs "read" access to the bucket.

I am closint it for now, assuming that your problem will be solved with this advice, please test it and comment here with your results and we can reopen it if needed.

[yehoshuadimarsky]

I see. The docs are confusing in that regard then, I thought the remote_log_conn_id is used for both reading and writing - for all operations for remote logging.

So maybe I can broaden my question then - why are there 2 different connection IDs, remote_log_conn_id and google_key_path? Maybe the GCP provider package should change the remote logging to use remote_log_conn_id for both read and write?

[potiuk]

I see. The docs are confusing in that regard then, I thought the remote_log_conn_id is used for both reading and writing - for all operations for remote logging.

I heartily invite you to make a PR that will make it clearer there. Airflow is created by > 1800 contributors - most of them contributing code and docs voluntarily and this is an easy way to become one of the contributors and "give back" for the software you get for free. This is super easy - you just use "Suggest a change on this page" button at the bottom-right of the doc page and you will get a possibility to submit changes to the dodcumentation using GitHub UI as a new PR. It's even easier to make a PR than to create an issue about it in fact. And you as the user are one of the best to know what and how to describe it in the way that other users will find useful.

So maybe I can broaden my question then - why are there 2 different connection IDs, remote_log_conn_id and google_key_path? Maybe the GCP provider package should change the remote logging to use remote_log_conn_id for both read and write?

I belive (I have not implemented it) this is because of security. The conn_id is there and (currently) available to anyone who has access to the DB (including the webserver to read the logs). However writing should be exclusively available to the tasks "workload". Write access to the bucket is generally more "precious" and for example workload identity and using metadata server in GCP gives much better security, because the access to write is only granted to that particular machine executing the workload and only for the time of executing the task. Even if such "authtentication" leaks, it is useless outside of this single task execution. On the other hand the "conn_id" details are static and leaking it could give anyone unlimited and uncontrolled access to write any data to the bucket. Which could lead to a number of security issues (such as log injection vulnerabilities etc).

[yehoshuadimarsky]

Since your PODs are likely running in GKE you should have ideallly workload identity configured, so that your PODs can have write access to your bucket: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.

I followed all the steps in the Google docs for Workload Identity but am still having issues. I see that there are many K8s service accounts created by default by the Airflow Helm chart, but for Workload Identity you need to specify a single K8s service account to connect to a GCP service account. Which K8s service account does Airflow use? Are you able to shed more light on the role of all of the service accounts used/created by the Helm chart?

[yehoshuadimarsky]

I see. The docs are confusing in that regard then, I thought the remote_log_conn_id is used for both reading and writing - for all operations for remote logging.

I heartily invite you to make a PR that will make it clearer there. Airflow is created by > 1800 contributors - most of them contributing code and docs voluntarily and this is an easy way to become one of the contributors and "give back" for the software you get for free. This is super easy - you just use "Suggest a change on this page" button at the bottom-right of the doc page and you will get a possibility to submit changes to the dodcumentation using GitHub UI as a new PR. It's even easier to make a PR than to create an issue about it in fact. And you as the user are one of the best to know what and how to describe it in the way that other users will find useful.

Thank you! I will definitely try to submit a PR to change the docs, thank you for making it easy and inviting.

So maybe I can broaden my question then - why are there 2 different connection IDs, remote_log_conn_id and google_key_path? Maybe the GCP provider package should change the remote logging to use remote_log_conn_id for both read and write?

I belive (I have not implemented it) this is because of security. The conn_id is there and (currently) available to anyone who has access to the DB (including the webserver to read the logs). However writing should be exclusively available to the tasks "workload". Write access to the bucket is generally more "precious" and for example workload identity and using metadata server in GCP gives much better security, because the access to write is only granted to that particular machine executing the workload and only for the time of executing the task. Even if such "authtentication" leaks, it is useless outside of this single task execution. On the other hand the "conn_id" details are static and leaking it could give anyone unlimited and uncontrolled access to write any data to the bucket. Which could lead to a number of security issues (such as log injection vulnerabilities etc).

This makes sense, thank you.

[potiuk]

I followed all the steps in the Google docs for Workload Identity but am still having issues. I see that there are many K8s service accounts created by default by the Airflow Helm chart, but for Workload Identity you need to specify a single K8s service account to connect to a GCP service account. Which K8s service account does Airflow use? Are you able to shed more light on the role of all of the service accounts used/created by the Helm chart?

It's not what "Helm Chart" does by default. If you are using K8S executor, then you can (and even should) define pod template https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html#configuration and it will create you the PODs in the way and configuration that you specify. What you need to do is to create your POD template with all the necessary configuration for K8S.

[yehoshuadimarsky]

Got it. Thank you, that worked! I had a whole comment written up asking why the pod template wasn't using the Workload Identity service account, but it turned out that I was just a little impatient and it ended up working correctly after some time. Thank you for all your help.