2.0.1 GSCTaskHandler doesn't upload logs

[Jaxing]

Hey we are running Airflow2.0.1 on kubernetes with KubernetesExecutor.

These are our logging logs:

base_log_folder = /opt/airflow/logs
remote_logging = True
remote_log_conn_id =
google_key_path = /opt/airflow/key.json
remote_base_log_folder = gs://my-bucket/logs

The webserver can read the logs when the are uploaded to the correct place (tested by uploading dummy logs manually for a task).

The following code executed on a side car container running the same image and the same config as my base container:

from airflow.providers.google.cloud.log.gcs_task_handler import GCSTaskHandler
task_handler = GCSTaskHandler(
    base_log_folder="/opt/airflow/logs/",
    gcs_log_folder="gs://my-bucket/test",
    filename_template="{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log",
    gcp_key_path="/opt/airflow/key.json")
task_handler.gcs_write("test", ""gs://my-bucket/test/test.log")

gives the following output:

[2021-04-16 07:12:37,067] {gcs_task_handler.py:182} INFO - Previous log discarded: 404 GET https://storage.googleapis.com/download/storage/v1/b/my-bucket/o/test%2Ftest.log?alt=media: No such object: mybucket/test/test.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

However I can see the file test.log in my bucket.

My understanding is that remote_log_conn_id is not needed if I have google_key_path.

[annlund]

I'm facing the same issue. It's possible to use the GCSTaskHandler manually (executed as above) to write logs to the bucket, but the actual logs from the tasks are not written to the bucket when the scheduler runs.