KubernetesPodOperator can't mount sercret as volume

[scauglog]

Apache Airflow version: 1.10.12 and 2.0.0b3

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.19.2

Environment:

• Cloud provider or hardware configuration: minikube
• OS (e.g. from /etc/os-release): Ubuntu 20.04
• Kernel (e.g. uname -a): 5.4.0-53

What happened:
With the kubernetePodOperator and KubernetesExecutor when I try to mount secret as volume the pod (the one in kubernetesPodOperator) didn't launch and the task return as failed. The error occur with either V1Volume object or airflow Secret object.
Persistent volume claim work perfectly.

What you expected to happen:

airflow launch a worker pod on kubernetes and the worker pod launch a pod on kubernetes

How to reproduce it:

import airflow
from airflow import DAG
from kubernetes.client import models as k8s
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.kubernetes.secret import Secret

default_args = {
    'owner': 'debug',
    'depends_on_past': False,
    'start_date': airflow.utils.dates.days_ago(1),
}

pvc_volume = k8s.V1Volume(
    name='ml-data',
    persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='ml-data')

)

secret_volume = k8s.V1Volume(
    name='deploy-key',
    secret=k8s.V1SecretVolumeSource(default_mode=600, secret_name="dvc-deploy-key")
)

pvc_volume_mount = k8s.V1VolumeMount(
    name='ml-data', mount_path='/data/', sub_path=None, read_only=False
)

secret_volume_mount = k8s.V1VolumeMount(
    name='deploy-key', mount_path='/root/.ssh', sub_path=None, read_only=True
)

secret_file = Secret(deploy_type='volume',
                     deploy_target='/root/.ssh/',
                     secret='dvc-deploy-key')

dag = DAG(
    "testing",
    default_args=default_args,
    description='indexation train pipeline',
    schedule_interval=None
)

start = DummyOperator(
    task_id="start",
    dag=dag
)

t1 = KubernetesPodOperator(
    task_id=f't1',
    name=f't1',
    namespace='airflow',
    image="busybox",
    cmds=["echo", "main"],
    volume_mounts=[pvc_volume_mount],
    volumes=[pvc_volume],
    is_delete_operator_pod=False,
    get_logs=True,
    dag=dag,
)

t10 = KubernetesPodOperator(
    task_id=f't10',
    name=f't10',
    namespace='airflow',
    image="busybox",
    cmds=["echo", "main"],
    secrets=secret_file,
    volume_mounts=[pvc_volume_mount],
    volumes=[pvc_volume],
    is_delete_operator_pod=False,
    get_logs=True,
    dag=dag,
)

t2 = KubernetesPodOperator(
    task_id=f't2',
    name=f't2',
    namespace='airflow',
    image="busybox",
    cmds=["echo", "main"],
    volume_mounts=[secret_volume_mount],
    volumes=[secret_volume],
    is_delete_operator_pod=False,
    get_logs=True,
    dag=dag,
)

t3 = KubernetesPodOperator(
    task_id=f't3',
    name=f't3',
    namespace='airflow',
    image="busybox",
    cmds=["echo", "main"],
    volume_mounts=[pvc_volume_mount, secret_volume_mount],
    volumes=[pvc_volume, secret_volume],
    is_delete_operator_pod=False,
    get_logs=True,
    dag=dag,
)
start >> [t1, t10, t2, t3]

Anything else we need to know:
I'm not sure if it's an airflow issue or a kubernetes-client issue

How often does this problem occur? Once? Every time etc?
This problem occur every time
Any relevant logs to include? Put them here in side a detail tag:

worker.log

airflow@testingpulldata:/opt/airflow$ airflow tasks run testing pull_data "2020-12-02T15:12:29.757041+00:00" --local --pool default_pool --subdir /opt/airflow/dags/test.py

[2020-12-02 15:30:19,007] {dagbag.py:440} INFO - Filling up the DagBag from /opt/airflow/dags/test.py
/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use kub ernetes.client.models.V1Volume.
/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use kub ernetes.client.models.V1VolumeMount.
Running <TaskInstance: testing.pull_data 2020-12-02T15:12:29.757041+00:00 [success]> on host testingpulldata

[VBhojawala]

Hello @scauglog ,

I Just wrote KubernetesPodOperator Guide Here.https://github.com/VBhojawala/airflow/blob/k8s-docs/docs/apache-airflow-providers-cncf-kubernetes/operators.rst#mounting-secrets-as-volume
Example Dag in Guide runs fine and mounts Secret volume.

The worker.log given mentions Only DeprecationWarning which does not stop Pod from being launch IMHO.

[sstevens303]

@VBhojawala I read your document and I was wondering how do map a volume. I want to do the equivalent of
docker run -v /host/directory:/container/directory using airflow and kubernetespodoperator. So I can take an existing directory path available to airflow and map it to a directory path inside the container. I cannot find any examples of this.

[VBhojawala]

There is an example of mounting host volume to pod :

Here is the link https://github.com/VBhojawala/airflow/blob/k8s-docs/docs/apache-airflow-providers-cncf-kubernetes/operators.rst#mounting-persistent--volume-to-pod

In this example local machine's /tmp/myapp is mounted to pod

[jedcunningham]

@sstevens303 are you still having issues mounting secrets or can this be closed?

[sstevens303]

This can be closed

…

________________________________ From: Jed Cunningham ***@***.***> Sent: Wednesday, March 10, 2021 5:18 PM To: apache/airflow ***@***.***> Cc: Sharon Stevens ***@***.***>; Mention ***@***.***> Subject: [External Origin] Re: [apache/airflow] KubernetesPodOperator can't mount sercret as volume (#12760) @sstevens303<https://github.com/sstevens303> are you still having issues mounting secrets or can this be closed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#12760 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARY5W5E27S3XZD7R4RTFTNDTC7O27ANCNFSM4UKYLRIA>.