Kubernetes Auth Type not working as VaultBackend in Airflow

[eshaingle]

I am integration Airflow with Vault using the VaultBackend provided under airflow.contrib.secrets.hashicorp_vault.VaultBackend or airflow.providers.hashicorp.secrets.vault.VaultBackend. I installed apache-airflow-backport-providers-hashicorp backport package as it has support for various auth types.

Vault & Airflow are installed inside minikube using helm charts.
Vault version: 1.2.2
Airflow version: 1.10.12-Python-3.6
Minikube version: v1.14.2

Then through DAG file, using KubernetesPodOperator, I simply run a pod to execute below curl :

passing = KubernetesPodOperator(namespace='default', service_account_name="vault-auth", image="alpine:3.7", cmds=["sh", "-cx"], arguments=["apk add curl && curl --request POST \ --data '{"'"jwt"'": "'"$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"'", "'"role"'": "'"example"'"}' \ http://${VAULT_ADDR}/v1/auth/kubernetes/login "], labels={"test-airflow": "firstversion"}, name="passing-test", task_id="passing-task", get_logs=True, dag=dag )

I received the vault token as client_token.

I tried to connect vault through 2 methods. First using default token as auth type and passing above received token,

os.environ['AIRFLOW__SECRETS__BACKEND'] = "airflow.providers.hashicorp.secrets.vault.VaultBackend" os.environ['AIRFLOW__SECRETS__BACKEND_KWARGS'] = '{"connections_path": "myapp", "mount_point": "secret", "auth_type": "token", "token": "$TOKEN", "url": "http://vault:8200"}'

And second using kubernetes as auth type.

os.environ['AIRFLOW__SECRETS__BACKEND_KWARGS'] = '{"connections_path": "myapp", "auth_mount_point": "kubernetes", "mount_point": "secret", "auth_type": "kubernetes", "kubernetes_role": "example", "kubernetes_jwt_path":"/var/run/secrets/kubernetes.io/serviceaccount/token", "url": "http://vault:8200"}'

But for both it gives same error.

Error: File "/home/airflow/.local/lib/python3.6/site-packages/hvac/utils.py", line 47, in raise_for_error raise exceptions.InternalServerError(message, errors=errors, method=method, url=url) hvac.exceptions.InternalServerError: service account name not authorized, on post http://vault:8200/v1/auth/kubernetes/login

Please suggest a way to work out kubernetes auth login to access vault token in airflow.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[saikatharryc]

I was trying a different unconventional approach,

in docker, entry point doing source /vault/secret* [this file is generated by the vault agent injector sidecar]
which should set the ENV vars, which is working in scheduler and webserver, but not in the worker pods,
which are generated to run the dags.

somehow it's not executing the entry point stage from docker. while starting the pod.
otherwise, this would work I believe.

[eshaingle]

I was trying a different unconventional approach,

in docker, entry point doing source /vault/secret* [this file is generated by the vault agent injector sidecar]
which should set the ENV vars, which is working in scheduler and webserver, but not in the worker pods,
which are generated to run the dags.

somehow it's not executing the entry point stage from docker. while starting the pod.
otherwise, this would work I believe.

Hi Thank you for the reply.
I tried it but always get '/vault/secrets': No such file or directory

[saikatharryc]

I believe you are not running with the side car? which is the vault agent injector?

Hi Thank you for the reply.
I tried it but always get '/vault/secrets': No such file or directory

[eshaingle]

I believe you are not running with the side car? which is the vault agent injector?

Hi Thank you for the reply.
I tried it but always get '/vault/secrets': No such file or directory

I am injecting secrets via vault helm sidecar and vault-agent-injector pod performs the injection based on the annotations.

[saikatharryc]

@eshaingle
okay, now I know you are using k8s and vault injector & airflow in k8s.

---

I have tried this generic way, where I have installed vault agent injector, and by mentioning the annotations I get the secret file mounted in the pod.

later I source it, in the entry point (of the docker container in case the file is present)

Later I realized the workers which are spawned for the dags, they don't get the annotations and
I edited the config map to do so,
after I realized somehow it's not sourcing since the entry point is not getting triggered.

---

I found a way to make it work using operators

---

I have used this one https://github.com/ricoberger/vault-secrets-operator instead of vault agent injector and what this does is basically, it creates a secret and you can use secret with the helm chart and access the values from the ENV.

there are other ways too through the operator, check this out: https://github.com/banzaicloud/bank-vaults

[kaxil]

We can re-open this if other users have this issue too -- and looks like you were able to make it work with Vault Secrets Operator