Setting up an S3 Connection with Vault Secrets Backend

[damonearp]

I'm trying to setup task logging to our internal S3 Storage Cluster. While I can get it to work by storing the connection in the database, storing the connection information in Vault doesn't seem to be feasible.

Version

• Apache Airflow 2.0.1
• Vault KV API 2

(Works) Database Approach

Not much to say here, tasks write logs to our local cluster and our webui can retrieve them.

> select * from connections;
id | conn_id | conn_type | host | schema | login | password | port |                                                              extra                       | is_encrypted | is_extra_encrypted | description 
---+---------+-----------+------+--------+-------+----------+------+------------------------------------------------------------------------------------------+--------------+--------------------+-------------
 1 | s3conn  | s3        |      |        |       |          |      | {"host":"http://hostname/", "aws_access_key_id":"id", "aws_secret_access_key": "secret"} | f            | f                  | 

(Fails) Vault Attempt 1

Since Vault allows several key value pairs under an entry I thought the following would simply work.

> vault kv get airflow/connections/s3conn
====== Metadata ======
...
====== Data ======
Key          Value
---          -----
conn_type    s3
extra        {"host":"http://hostname/", "aws_access_key_id":"id", "aws_secret_access_key": "secret"}

Using the above, the WebUI fails to retrieve a log with the following error.

*** Failed to verify remote log exists s3://airflow/dag01/create/2021-07-06T20:15:47.379962+00:00/1.log.
The conn_id `s3conn` isn't defined

I altered the secret by adding conn_id: s3conn explicitly, and as expected I got the same error.

(Fails) Vault Attempt 2

After looking at the examples on the official site and the API call airflow.providers.hashicorp.secrets.vault.VaultBackend, I assume the conn_uri is the only valid key/value pair that is read from the vault secret.

So naturally I tried:

> vault kv get airflow/connections/s3conn
====== Metadata ======
...
====== Data ======
Key            Value
---            -----
conn_uri       http://id:secret@hostname/

With this setup the Airflow WebUI doesn't complain about no such conn_id, however now the aws/s3 provider is misconfigured and trying to talk to Amazon and not our local S3 cluster.

Questions

• Is what I'm trying to do possible?
• Is this a limitation of the VaultBackend or Secret Backends in general?

Thanks for any help, we'd love to store our credentials in Vault over the DB (even with encryption setup).

[kbumsik]

Airflow has its own URI format described here: https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#connection-uri-format

So, in your case:

import json
from airflow.models.connection import Connection

c = Connection(
    conn_id='s3conn',
    conn_type='s3',
    extra=json.dumps({"host":"http://hostname/", "aws_access_key_id":"id", "aws_secret_access_key": "secret"}),
)

c.get_uri()

>>> 's3://?host=http%3A%2F%2Fhostname%2F&aws_access_key_id=id&aws_secret_access_key=secret'

So, conn_uri should be s3://?host=http%3A%2F%2Fhostname%2F&aws_access_key_id=id&aws_secret_access_key=secret

Try making a URI using the example :)

[damonearp]

That did it! Thank you for taking the time to point me in the right direction and providing a method to generate these uri's. I'm green in all of this and hadn't come across this custom URI format yet, nor knew to search for it.

[kbumsik]

No worries. Airflow doc is great but it sometimes takes time to figure out the right page :)