re: Airflow credential management

[ibrahimdurmus]

Hi Airflow members

I wanted to take your opinion on how do you manage connections/credentials. Today we have a section of UI, where admin can update connection details. In typical work setup, not everybody has admin permission, thus, user of Airflow are presumably asking to engineers who has admin access. For large deployment with tens (if not 100s) of connection, this might be boring/repetitive task.

Is there any part of the assumption inaccurate or do any of you have other means of managing/automating it.

I am asking, since I am evaluating the ROI of idea of revisiting the current implementation?

Thanks for your time

[potiuk]

For many users, Managing connections via Secrets Managers is the way to manage it using the capabilities/acces management of systems that are dedicated to do it.

As usual in Airflow - Airflow is a platform that can integrate with external tools that do the particular tasks better than the basic implementation in Airflow. This is the same with connections. You can integrate Airflow with a number of Secret's backend and they are better in managing secrets (connections are basically about secrets): https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/secrets-backends.html. Airflow simply delegates the task of managing the connections to other, dedicated systems this way (and it only uses the connection that the external system manages).

And you can even implement your own secrets backend to manage the connections through whatever storage for secrets.

[ibrahimdurmus]

thanks Jarek for insights