Grouping of connections

[tangerine2003]

Description

Have an additional field called "Connection Group" within the connections form to place a set of connections under a specific grouping.

Use case/motivation

Our company uses airflow amongst a few different teams and the databases we interact with are decentralized. For a single DAG, we can have 25-50 connections. Often times finding all the connections that pertain the particular DAG can be difficult. We would like to assign connections to a group. This way we could easily search and filter by this group.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[eladkal]

By group do you mean owner? Like there is an owner field for DAGs?

[tangerine2003]

We were thinking about grouping by some sort of identifier. That identifier could be by project or by owner. If it was only by owner, that would suffice

[eladkal]

We were thinking about grouping by some sort of identifier. That identifier could be by project or by owner. If it was only by owner, that would suffice

Why setting prefix to the conn_id isnt enough?

[tangerine2003]

Setting a prefix wouldn't allow us to assign to a particular user. If we had it assigned to an owner then we could filter the view depending on the owner. This would provide a level of protection when there are a bunch of users within the application around who has access to specific connections.

[mik-laj]

Instead of groups / categories (1->m relation), we can think of introducing tags (n->m relation) to be consistent with DAGs that have tags.

[tangerine2003]

Instead of groups / categories (1->m relation), we can think of introducing tags (n->m relation) to be consistent with DAGs that lhave tags.

If we used tags in the connections, would there be any security concerns for users not needing to access and update certain connections? That would be my concern as we have about 10 teams using a single instance of airflow. We could mitigate it through training with the tags I suppose

[uranusjr]

If we used tags in the connections, would there be any security concerns for users not needing to access and update certain connections?

I'm not sure I follow. How would this be different if we use groups instead? Groups (or categories) and tags are essentially the same, except you can apply multiple tags to a Connection instead of just one.

[MM-Lehmann]

having any sort of access control on connections would be really helpful in diverse project/user environments. It could partly
solve the grouping request as well.
Is the access_control -> user roles model which is used in DAG configuration anyhow transferable?

[tangerine2003]

having any sort of access control on connections would be really helpful in diverse project/user environments. It could partly
solve the grouping request as well.
Is the access_control -> user roles model which is used in DAG configuration anyhow transferable?

I think this is ultimately what we are trying to get to. Having access controls around the connections and filtering the connection view based on which connections the users have access to. I was hoping to accomplish this with a grouping or assigning the connection to a user or owner.

[MM-Lehmann]

You still want your superusers and admins be able to see all of them so it's more than just filtering

…

Am 26.10.2021 in 13:23, tangerine2003 ***@***.***> schrieb: > > > having any sort of access control on connections would be really helpful in diverse project/user environments. It could partly > solve the grouping request as well. > Is the access_control -> user roles model which is used in DAG configuration anyhow transferable? > > I think this is ultimately what we are trying to get to. Having access controls around the connections and filtering the connection view based on which connections the users have access to. I was hoping to accomplish this with a grouping or assigning the connection to a user or owner. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[potiuk]

I think it makes very little sense to focus on connections, where we even have no "group" access to DAGs. Airflow'w mutli-tenancy features are virtualy non-existing and connection is part of multi-tennancy implementation. Just isolating access to connections where we have no "dag" isolation is really band-aid which might indeed improve the security "somewhat" but I think implementing it as isolated feature will create more mess than it solves, especially that we are moving away from the old UI to the the new one in foreseable future. I tihnk this should be part of the bigger "mutlitenancy" AIP and it should be targeted for Airflow 3.0.

[potiuk]

BTW. I am very likely going to start focusing big part of my time on moving "multi-tenancy" forward in the coming months, and I belive this should be part of the overall AIP.

[MM-Lehmann]

I think it makes very little sense to focus on connections, where we even have no "group" access to DAGs. Airflow'w mutli-tenancy features are virtualy non-existing and connection is part of multi-tennancy implementation. Just isolating access to connections where we have no "dag" isolation is really band-aid which might indeed improve the security "somewhat" but I think implementing it as isolated feature will create more mess than it solves, especially that we are moving away from the old UI to the the new one in foreseable future. I tihnk this should be part of the bigger "mutlitenancy" AIP and it should be targeted for Airflow 3.0.

Not sure what you are referring to, because I am using both access_control (roles) and tags to achieve exactly that. There is even a flask-appbuilder option to sync roles to your auth-backend.
This issues seems to seek the same: #19226

[potiuk]

How are you using acess_control roles to achieve that? Could you elaborate a bit?

[MM-Lehmann]

Well, I thought it was obvious but trying to provide you with a doc link, I must admit it's actually not that well documented. But there is third-party info on it. Just like here, when you instantiate your dag:

dag = DAG(
  access_control={
    'Role_1': {'can_dag_edit', 'can_dag_read'},
  },
  ...
  )

Make sure that the role you're referencing does exist. Upon Dag parsing, the role is then extended automagically with the new dag-level permissions.

[potiuk]

We have now Multi-tenancy effort in progress which I am leading. And while the first two AIPs that are very draft (but will soon be updated) do not address this final granularity yet, they pave the way for the third AIP that is going to address also this use case. This is in on the roadmap and planned and part of the bigger multi-tenancy effort. @MM-Lehmann @tangerine2003 - if you are interested in joining the effort, please join Airlfow Devlist and possibly #sig-multitenancy slack channel and take part in the discussions.

You can find last meeting mintues and even recording of the meeting where we discussed the plans for Multitenancy. https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-1%3A+Improve+Airflow+Security

Note that this is for a long haul - full implementation of the multitenancy (and even discussion on the AIPs) will take quite some time (several months/half a year at the least as this is a really big set of features to make it possible)

[anthonynassar]

Any updates related to this issue?