DAG-level permissions are periodically being deleted

[alexakra]

Apache Airflow version

2.6.2

What happened

After upgrading to version 2.6.2 from 2.4.3, all the DAG-level permissions (ab_permission_view_role) are periodically being deleted.

What you think should happen instead

The DAG-level permissions should not be deleted once defined as it was in pre-3.6 versions.

How to reproduce

Define a role with DAG-level permission. The permission will be removed from the role after short period of time.

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Helm Chart 1.9.0

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

I think this is something on your side - some script that is run periodically. The infoirmation you provided is not nearly detailed enough to act on it. But you if you look at your logs and deployment and try to find the places/logs where you will see some deletions happening, maybe someone will be able to help to solve your issue.

Thanks in advance for more details, for now I converted it into discussion until more details are provided - enough for someone to be able to help you.

[alexakra]

When I disable airflow-dag-processor, the permissions are not being deleted. So this is somewhere in airflow-dag-processor process.

[uranusjr]

By deleted is the permission just gone? Generally when a permission is changed it emits a log message. Try to find that would help identify the cause and help figure out whether it is working as designed or not. There’s too little information to work from, to be honest.

[alexakra]

How do you create entries in the ab_permission_view_role?

• I edit the roles in UI. I have just found that they are kept and deleted from ab_permission_view_role.

Setting it to False might be short-term solution to your problem.

• Setting update_fab_perms to false, didn't solve the problem.

Generally when a permission is changed it emits a log message

• in which service and which message should I look for?

[sbrinkhorst]

We are facing the same issue with a similar setup (airflow 2.6.2 with official helm chart 1.9.0).

I have a script that creates the roles and the log messages say the permissions are added. On my local environment the DAG level permissions stay there, but if I apply the script on our acceptance environment I do not see all DAGs and after a while all DAG:[dag name] permissions are gone. (Can't tell you exactly how long until they are gone, last time I checked at the end of the working day some were still there and the next morning they were gone.)

Setting update_fab_perms does not help here either. I don't see any smoking guns in the scheduler logs.

[mdering]

We're facing the same issue as well. airflow 2.6.2. In our experiments we also found that this only impacts certain dags, and these permissions cannot be assigned to any role, even a new role. It seems like it lasts just a few minutes, maybe less.

[mdering]

after setting the scheduler logs to DEBUG, it seems likely that this happens whenever the dag in question is re-parsed. The only other lead I have is that the dag is relatively large and complex, so it can take over 1 second to re-parse. I wonder if this is bumping it from other relevant places, and the resources are being removed from the model.

[vincbeck]

I've always been quite away from the "sync-perm" part and it's been always a bit of mistery for me what it does and when it is needed (and it's almost not described in our documentation) - but maybe if you explain it here, we can (I am happy to do that) explain it better in our docs. @vincbeck -> I think that also might be a useful thing to clarify for the upcoming AIP-56 because once we separate FAB to a separate "provider", the sync-perm command will have to be part of that separation.

I created an issue as part of the AIP-56 project to track this effort: #32348

[isbushcar]

Hello!
We face the same issue.
We use git-sync, our repo contains about 40 DAGs and in our case it can be reproduced even when we use clean setup to new database:

1. Create new custom role (I didn't try to use predefined)
2. Grant permission on DAG to new role (e.g. can read on DAG:my_example_dag)
3. Permission is visible in UI and works, everything is cool
4. Update my_example_dag - its' graph or .sql files that DAG uses
5. After syncing with git permission is gone :) It's just get deleted from database.

As a workaround you can revoke delete on table ab_permission_view_role from airflow db user, but you won't be able to delete any role permissions via UI.
Hope it would be useful

[punx120]

do we know which change broke this?

[isbushcar]

That should be the issue
#32839

[punx120]

my bad - i didn't see the comment - PR is #30340

[relyabreon]

We are experiencing this issue as well. i don't understand the problem PR #30340 was intended to solve well enough to comment on the stalled fix PR #32999 (which apparently reintroduced the original sync issue).

Our (not fluent in Airflow code) understanding of the issue:

• Problem happens during file processing (forked subprocess) in the scheduler.
• At some point during processing, DagBag invokes SecurityManager.sync_perm_for_dag()
• access_control is not already populated
  • I don't know in what scenarios it would already be present
• if access_control is empty, then _revoke_all_stale_permissions local method is invoked
• That method abruptly revokes per-DAG access from all roles except for "Admin"

Presumably access_control is supposed to be populated at this point, but in our environment (and apparently for others as well) it isn't.

We're in a kubernetes environment. DAGs are defined in the container image, and an automated run process makes Airflow API calls to create roles, each with permissions on a logical subset of our DAGs. If there's another way to set up our roles that mitigates this issue, we'd be happy to update our roles setup, but it doesn't seem like we're doing anything particularly extreme.

[punx120]

access_control can be set on the DAG in the code, something like

with DAG('Test', schedule_interval='30 4 * * 1-5', access_control = {'Public' : {'can_read', 'can_edit', 'can_delete'}}) as dag:

This is actually what i'm doing now as a workaround to this issue.

[relyabreon]

Thanks. While that's a slightly less flexible approach than we'd like to use, it's something we could conceivably automate without too much pain, and have in our back pocket if we need to upgrade (we're currently holding at 2.6.1) before this issue is resolved.

[TsubasaBE]

Good solution. I can confirm this works great. Thanks.

[emre1e]

any news on this topic. It´s really hard to add this permission daily over and over.

[isbushcar]

You can try downgrading to 2.6.1 for now - it should be safe

[emre1e]

yeah but its not an solution :) Is it fixed in 2.7.0? Thank you

[bedag-bad]

Same question here is this going to be fixed in 2.7.0?

[potiuk]

There is a fix scheduled for 2.7.1 -> #33632 - please watch the devlist and issues here. The best way you can help to see and test it is when we release the RC1 for 2.7.1 try it out and confirm that it fixes the problem.

Can we count on your help with testing it @bedag-bad @emre1e ? Otherwise, if the fix does not work you will have to re-create the issue if it will be still persisting for you. It should be released within a week or so.