Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up airlfow roles list command #28244

Merged
merged 1 commit into from
Dec 9, 2022

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Dec 8, 2022

The command initialized whole flask_app of ours, but what we really needed was FAB security manager only.

Fixes: #28242


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk
Copy link
Member Author

potiuk commented Dec 8, 2022

Maybe not the "FINAL" fix but on my slow local DB:

 time airflow roles list

Before:

name
======
Admin
Op
Public
User
Viewer

airflow roles list  5.49s user 1.79s system 136% cpu 5.339 total

After:

name
======
Admin
Op
Public
User
Viewer

airflow roles list  2.43s user 1.75s system 198% cpu 2.103 total

@malthe - can you check it on your system please ?

@potiuk potiuk requested a review from malthe December 8, 2022 22:54
@potiuk potiuk changed the title Speed up airlfow role list command Speed up airlfow roles list command Dec 8, 2022
@malthe
Copy link
Contributor

malthe commented Dec 9, 2022

This cut our runtime from 49s to 9s.

Copy link
Contributor

@malthe malthe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this improvement be applied to roles_create as well?

@potiuk
Copy link
Member Author

potiuk commented Dec 9, 2022

Could this improvement be applied to roles_create as well?

Very much so. I can do another PR

@potiuk
Copy link
Member Author

potiuk commented Dec 9, 2022

There are quite a few other commands like that in user/role management that we could improve this way.

Hey @ashb @kaxil @jhtimmins and others - do you think there is any drawback to this solution ?

It does not initialize quite a lot of Airlfow Webserver pieces and we get 3x - 5x speedup for those commands.

Potentially I can imagine some subtle changes that somoene might want to add in their plugins might stop working with the CLI commands if we limit initialization to only Flask App Builder. However, if you have your own Airflow Security Manager - it should continue to work.

But I'd argue the CLIs of our was never meant to be extended by plugins, so I see that as "feature" not "bug/breaking change".

@potiuk
Copy link
Member Author

potiuk commented Dec 9, 2022

BTW. I think we can speed it up even more by reaping apart some of the FAB code doing the intialization - but this is the "Pareto" rule in full swing - 80% of gain in 20% of the effort.

The command initialized whole flask_app of ours, but what
we really needed was FAB security manager only.

Fixes: apache#28242
@potiuk potiuk force-pushed the speed-up-airflow-roles-list branch from 56fbfe7 to 2a20acc Compare December 9, 2022 10:28
@potiuk
Copy link
Member Author

potiuk commented Dec 9, 2022

@malthe - i sped it up a bit more - I suspected I do not even need "import_all_models" - so now it should be even faster.

@ashb
Copy link
Member

ashb commented Dec 9, 2022

LGTM.

@potiuk potiuk merged commit e247336 into apache:main Dec 9, 2022
@potiuk potiuk deleted the speed-up-airflow-roles-list branch December 9, 2022 12:47
@potiuk
Copy link
Member Author

potiuk commented Dec 9, 2022

I will apply it elsewhere then :)

potiuk added a commit to potiuk/airflow that referenced this pull request Dec 10, 2022
Originally those command were initializing whole Airflow Flask
webserver, which was much more than what was needed - we only
need to initialize Flask Appblication Builder. In case you have
slow DB this limits significantly not only a number of imported
classses but also a number of DB connections.

In some cases the speed of those CLI commands will visibly
go down from 10s of seonds to individual seconds (3x - 5x times).

Follow up after apache#28242, apache#28244
potiuk added a commit that referenced this pull request Dec 10, 2022
Originally those command were initializing whole Airflow Flask
webserver, which was much more than what was needed - we only
need to initialize Flask Appblication Builder. In case you have
slow DB this limits significantly not only a number of imported
classses but also a number of DB connections.

In some cases the speed of those CLI commands will visibly
go down from 10s of seonds to individual seconds (3x - 5x times).

Follow up after #28242, #28244
@pierrejeambrun pierrejeambrun added this to the Airflow 2.6.0 milestone Jan 9, 2023
@pierrejeambrun pierrejeambrun added the type:improvement Changelog: Improvements label Jan 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:CLI area:webserver Webserver related Issues type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Airflow CLI to list roles is slow
4 participants