-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
start-up performance improvements #8221
Conversation
With flask-session, CKAN now has implicit msgspec dependency. Even if we ignore all the cool possibilities of this library and focus only on really fast parsing, we can improve this number. Here are my local results for In [10]: %timeit yaml.safe_load(open(filename, "rb").read())
154 ms ± 245 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [11]: %timeit msgspec.yaml.decode(open(filename, "rb").read())
10.4 ms ± 40.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) |
@smotornyuk great idea. Sounds like an easy fix to bring that 13% down to 1%. |
@amercader I didn't see an easy way to add decorators with click for load_environment and make_flask that are backwards compatible, so I'm leaving it for now. This PR is ready for review. xychart-beta horizontal
title "CLI performance improvements"
x-axis ["ckan user list", "NEW ckan user list", "ckan --help", "NEW ckan --help"]
y-axis "Speed (invocations/s)" 0 --> 0.8
bar [.313, .376, .317, .649]
|
failure looks unrelated, passes locally. |
@amercader my preference with That would allow quick handling for catching errors in parameters etc. and involve no hacky special cases for External extensions with cli interfaces can check for the existence of the WDYT? |
Co-authored-by: Adrià Mercader <amercadero@gmail.com>
requirements.txt
Outdated
@@ -1,5 +1,5 @@ | |||
# | |||
# This file is autogenerated by pip-compile with Python 3.9 | |||
# This file is autogenerated by pip-compile with Python 3.10 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is a pain but we need to generate the requirements.txt file in a 3.9 env otherwise we might lose requirements needed there. Or run pip-compile -P msgspec requirements.in
to just touch that package
Fixes #8219
Proposed fixes:
plugins_update
only when plugins changed (~25% reduction)make_app
for--help
(~50% reduction)ckan run
andckan translate js
(~2% reduction)Remaining time spent:
Ideas: