Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speed up import time #5770

Merged
merged 1 commit into from
Oct 17, 2023
Merged

speed up import time #5770

merged 1 commit into from
Oct 17, 2023

Conversation

oliver-sanders
Copy link
Member

@oliver-sanders oliver-sanders commented Oct 16, 2023

Discovered a number of cases where modules were being imported just for type checking, slowing the code needlessly. Added a flake-8 check which will flag these for future reference. Also targeted some of the biggest offenders in the critical cylc-message import pathway bringing import time down to <0.4s on my system.

  • Don't import modules for type checking purposes.
  • Move some light weight functions out of a heavy weight module to reduce the import penalty.
  • Make a couple of heavy weight imports dynamic to avoid importing them when not necessary.
  • This reduces the import time of cylc.flow.scripts.message by ~66%.
  • This reduces the task runtime (script=true) by ~1s.

FYI, to investigate import time:

$ echo 'import cylc.flow.scripts.message' > testscript.py
$ python -X importtime testscript.py 2> import.log
$ pip install tuna
$ tuna import.log

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
  • Tests are included (or explain why tests are not needed).
  • CHANGES.md entry included if this is a change that can affect users
  • Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@oliver-sanders oliver-sanders added small efficiency For notable efficiency improvements labels Oct 16, 2023
@oliver-sanders oliver-sanders added this to the cylc-8.3.0 milestone Oct 16, 2023
@oliver-sanders oliver-sanders self-assigned this Oct 16, 2023
* Don't import modules for type checking purposes.
* Move some light weight functions out of a heavy weight module to
  reduce the import penalty.
* Make a couple of heavy weight imports dynamic to avoid importing them
  when not necessary.
* This reduces the import time of cylc.flow.scripts.message by ~66%.
* This reduces the task runtime (`script=true`) by ~1s.
@MetRonnie MetRonnie merged commit cd409a2 into cylc:master Oct 17, 2023
35 of 36 checks passed
@MetRonnie MetRonnie mentioned this pull request Oct 17, 2023
8 tasks
@oliver-sanders oliver-sanders deleted the import_time branch October 17, 2023 16:44
wxtim added a commit to wxtim/cylc that referenced this pull request Oct 18, 2023
* upstream/master:
  speed up import time (cylc#5770)
  graphql: remove extraLogs field (cylc#5672)
  give a more useful error message if remote init fails (cylc#5720)
  clarify SLURM job handler docs (cylc#5748)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
efficiency For notable efficiency improvements small
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants