Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caching pipeline_project outside of loop to avoid multple lookups #4887

Conversation

hugabora
Copy link
Contributor

@hugabora hugabora commented Apr 4, 2024

Description

Scheduler checks in a loop whether a block is an integration block. This instantiates Project object in each call which is quite slow and adds considerably, esp. when many blocks are in the pipeline.

The change caches the pipeline project outside of the loop. The method now takes the cached object as an optional parameter and avoids lookups.

How Has This Been Tested?

Tested in my local project.

Checklist

  • The PR is tagged with proper labels (bug, enhancement, feature, documentation)
  • I have performed a self-review of my own code
  • I have added unit tests that prove my fix is effective or that my feature works
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation

cc:
@wangxiaoyou1993

@wangxiaoyou1993 wangxiaoyou1993 added the enhancement Polish or UX improvements label Apr 4, 2024
@wangxiaoyou1993 wangxiaoyou1993 merged commit 802cd3a into mage-ai:master Apr 4, 2024
4 checks passed
oonyoontong pushed a commit to bunker-tech/mage-ai that referenced this pull request May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Polish or UX improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants