Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy loading of non-core dependencies #62

Merged
merged 9 commits into from
Nov 19, 2021
Merged

Conversation

ravi-mosaicml
Copy link
Contributor

@ravi-mosaicml ravi-mosaicml commented Nov 3, 2021

Resolved lazy-loading of non-core dependencies by using guarded imports.

Non-core dependencies should be imported either a) in a function, and/or b) if guarded with TYPE_CHECKING (we run pyright when installing everything).

Updated the tests to selectively ignore some ModuleNotFound errors and instead trigger a pytest.skip.

Tested manually by running tests on an installation of pip install composer[dev]. In the future, Jenkins will run this test.

@hanlint hanlint linked an issue Nov 9, 2021 that may be closed by this pull request
1. Conditioning imports of optional modules at the source
2. In tests, explicitely surpressing ModuleNotFoundErrors for optional imports
@ravi-mosaicml ravi-mosaicml changed the title WIP: Lazy Loading Lazy loading of non-core dependencies Nov 10, 2021
@ravi-mosaicml ravi-mosaicml requested review from a team and removed request for a team November 10, 2021 16:19
tests/test_load.py Show resolved Hide resolved
@ravi-mosaicml ravi-mosaicml requested a review from a team November 11, 2021 16:53
Copy link
Contributor

@hanlint hanlint left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@ajaysaini725 ajaysaini725 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@ravi-mosaicml ravi-mosaicml merged commit 53f854f into dev Nov 19, 2021
@ravi-mosaicml ravi-mosaicml deleted the ravi/lazy_loading branch November 19, 2021 02:36
coryMosaicML pushed a commit to coryMosaicML/composer that referenced this pull request Feb 23, 2022
Resolved lazy-loading of non-core dependencies by using guarded imports.

Non-core dependencies should be imported either a) in a function, and/or b) if guarded with TYPE_CHECKING (we run pyright when installing everything).

Updated the tests to selectively ignore some ModuleNotFound errors and instead trigger a pytest.skip.

Tested manually by running tests on an installation of pip install composer[dev]. In the future, Jenkins will run this test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Lazy loading of non-core dependencies Fix the load_model test for unet and GPT
3 participants