Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds conda support to envd plugin #2020

Merged
merged 1 commit into from
Jan 13, 2024
Merged

Conversation

thomasjpfan
Copy link
Member

@thomasjpfan thomasjpfan commented Dec 4, 2023

Why are the changes needed?

This PR adds conda support to envd. This makes it simple to use Nvidia's runtime environment for PyTorch:

image_spec = ImageSpec(
    base_image="ubuntu:20.04",
    python_version="3.11",
    packages=["flytekit"],
    conda_packages=["pytorch", "pytorch-cuda=12.1"],
    conda_channels=["pytorch", "nvidia"]
)

which is inline with PyTorch's docs.

Furthermore, I do not think it is possible to install cuda=12.1 with envd. Using conda allows for cuda=12.1 and makes it simple to switch between Nvidia runtimes.

What changes were proposed in this pull request?

This PR adds conda_packages and conda_channels to ImageSpec to specific the conda packages and channels.

How was this patch tested?

I added a unit test to check the expected behavior.

Docs link

Copy link

codecov bot commented Dec 4, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (9bf7afa) 85.76% compared to head (456e6ac) 85.78%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2020      +/-   ##
==========================================
+ Coverage   85.76%   85.78%   +0.01%     
==========================================
  Files         313      313              
  Lines       23426    23447      +21     
  Branches     3510     3510              
==========================================
+ Hits        20092    20114      +22     
  Misses       2727     2727              
+ Partials      607      606       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@pingsutw pingsutw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, could you resolve the merge conflict?

Copy link
Member

@pingsutw pingsutw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thomasjpfan unit tests in envd-plugin are failing

pingsutw
pingsutw previously approved these changes Jan 4, 2024
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
@pingsutw pingsutw merged commit 892b474 into flyteorg:master Jan 13, 2024
84 checks passed
Future-Outlier pushed a commit that referenced this pull request Jan 18, 2024
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Future-Outlier <eric901201@gmai.com>
Future-Outlier pushed a commit that referenced this pull request Jan 18, 2024
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants