Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add check on conda-package metadata to guard against .conda packages #118

Merged
merged 4 commits into from Feb 10, 2020

Conversation

@savingoyal
Copy link
Member

savingoyal commented Feb 10, 2020

Looks like conda doesn't reliably/correctly update the env/conda-meta/<package>.json file which Metaflow relies on to identify which packages should be moved to S3 and their installation order on AWS Batch.

In this specific scenario, when the compute needs to execute on AWS Batch, Metaflow coaxes conda to generate an environment for linux-64 architectures using only .tar.bz2 dependencies, irrespective of where conda is executed from (usually a user's macOS or linux instance). In some very specific scenarios, conda correctly picks up the .tar.bz2 dependencies from upstream repos but fails to update the manifest, incorrectly pointing to .conda dependencies instead. This leads to incorrect bundling of packages and the user job fails when executing on AWS Batch since we end up packaging a .conda dependency as a .tar.bz2 dependency.

…a packages

Looks like conda doesn't reliably/correctly update the
`env/conda-meta/<package>.json` file which Metaflow relies on to identify
which packages should be moved to S3 and their installation order on AWS Batch.
In this specific scenario, when the compute needs to execute on AWS Batch,
Metaflow coaxes conda to generate an environment for `linux-64` architectures
using only `.tar.bz2` dependencies, irrespective of where conda is executed from
(usually a user's macOS or linux instance). In some very specific scenarios,
conda correctly picks up the `.tar.bz2` dependencies from upstream repos but fails
to update the manifest, incorrectly pointing to `.conda` dependencies instead. This
leads to incorrect bundling of packages and the user job fails when executing on
AWS Batch since we end up packaging a .conda dependency as a .tar.bz2 dependency.
@savingoyal

This comment has been minimized.

Copy link
Member Author

savingoyal commented Feb 10, 2020

Fixes #108

savingoyal added 3 commits Feb 10, 2020
…a packages

Looks like conda doesn't reliably/correctly update the
`env/conda-meta/<package>.json` file which Metaflow relies on to identify
which packages should be moved to S3 and their installation order on AWS Batch.
In this specific scenario, when the compute needs to execute on AWS Batch,
Metaflow coaxes conda to generate an environment for `linux-64` architectures
using only `.tar.bz2` dependencies, irrespective of where conda is executed from
(usually a user's macOS or linux instance). In some very specific scenarios,
conda correctly picks up the `.tar.bz2` dependencies from upstream repos but fails
to update the manifest, incorrectly pointing to `.conda` dependencies instead. This
leads to incorrect bundling of packages and the user job fails when executing on
AWS Batch since we end up packaging a .conda dependency as a .tar.bz2 dependency.
Copy link
Contributor

romain-intel left a comment

LGTM.

@savingoyal savingoyal merged commit 52053e2 into master Feb 10, 2020
2 checks passed
2 checks passed
Test on ubuntu-latest
Details
Test on macos-latest
Details
savingoyal added a commit that referenced this pull request Feb 10, 2020
1. Pin click to v7.0 or greater (#107)
2. Add checks to conda-package metadata to guard against .conda packages (#118)
@savingoyal savingoyal deleted the conda-bug branch Feb 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.