Clean up conda environment when `pip install` fails during `conda create` #5505

harupy · 2022-03-17T06:50:04Z

Signed-off-by: harupy 17039389+harupy@users.noreply.github.com

What changes are proposed in this pull request?

conda doesn't clean up an environment when pip install fails during conda create. This behavior results in creating an incomplete conda environment and causes the following issue:

mlflow models serve ...
# fails a temporary network issue and leaves an incomplete conda environment

mlflow models serve ...
# serves the model in the incomplete environment created in the first attempt

How is this patch tested?

Unit test

Does this PR change the documentation?

No. You can skip the rest of this section.
Yes. Make sure the changed pages / sections render correctly by following the steps below.

Check the status of the ci/circleci: build_doc check. If it's successful, proceed to the
next step, otherwise fix it.
Click Details on the right to open the job page of CircleCI.
Click the Artifacts tab.
Click docs/build/html/index.html.
Find the changed pages / sections and make sure they render correctly.

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

(Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change.)

What component(s), interfaces, languages, and integrations does this PR affect?

Components

Interface

area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
area/windows: Windows support

Language

language/r: R APIs and clients
language/java: Java APIs and clients
language/new: Proposals for new client languages

Integrations

integrations/azure: Azure and Azure ML integrations
integrations/sagemaker: SageMaker integrations
integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

WeichenXu123 · 2022-03-17T11:35:39Z

mlflow/utils/conda.py

+                    ],
+                    stream_output=True,
+                )
+        except Exception:


Even if subprocess succeeds, we need to also check CondaEnvException in subprocess stdout/stderr output.

I suggest do:

exit_code, stdout, stderr = process.exec_cmd(..., throw_on_error=False) if exit_code != 0 or "CondaEnvException" in stdout or "CondaEnvException" in stderr: # clean created env here...

Even if subprocess succeeds, we need to also check CondaEnvException in subprocess stdout/stderr output.

Why?

I actually found conda create fails when pip install fails.

Here's a proof:

% cat conda.yaml name: yes channels: - conda-forge dependencies: - python=3.7.12 - pip - pip: - mlflow==999.999.999 % conda env create -f conda.yaml Collecting package metadata (repodata.json): done Solving environment: done Preparing transaction: done Verifying transaction: done Executing transaction: done Ran pip subprocess with arguments: ['/Users/harutakakawamura/.pyenv/versions/miniconda3-4.7.12/envs/yes/bin/python', '-m', 'pip', 'install', '-U', '-r', '/Users/harutakakawamura/Desktop/repositories/mlflow/condaenv.j5yolst2.requirements.txt'] Pip subprocess output: Pip subprocess error: ERROR: Could not find a version that satisfies the requirement mlflow==999.999.999 (from versions: 0.0.1, 0.1.0, 0.2.0, 0.2.1, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.9.0.1, 0.9.1, 1.0.0, 1.1.0, 1.1.1.dev0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.9.0, 1.9.1, 1.10.0, 1.11.0, 1.12.0, 1.12.1, 1.13, 1.13.1, 1.14.0, 1.14.1, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.20.1, 1.20.2, 1.21.0, 1.22.0, 1.23.0, 1.23.1, 1.24.0) ERROR: No matching distribution found for mlflow==999.999.999 CondaEnvException: Pip failed % echo $? 1

This indicates CondaEnvException aborts conda create.

Please let me know if there are cases where conda succeeds even when CondaEnvException is raised.

I even tried installing an invalid python version (in this case, the conda env doesn't get built (obviously)) but it still returns (1) for this failure.

~/Documents via 🅒 base took 10s ➜ cat conda.yaml name: hodor channels: - conda-forge dependencies: - python=3.14.1 - pip - pip: - mlflow=42.42.42 (base) ~/Documents via 🅒 base ➜ conda env create -f conda.yaml Collecting package metadata (repodata.json): done Solving environment: failed ResolvePackageNotFound: - python=3.14.1 (base) ~/Documents via 🅒 base took 9s ➜ echo $? 1 ➜ conda env list # conda environments: # base * /Users/benwilson/opt/anaconda3 diviner-dev /Users/benwilson/opt/anaconda3/envs/diviner-dev mlflow-dev-env /Users/benwilson/opt/anaconda3/envs/mlflow-dev-env

Thanks Ben for the investigation :)

Given these results, I think the current approach is ok. @WeichenXu123 What do you think?

WeichenXu123 · 2022-03-17T11:38:15Z

mlflow/utils/conda.py

+                        "-n",
+                        project_env_name,
+                        "--file",
+                        conda_env_path,


Question:
Shall we add "--yes" argument for conda env create command ?

conda env create doesn't have --yes flag:

usage: conda-env create [-h] [-f FILE] [-n ENVIRONMENT | -p PATH] [-C] [-k] [--offline] [--force] [--json] [-v] [-q] [remote_definition] Create an environment based on an environment file Options: positional arguments: remote_definition remote environment definition / IPython notebook optional arguments: -h, --help Show this help message and exit. -f FILE, --file FILE environment definition file (default: environment.yml) --force force creation of environment (removing a previously existing environment of the same name). Target Environment Specification: -n ENVIRONMENT, --name ENVIRONMENT Name of environment. -p PATH, --prefix PATH Full path to environment location (i.e. prefix). Networking Options: -C, --use-index-cache Use cache of channel index files, even if it has expired. -k, --insecure Allow conda to perform "insecure" SSL connections and transfers. Equivalent to setting 'ssl_verify' to 'false'. --offline Offline mode. Don't connect to the Internet. Output, Prompt, and Flow Control Options: --json Report all output as json. Suitable for using conda programmatically. -v, --verbose Use once for info, twice for debug, three times for trace. -q, --quiet Do not display progress bar.

Added --yes to conda create for safety.

Without --yes, conda create does ask for confirmation (Proceed [y]/n?) but it appears subprocess.Popen(..., stdin= subprocess.PIPE).communicate() automatically accepts it. We can confirm this behavior using the code below:

import subprocess import uuid prc = subprocess.Popen( [ "conda", "create", "-c", "conda-forge", "-n", uuid.uuid4().hex, "python=3.7", ], stdin=subprocess.PIPE, ) prc.communicate()

Can confirm.

~/Documents via 🅒 base ➜ conda env list # conda environments: # base * /Users/benwilson/opt/anaconda3 diviner-dev /Users/benwilson/opt/anaconda3/envs/diviner-dev hodor /Users/benwilson/opt/anaconda3/envs/hodor mlflow-dev-env /Users/benwilson/opt/anaconda3/envs/mlflow-dev-env (base) ~/Documents via 🅒 base took 2s ➜ (cat conda.yaml && conda env remove -n hodor) name: hodor channels: - conda-forge dependencies: - python=3.9 - pip - pip: - mlflow==1.20.0 Remove all packages in environment /Users/benwilson/opt/anaconda3/envs/hodor: (base) ~/Documents via 🅒 base took 9s ➜ conda env list # conda environments: # base * /Users/benwilson/opt/anaconda3 diviner-dev /Users/benwilson/opt/anaconda3/envs/diviner-dev mlflow-dev-env /Users/benwilson/opt/anaconda3/envs/mlflow-dev-env (base)

Subprocesses auto-accept :)

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

BenWilson2

I think the implementation will work well :) LGTM!

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

…ate` (mlflow#5505) * Clean up conda environment when pip install fails Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> * refactor _list_conda_environments Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> * added yes to conda create Signed-off-by: harupy <17039389+harupy@users.noreply.github.com> * nit Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

Clean up conda environment when pip install fails

a6dd990

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

github-actions bot added the rn/none List under Small Changes in Changelogs. label Mar 17, 2022

refactor _list_conda_environments

ea970b8

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

harupy requested a review from WeichenXu123 March 17, 2022 07:14

WeichenXu123 reviewed Mar 17, 2022

View reviewed changes

harupy requested a review from WeichenXu123 March 17, 2022 12:08

added yes to conda create

8452b6e

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

BenWilson2 self-requested a review March 18, 2022 01:39

BenWilson2 approved these changes Mar 18, 2022

View reviewed changes

nit

fa86214

Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>

harupy merged commit 6a6d236 into mlflow:master Mar 18, 2022

harupy deleted the clean-up-conda-env branch March 18, 2022 02:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up conda environment when `pip install` fails during `conda create` #5505

Clean up conda environment when `pip install` fails during `conda create` #5505

harupy commented Mar 17, 2022

WeichenXu123 Mar 17, 2022

WeichenXu123 Mar 17, 2022

harupy Mar 17, 2022

harupy Mar 17, 2022 •

edited

Loading

harupy Mar 17, 2022 •

edited

Loading

harupy Mar 17, 2022 •

edited

Loading

BenWilson2 Mar 17, 2022 •

edited

Loading

harupy Mar 18, 2022

harupy Mar 18, 2022 •

edited

Loading

WeichenXu123 Mar 17, 2022

harupy Mar 17, 2022 •

edited

Loading

harupy Mar 17, 2022 •

edited

Loading

harupy Mar 18, 2022 •

edited

Loading

BenWilson2 Mar 18, 2022 •

edited

Loading

BenWilson2 left a comment

Clean up conda environment when pip install fails during conda create #5505

Clean up conda environment when pip install fails during conda create #5505

Conversation

harupy commented Mar 17, 2022

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change the documentation?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

WeichenXu123 Mar 17, 2022

Choose a reason for hiding this comment

WeichenXu123 Mar 17, 2022

Choose a reason for hiding this comment

harupy Mar 17, 2022

Choose a reason for hiding this comment

harupy Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

harupy Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

harupy Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

BenWilson2 Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

harupy Mar 18, 2022

Choose a reason for hiding this comment

harupy Mar 18, 2022 • edited Loading

Choose a reason for hiding this comment

WeichenXu123 Mar 17, 2022

Choose a reason for hiding this comment

harupy Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

harupy Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

harupy Mar 18, 2022 • edited Loading

Choose a reason for hiding this comment

BenWilson2 Mar 18, 2022 • edited Loading

Choose a reason for hiding this comment

BenWilson2 left a comment

Choose a reason for hiding this comment

Clean up conda environment when `pip install` fails during `conda create` #5505

Clean up conda environment when `pip install` fails during `conda create` #5505

harupy Mar 17, 2022 •

edited

Loading

harupy Mar 17, 2022 •

edited

Loading

harupy Mar 17, 2022 •

edited

Loading

BenWilson2 Mar 17, 2022 •

edited

Loading

harupy Mar 18, 2022 •

edited

Loading

harupy Mar 17, 2022 •

edited

Loading

harupy Mar 17, 2022 •

edited

Loading

harupy Mar 18, 2022 •

edited

Loading

BenWilson2 Mar 18, 2022 •

edited

Loading