Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[runtime env] Make env_vars take effect when pip install packages #22730

Merged
merged 1 commit into from
Mar 3, 2022

Conversation

Catch-Bull
Copy link
Contributor

@Catch-Bull Catch-Bull commented Mar 1, 2022

Why are these changes needed?

Previously, for the stability of pip installation, we set env to empty, but when pip installs some gzip package, maybe need env_vars. like this issue: #22610

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Copy link
Contributor

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also make this apply to conda while we're at it?

@architkulkarni architkulkarni self-assigned this Mar 1, 2022
Copy link
Contributor

@architkulkarni architkulkarni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this feature, and nice test! In my opinion it's okay to leave the conda case for the future. We haven't had anyone ask about environment variables for conda, but several users have asked about it for pip.

await check_output_cmd(pip_install_cmd, logger=logger, cwd=cwd, env={})
pip_env = os.environ.copy()
pip_env.update(env_vars)
await check_output_cmd(pip_install_cmd, logger=logger, cwd=cwd, env=pip_env)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the common case, env_vars is empty, so here we would be changing env={} to env=os.environ.copy(). These two should behave exactly the same with check_output_cmd, right? Just want to double check.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the argument env means overwrite not update, for example, you can run this command:

python test_subprocess.py

test_subprocess.py:

import subprocess

print(subprocess.check_output(["env"], env={}).decode("utf-8"))

It will print nothing, env=None and env=os.environ.copy() has same behave, you can verify it by the above command.

@Catch-Bull
Copy link
Contributor Author

Can we also make this apply to conda while we're at it?

maybe I should open a new PR to make env_vars apply to conda, we haven't used conda in ray cluster before, maybe we need to see if there is any problem

f.writelines(python_code)

gz_filename = os.path.join(tmpdir, package_name + ".tar.gz")
subprocess.check_call(["tar", "-zcvf", gz_filename, package_name])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good test case!

@architkulkarni
Copy link
Contributor

rllib:learning_tests_pendulum_ddpg flaky on master.

@architkulkarni architkulkarni added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Mar 3, 2022
@edoakes edoakes merged commit 207d93a into ray-project:master Mar 3, 2022
frosk1 pushed a commit to frosk1/ray that referenced this pull request Mar 9, 2022
…y-project#22730)

Previously, for the stability of pip installation, we set env to empty, but when pip installs some gzip package, maybe need env_vars.  like this issue: ray-project#22610
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants