-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conda Env Support #57
Conversation
8cf285e
to
f0fcfa6
Compare
1049b01
to
e8f484e
Compare
a5096a2
to
a579675
Compare
d50dd53
to
b904990
Compare
13e8c14
to
39113f0
Compare
7c6cdca
to
918f69a
Compare
@@ -58,11 +57,8 @@ def InstallPackages(self, request, context): | |||
else: | |||
raise ValueError(f"package {package} not recognized") | |||
|
|||
if (str(pkg)) in self._installed_packages: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure if this is a common path where we could get a lot of time savings, but a bit misleading, if a user uninstalls the library (e.g. system.run(["uninstall xxx"]), future install package calls won't install the library even though it's not actually installed at that moment
may be better to check if it is currently installed whether than if it was previously installed (auto handled for pip/conda where they won't reinstall existing packages). also what if a user wants to resync over a package (potentially with local changes) but it recognizes that the package is currently installed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are good questions, let's chat about them live
c1ba7a7
to
008944b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR is a beast in the best way, let's chat about it live!
@@ -25,11 +25,12 @@ Alternatively, to see logs on your local machine while running a remote function | |||
Debugging | |||
~~~~~~~~~ | |||
|
|||
To set a breakpoint, add ``import pdb; pdb.set_trace()`` to whereever you want to set your debugging session. | |||
For general debugging that doesn't occur within remote function calls, you can add | |||
``import pdb; pdb.set_trace()`` whereever you want to set your debugging session. | |||
If the code is being run locally at the point of the debugger, you'll be able to access the session from your | |||
local machine. If the code is being run remotely on a cluster, you will need to ssh into the cluster with | |||
``ssh cluster-name``, and then run ``screen -r`` inside the cluster. From there, you will see the GRPC logs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this still true given the ray point below?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm I guess it's still true, for if we're debugging runhouse code that's being run locally or remotely, but less applicable for users.
@@ -58,11 +57,8 @@ def InstallPackages(self, request, context): | |||
else: | |||
raise ValueError(f"package {package} not recognized") | |||
|
|||
if (str(pkg)) in self._installed_packages: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are good questions, let's chat about them live
# -------- CONDA ENV TESTS ----------- # | ||
|
||
|
||
def _get_conda_env(name="rh-test", python_version="3.10.9", pip_reqs=[]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this could be a fixture?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, thinking of waiting for Josh's test setup PR to be merged before adding fixtures/mark for env
@@ -126,18 +126,20 @@ def getpid(a=0): | |||
|
|||
def test_maps(): | |||
pid_fn = rh.function(getpid, system="^rh-cpu") | |||
num_pids = [1] * 50 | |||
num_pids = [1] * 20 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to fix the performance here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR is a beast in the best way, let's chat about it live!
c5af32a
to
575e3de
Compare
fn = getattr(rh_config.obj_store.imported_modules[module_name], fn_name) | ||
|
||
cuda_visible_devices = list(range(int(num_gpus))) | ||
os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(map(str, cuda_visible_devices)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
replace this with setting export RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1
prior to ray init
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tried setting os.environ["RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES"] = "1"
prior to the ray.init
in unary_server, but looked like it was still blocking cuda and I would only see 1 gpu within the function call
2a376c2
to
21f5a16
Compare
commit fceb5ee Merge: b904990 6d9332e Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 27 15:04:28 2023 -0400 Merge branch 'main' into conda-env-2 commit 6d9332e Merge: 93fd391 39113f0 Author: Caroline Chen <chen.caroline12@gmail.com> Date: Thu Apr 27 12:14:04 2023 -0400 Base env implementation commit b904990 Merge: e8f484e e080dae Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 24 10:12:36 2023 -0400 Merge branch 'base-env' into conda-env-2 commit e8f484e Author: Caroline <chen.caroline12@gmail.com> Date: Sun Apr 23 23:04:53 2023 -0400 rebase commit 32e7743 Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 20 09:44:17 2023 -0400 updates commit e080dae Author: Caroline <chen.caroline12@gmail.com> Date: Sun Apr 23 23:04:53 2023 -0400 rebase commit 9290e3a Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 22:30:17 2023 -0400 reqs to env commit 8171c50 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 16:20:39 2023 -0400 cluster detection and utils commit 1dc7049 Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 15:01:42 2023 -0400 pkg to file system commit 730b1c0 Author: Caroline <chen.caroline12@gmail.com> Date: Wed Apr 19 21:38:19 2023 -0400 clean up + docs commit 6f40373 Author: Caroline <chen.caroline12@gmail.com> Date: Tue Apr 18 17:03:34 2023 -0400 update some fn behaviors commit e21dfc4 Merge: 0adbd04 7c920c7 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 17:38:21 2023 -0400 Merge branch 'conda-env-2' of github.com:run-house/runhouse into conda-env-2 commit 0adbd04 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 11:23:30 2023 -0400 conda env commit 9493741 Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 22:30:17 2023 -0400 reqs to env commit 693564d Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 16:20:39 2023 -0400 cluster detection and utils commit bab4d41 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 16:20:39 2023 -0400 cluster detection and utils commit 1b61a77 Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 18:47:47 2023 -0400 update local package handling commit 4db366c Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 3 16:35:43 2023 -0400 update reqs handling commit 4fa44b9 Author: Caroline <chen.caroline12@gmail.com> Date: Sat Apr 1 15:02:53 2023 -0400 handle git pkg with ./ commit a7a7e25 Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 15:01:42 2023 -0400 pkg to file system commit 1057e10 Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 12:40:45 2023 -0400 local package syncing commit 7c920c7 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 11:23:30 2023 -0400 conda env commit bc67eaa Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 22:30:17 2023 -0400 reqs to env commit 73144af Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 18:47:47 2023 -0400 update local package handling commit 45a93df Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 3 16:35:43 2023 -0400 update reqs handling commit 6e953fd Author: Caroline <chen.caroline12@gmail.com> Date: Sat Apr 1 15:02:53 2023 -0400 handle git pkg with ./ commit 800427b Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 15:01:42 2023 -0400 pkg to file system commit ea3c8a4 Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 12:40:45 2023 -0400 local package syncing
de80249
to
37d3db7
Compare
37d3db7
to
60eff80
Compare
see #20
Supports creating and using conda environments on the cluster for running functions. CondaEnv can be created in the following ways
.yaml
file representing the env (from something likeconda env export
)If you already have an env set up on the cluster and simply want to reuse it, that can be done with the Python dict approach, simply passing in `{"name": "env-name"}, as we skip the env construction if the env already exists.
Features:
.to(file system or cluster)
to sync envsystem.install_packages(packages, env)
Notes:
runtime_env
to support conda environments, so nowfn_type == call
will use also useray.remote()
for running the function.get_fn_by_name
and pickling are now done throughray.remote()
call as well, so that fn imports and pickling is done within the correctruntime_env
and not the base envAdditional features to support (sep PRs)
with activate(env):
functionalitySeparate commit: