Conda Env Support #57

carolineechen · 2023-04-20T01:58:33Z

Supports creating and using conda environments on the cluster for running functions. CondaEnv can be created in the following ways

passing in a local .yaml file representing the env (from something like conda env export)
passing in a Python dict representing the env
passing in the name of a local conda env

If you already have an env set up on the cluster and simply want to reuse it, that can be done with the Python dict approach, simply passing in `{"name": "env-name"}, as we skip the env construction if the env already exists.

Features:

additional reqs and setup commands can be added
.to(file system or cluster) to sync env
system.install_packages(packages, env)

Notes:

We use Ray's runtime_env to support conda environments, so now fn_type == call will use also use ray.remote() for running the function.
get_fn_by_name and pickling are now done through ray.remote() call as well, so that fn imports and pickling is done within the correct runtime_env and not the base env

Additional features to support (sep PRs)

with activate(env): functionality
get list of environments that exist on a cluster (env prev sent to cluster)

Separate commit:

nested serializing

runhouse/rns/run_module_utils.py

carolineechen · 2023-04-28T21:34:39Z

runhouse/servers/grpc/unary_server.py

@@ -58,11 +57,8 @@ def InstallPackages(self, request, context):
                else:
                    raise ValueError(f"package {package} not recognized")

-                if (str(pkg)) in self._installed_packages:


not sure if this is a common path where we could get a lot of time savings, but a bit misleading, if a user uninstalls the library (e.g. system.run(["uninstall xxx"]), future install package calls won't install the library even though it's not actually installed at that moment

may be better to check if it is currently installed whether than if it was previously installed (auto handled for pip/conda where they won't reinstall existing packages). also what if a user wants to resync over a package (potentially with local changes) but it recognizes that the package is currently installed?

These are good questions, let's chat about them live

dongreenberg

This PR is a beast in the best way, let's chat about it live!

docs/rh_primitives/env.rst

docs/api/env.rst

dongreenberg · 2023-05-09T12:56:45Z

docs/debugging_logging.rst

@@ -25,11 +25,12 @@ Alternatively, to see logs on your local machine while running a remote function
 Debugging
 ~~~~~~~~~

-To set a breakpoint, add ``import pdb; pdb.set_trace()`` to whereever you want to set your debugging session.
+For general debugging that doesn't occur within remote function calls, you can add
+``import pdb; pdb.set_trace()`` whereever you want to set your debugging session.
 If the code is being run locally at the point of the debugger, you'll be able to access the session from your
 local machine. If the code is being run remotely on a cluster, you will need to ssh into the cluster with
 ``ssh cluster-name``, and then run ``screen -r`` inside the cluster. From there, you will see the GRPC logs


Is this still true given the ray point below?

hm I guess it's still true, for if we're debugging runhouse code that's being run locally or remotely, but less applicable for users.

docs/debugging_logging.rst

runhouse/rns/envs/conda_env.py

dongreenberg · 2023-05-09T13:54:09Z

runhouse/servers/grpc/unary_server.py

@@ -58,11 +57,8 @@ def InstallPackages(self, request, context):
                else:
                    raise ValueError(f"package {package} not recognized")

-                if (str(pkg)) in self._installed_packages:


These are good questions, let's chat about them live

tests/test_env.py

dongreenberg · 2023-05-09T13:58:52Z

tests/test_env.py

+# -------- CONDA ENV TESTS ----------- #
+
+
+def _get_conda_env(name="rh-test", python_version="3.10.9", pip_reqs=[]):


I think this could be a fixture?

sure, thinking of waiting for Josh's test setup PR to be merged before adding fixtures/mark for env

tests/test_env.py

dongreenberg · 2023-05-09T14:01:25Z

tests/test_function.py

@@ -126,18 +126,20 @@ def getpid(a=0):

 def test_maps():
    pid_fn = rh.function(getpid, system="^rh-cpu")
-    num_pids = [1] * 50
+    num_pids = [1] * 20


We need to fix the performance here

dongreenberg

This PR is a beast in the best way, let's chat about it live!

carolineechen · 2023-05-10T03:08:28Z

runhouse/rns/run_module_utils.py

+        fn = getattr(rh_config.obj_store.imported_modules[module_name], fn_name)
+
+    cuda_visible_devices = list(range(int(num_gpus)))
+    os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(map(str, cuda_visible_devices))


replace this with setting export RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 prior to ray init

tried setting os.environ["RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES"] = "1" prior to the ray.init in unary_server, but looked like it was still blocking cuda and I would only see 1 gpu within the function call

commit fceb5ee Merge: b904990 6d9332e Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 27 15:04:28 2023 -0400 Merge branch 'main' into conda-env-2 commit 6d9332e Merge: 93fd391 39113f0 Author: Caroline Chen <chen.caroline12@gmail.com> Date: Thu Apr 27 12:14:04 2023 -0400 Base env implementation commit b904990 Merge: e8f484e e080dae Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 24 10:12:36 2023 -0400 Merge branch 'base-env' into conda-env-2 commit e8f484e Author: Caroline <chen.caroline12@gmail.com> Date: Sun Apr 23 23:04:53 2023 -0400 rebase commit 32e7743 Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 20 09:44:17 2023 -0400 updates commit e080dae Author: Caroline <chen.caroline12@gmail.com> Date: Sun Apr 23 23:04:53 2023 -0400 rebase commit 9290e3a Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 22:30:17 2023 -0400 reqs to env commit 8171c50 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 16:20:39 2023 -0400 cluster detection and utils commit 1dc7049 Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 15:01:42 2023 -0400 pkg to file system commit 730b1c0 Author: Caroline <chen.caroline12@gmail.com> Date: Wed Apr 19 21:38:19 2023 -0400 clean up + docs commit 6f40373 Author: Caroline <chen.caroline12@gmail.com> Date: Tue Apr 18 17:03:34 2023 -0400 update some fn behaviors commit e21dfc4 Merge: 0adbd04 7c920c7 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 17:38:21 2023 -0400 Merge branch 'conda-env-2' of github.com:run-house/runhouse into conda-env-2 commit 0adbd04 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 11:23:30 2023 -0400 conda env commit 9493741 Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 22:30:17 2023 -0400 reqs to env commit 693564d Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 16:20:39 2023 -0400 cluster detection and utils commit bab4d41 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 16:20:39 2023 -0400 cluster detection and utils commit 1b61a77 Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 18:47:47 2023 -0400 update local package handling commit 4db366c Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 3 16:35:43 2023 -0400 update reqs handling commit 4fa44b9 Author: Caroline <chen.caroline12@gmail.com> Date: Sat Apr 1 15:02:53 2023 -0400 handle git pkg with ./ commit a7a7e25 Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 15:01:42 2023 -0400 pkg to file system commit 1057e10 Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 12:40:45 2023 -0400 local package syncing commit 7c920c7 Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 17 11:23:30 2023 -0400 conda env commit bc67eaa Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 22:30:17 2023 -0400 reqs to env commit 73144af Author: Caroline <chen.caroline12@gmail.com> Date: Thu Apr 13 18:47:47 2023 -0400 update local package handling commit 45a93df Author: Caroline <chen.caroline12@gmail.com> Date: Mon Apr 3 16:35:43 2023 -0400 update reqs handling commit 6e953fd Author: Caroline <chen.caroline12@gmail.com> Date: Sat Apr 1 15:02:53 2023 -0400 handle git pkg with ./ commit 800427b Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 15:01:42 2023 -0400 pkg to file system commit ea3c8a4 Author: Caroline <chen.caroline12@gmail.com> Date: Fri Mar 31 12:40:45 2023 -0400 local package syncing

carolineechen force-pushed the conda-env-2 branch 8 times, most recently from 8cf285e to f0fcfa6 Compare April 21, 2023 19:20

carolineechen force-pushed the base-env branch from 9493741 to e080dae Compare April 24, 2023 03:05

carolineechen force-pushed the conda-env-2 branch 2 times, most recently from 1049b01 to e8f484e Compare April 24, 2023 03:48

carolineechen force-pushed the base-env branch from e080dae to 3b8f169 Compare April 24, 2023 14:28

carolineechen marked this pull request as ready for review April 24, 2023 14:29

carolineechen force-pushed the conda-env-2 branch from a5096a2 to a579675 Compare April 24, 2023 21:22

carolineechen force-pushed the base-env branch from 3b8f169 to 5765854 Compare April 24, 2023 22:54

carolineechen force-pushed the conda-env-2 branch 2 times, most recently from d50dd53 to b904990 Compare April 25, 2023 20:09

carolineechen commented Apr 26, 2023

View reviewed changes

runhouse/rns/run_module_utils.py Outdated Show resolved Hide resolved

runhouse/rns/run_module_utils.py Outdated Show resolved Hide resolved

carolineechen force-pushed the base-env branch 2 times, most recently from 13e8c14 to 39113f0 Compare April 27, 2023 14:40

carolineechen changed the base branch from base-env to main April 27, 2023 16:22

carolineechen force-pushed the main branch from 6d9332e to 6fb3083 Compare April 27, 2023 19:05

carolineechen force-pushed the conda-env-2 branch 2 times, most recently from 7c6cdca to 918f69a Compare April 28, 2023 21:32

carolineechen commented Apr 28, 2023

View reviewed changes

carolineechen force-pushed the conda-env-2 branch 5 times, most recently from c1ba7a7 to 008944b Compare May 9, 2023 00:36

carolineechen force-pushed the conda-env-2 branch from 008944b to 2c23f8b Compare May 9, 2023 01:09

dongreenberg reviewed May 9, 2023

View reviewed changes

carolineechen force-pushed the conda-env-2 branch 3 times, most recently from c5af32a to 575e3de Compare May 10, 2023 03:07

carolineechen commented May 10, 2023

View reviewed changes

carolineechen force-pushed the conda-env-2 branch 2 times, most recently from 2a376c2 to 21f5a16 Compare May 11, 2023 01:03

dongreenberg approved these changes May 11, 2023

View reviewed changes

carolineechen added 5 commits May 11, 2023 13:18

update grpc

2c51dbb

env-related docs

71011d1

address comments

e16771d

additional conda env changes

6bcdf97

carolineechen force-pushed the conda-env-2 branch 2 times, most recently from de80249 to 37d3db7 Compare May 11, 2023 18:09

torch and testing rebase and updates

60eff80

carolineechen force-pushed the conda-env-2 branch from 37d3db7 to 60eff80 Compare May 11, 2023 18:11

carolineechen merged commit a2b1e78 into main May 11, 2023

carolineechen deleted the conda-env-2 branch May 11, 2023 18:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conda Env Support #57

Conda Env Support #57

carolineechen commented Apr 20, 2023 •

edited

Loading

carolineechen Apr 28, 2023 •

edited

Loading

dongreenberg May 9, 2023

dongreenberg left a comment

dongreenberg May 9, 2023

carolineechen May 10, 2023

dongreenberg May 9, 2023

dongreenberg May 9, 2023

carolineechen May 10, 2023

dongreenberg May 9, 2023

dongreenberg left a comment

carolineechen May 10, 2023

carolineechen May 10, 2023

		# -------- CONDA ENV TESTS ----------- #


		def _get_conda_env(name="rh-test", python_version="3.10.9", pip_reqs=[]):

Conda Env Support #57

Conda Env Support #57

Conversation

carolineechen commented Apr 20, 2023 • edited Loading

carolineechen Apr 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongreenberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongreenberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolineechen commented Apr 20, 2023 •

edited

Loading

carolineechen Apr 28, 2023 •

edited

Loading