Switch a module to use an rpyc server's modules

[david-drinn]

Take this code for example:

# ... snipping the import and setup boilerplate.
conn = server.classic_connect()

# A module that exists on the client only.
import useful_module

# Auto inspect useful_module and switch its imported
# modules, functions, built-ins, etc. to the rpyc server's.
rpyc.switch_module(useful_module, conn)

# Run on rpyc server transparently.
result = useful_module.do_something_useful()

print result

# And be able to switch everything back if no connection provided.
rpyc.switch_module(useful_module)

This of course assumes some things, such as the imported modules, functions, etc. used by useful_module are available on the rpyc server.

The biggest benefit I see here, is I can run a module on another host without touching the code or scp'ing it over and referencing conn every time I want to use the module. Most of the module's code that is host agnostic will run on the client, while mostly just the code required to run on the rpyc server is automatically switched to the rpyc server's. (I'm thinking of requirements such as subprocess.Popen, os.listdir, open, and sys.path.)

Once switch_module() is run, useful_module's functions run on the server transparently (when necessary), until you switch back.

I'm new to rpyc, so I'm curious what downsides others will immediately recognize, or if there exists a better way already.

Would anyone else find this feature useful?

I created an early prototype in a few hours that works only on a few modules I'm currently using. Automating this for most use cases seems feasible with python's introspection tools.

[coldfix]

I think this is probably too unreliable and hard to maintain for all but simple cases. A more reliable solution is probably to use a mechanism similar to the one used for the zerodeploy server - i.e. use plumbum to copy over the code. From rpyc/utils/zerodeploy.py:

        rpyc_root = local.path(rpyc.__file__).up()
        self._tmpdir_ctx = remote_machine.tempdir()
        tmp = self._tmpdir_ctx.__enter__()
        copy(rpyc_root, tmp / "rpyc")

And install dependencies on the remote using pip.

Closing this, but feel free to share your code if you still have it! I'd be interested to see how you did it.