New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Passing objects to a Service #244

Closed
botev opened this Issue Dec 12, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@botev

botev commented Dec 12, 2017

Sorry, as this not nesscarily a bug, but currently I did not manage to find anyway of passing an external object to a service, except via global variables.
Technically I want something like this:

class ClusterService(Service):
    def __init__(self, conn, lock: RLock, cluster: Cluster):
        super(ClusterService, self).__init__(conn)
        self.lock: RLock = lock
        self.cluster: Cluster = cluster

    def print_stats(self, project, status, print_pool):
        with self.lock:
            self.cluster.print_stats(project, status, print_pool)

    def submit_job(self, project, name, command, res_file, out_file, err_file, priority=None):
        with self.lock:
            self.cluster.submit_job(project, name, command,
                                    res_file, out_file, err_file, priority)

    def delete_job(self, project, names_or_uids):
        with self.lock:
            for name_or_uid in names_or_uids:
                self.cluster.delete_job(project, name_or_uid)

The issue is that there is no way to pass a partially constructed object to the ThreadedServer. Is there any work around this?

@coldfix

This comment has been minimized.

Collaborator

coldfix commented Dec 13, 2017

Hi,

up to 3.4 probably the simplest option is to pass parameters via the config dictionary that will be available via self._conn._config.

otherwise, you could do it like this, e.g.:

class FooService(rpyc.Service):
  def __init__(self, conn, lock, cluster):
    super().__init__(conn)
    self.lock = lock
    self.cluster = cluster
  def __call__(self, conn):
    return self.__class__(conn, self.lock, self.cluster)

server = rpyc.ThreadedServer(FooService(None, Lock(), cluster))

If you handle only a single connection (e.g. OneShotServer, or connect methods), you could do it better like this:

class FooService(rpyc.Service):
  def __init__(self, lock, cluster):
    self.lock = lock
    self.cluster = cluster
  def __call__(self, conn):
    super().__init__(conn)
    return self

server = rpyc.ThreadedServer(FooService(Lock(), cluster))

(which acts similar to binding arguments)

or more properly separating the individual responsibilities:

class FooFactory:
  def __init__(self, lock, cluster):
    self._conn = None
    self.lock = lock
    self.cluster = cluster
  def get_service_name(self):
    return "FOO"
  def get_service_aliases(self):
    return ("FOO",)
  def __call__(self, conn):
    return FooService(conn, self.lock, self.cluster)

class FooService(rpyc.Service):
  def __init__(self, conn, lock, cluster):
    super().__init__(conn)
    self.lock = lock
    self.cluster = cluster

server = rpyc.ThreadedServer(FooFactory(Lock(), cluster))

I agree, that this is poorly separated, rpyc.Service serves two different purposes (Factory + Service object) and I'm open for fixing this.

@botev

This comment has been minimized.

botev commented Dec 14, 2017

So I'm fine with using your solution. As a long-term might be worth to make the ThreadServrer accept kwargs which are passed to the Service object, to make this cleaner.

coldfix added a commit that referenced this issue Dec 21, 2017

Can pass Service as instance or class!
This allows

- sharing the same service object among multiple clients on a server
  (as was the expectation by the user in #198)

- using an pre-initialized instance when connecting a client or hosting
  a oneshot server (partial resolution for #244)

coldfix added a commit that referenced this issue Dec 21, 2017

Can pass Service as instance or class!
This allows

- sharing the same service object among multiple clients on a server
  (as was the expectation by the user in #198)

- using an pre-initialized instance when connecting a client or hosting
  a oneshot server (partial resolution for #244)

@coldfix coldfix closed this in 3d01fe5 Dec 21, 2017

@coldfix

This comment has been minimized.

Collaborator

coldfix commented Dec 21, 2017

Hi, my recommended solution as of now is to use:

def classpartial(*args, **kwargs):
    """Bind arguments to a class's __init__."""
    cls, args = args[0], args[1:]
    class Partial(cls):
        __doc__ = cls.__doc__
        def __new__(self):
            return cls(*args, **kwargs)
    Partial.__name__ = cls.__name__
    return Partial

server = rpyc.ThreadedServer(classpartial(FooService, Lock(), cluster))

This function will also be available as rpyc.utils.helpers.classpartial in the upcoming release.

However, also note that the connection will not be passed to Service.__init__ anymore in the next release! Instead, you get it as on_connect(self, conn).

Best, Thomas

@botev

This comment has been minimized.

botev commented Dec 21, 2017

I guess when you switch fully to on_connect than we can just pass the already constructed object directly. Thanks for the feedback.

@coldfix

This comment has been minimized.

Collaborator

coldfix commented Dec 21, 2017

That's true only if you either serve only a single connection (e.g. as client or OneShotServer) or if want to use the same service object for the all your clients.

coldfix added a commit that referenced this issue Jun 11, 2018

Release rpyc 4.0.0
This release brings a few minor backward incompatibilities, so be sure to read
on before upgrading. However, fear not: the ones that are most likely relevant
to you have a relatively simple migration path.

Backward Incompatibilities
^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``classic.teleport_function`` now executes the function in the connection's
  namespace by default. To get the old behaviour, use
  ``teleport_function(conn, func, conn.modules[func.__module__].__dict__)``
  instead.

* Changed signature of ``Service.on_connect`` and ``on_disconnect``, adding
  the connection as argument.

* Changed signature of ``Service.__init__``, removing the connection argument

* no longer store connection as ``self._conn``. (allows services that serve
  multiple clients using the same service object, see `#198`_).

* ``SlaveService`` is now split into two asymetric classes: ``SlaveService``
  and ``MasterService``. The slave exposes functionality to the master but can
  not anymore access remote objects on the master (`#232`_, `#248`_).
  If you were previously using ``SlaveService``, you may experience problems
  when feeding the slave with netrefs to objects on the master. In this case, do
  any of the following:

  * use ``ClassicService`` (acts exactly like the old ``SlaveService``)
  * use ``SlaveService`` with a ``config`` that allows attribute access etc
  * use ``rpyc.utils.deliver`` to feed copies rather than netrefs to
    the slave

* ``RegistryServer.on_service_removed`` is once again called whenever a service
  instance is removed, making it symmetric to ``on_service_added`` (`#238`_)
  This reverts PR `#173`_ on issue `#172`_.

* Removed module ``rpyc.experimental.splitbrain``. It's too confusing and
  undocumented for me and I won't be developing it, so better remove it
  altogether. (It's still available in the ``splitbrain`` branch)

* Removed module ``rpyc.experimental.retunnel``. Seemingly unused anywhere, no
  documentation, no clue what this is about.

* ``bin/rpyc_classic.py`` will bind to ``127.0.0.1`` instead of ``0.0.0.0`` by
  default

* ``SlaveService`` no longer serves exposed attributes (i.e., it now uses
  ``allow_exposed_attrs=False``)

* Exposed attributes no longer hide plain attributes if one otherwise has the
  required permissions to access the plain attribute. (`#165`_)

.. _#165: #165
.. _#172: #172
.. _#173: #173
.. _#198: #198
.. _#232: #232
.. _#238: #238
.. _#248: #248

What else is new
^^^^^^^^^^^^^^^^

* teleported functions will now be defined by default in the globals dict

* Can now explicitly specify globals for teleported functions

* Can now use streams as context manager

* keep a hard reference to connection in netrefs, may fix some ``EOFError``
  issues, in particular on Jython related (`#237`_)

* handle synchronous and asynchronous requests uniformly

* fix deadlock with connections talking to each other multithreadedly (`#270`_)

* handle timeouts cumulatively

* fix possible performance bug in ``Win32PipeStream.poll`` (oversleeping)

* use readthedocs theme for documentation (`#269`_)

* actually time out sync requests (`#264`_)

* clarify documentation concerning exceptions in ``Connection.ping`` (`#265`_)

* fix ``__hash__`` for netrefs (`#267`_, `#268`_)

* rename ``async`` module to ``async_`` for py37 compatibility (`#253`_)

* fix ``deliver()`` from IronPython to CPython2 (`#251`_)

* fix brine string handling in py2 IronPython (`#251`_)

* add gevent_ Server. For now, this requires using ``gevent.monkey.patch_all()``
  before importing for rpyc. Client connections can already be made without
  further changes to rpyc, just using gevent's monkey patching. (`#146`_)

* add function ``rpyc.lib.spawn`` to spawn daemon threads

* fix several bugs in ``bin/rpycd.py`` that crashed this script on startup
  (`#231`_)

* fix problem with MongoDB, or more generally any remote objects that have a
  *catch-all* ``__getattr__`` (`#165`_)

* fix bug when copying remote numpy arrays (`#236`_)

* added ``rpyc.utils.helpers.classpartial`` to bind arguments to services (`#244`_)

* can now pass services optionally as instance or class (could only pass as
  class, `#244`_)

* The service is now charged with setting up the connection, doing so in
  ``Service._connect``. This allows using custom protocols by e.g. subclassing
  ``Connection``.  More discussions and related features in `#239`_-`#247`_.

* service can now easily override protocol handlers, by updating
  ``conn._HANDLERS`` in ``_connect`` or ``on_connect``. For example:
  ``conn._HANDLERS[HANDLE_GETATTR] = self._handle_getattr``.

* most protocol handlers (``Connection._handle_XXX``) now directly get the
  object rather than its ID as first argument. This makes overriding
  individual handlers feel much more high-level. And by the way it turns out
  that this fixes two long-standing issues (`#137`_, `#153`_)

* fix bug with proxying context managers (`#228`_)

* expose server classes from ``rpyc`` top level module

* fix logger issue on jython

.. _#137: #137
.. _#146: #146
.. _#153: #153
.. _#165: #165
.. _#228: #228
.. _#231: #231
.. _#236: #236
.. _#237: #237
.. _#239: #239
.. _#244: #244
.. _#247: #247
.. _#251: #251
.. _#253: #253
.. _#264: #264
.. _#265: #265
.. _#267: #267
.. _#268: #268
.. _#269: #269
.. _#270: #270

.. _gevent: http://www.gevent.org/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment