New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async requests in reversed order #158

Closed
aviralg opened this Issue Feb 14, 2015 · 7 comments

Comments

Projects
None yet
3 participants
@aviralg

aviralg commented Feb 14, 2015

I am making multiple async requests in a loop. What I observe is that these requests are serviced in batches and in a particular batch all the requests are served in reverse order.

Here is a code snippet -

def _run_simulation(self, callback):
    async_callback = rpyc.async(callback)
    x = 0
    while x < 1000:
        async_callback({ "runtime" : time.time()})
        time.sleep(0.5)
        x += 1

Is it possible to ensure that the requests are somehow serviced in correct order ?
I am aware of the weakref issue and holding the async_callback reference as well.

@power-zhy

This comment has been minimized.

power-zhy commented Aug 21, 2015

I hit the same problem too, and I found it only occurs when both async call and by-reference parameter is used, and two async request should be send simultaneously. (Here by-reference means rpyc server would call back to fetch data inside the parameter, see _box function in protocol.py for more detail)
One temporary solution which I used is, use pickle module to pack the parameter to a string so that it would be transferred by-value instead of by-reference.

I wonder if it can be easily solved in rpyc, the problem is, when server trying to send GETATTR call back to client to fetch specific data in parameters, it launch a sync call and start waiting, then it receives another async request instead of the sync response, which gives the second async request a higher order.
When the server side trying to send GETATTR calls, can it use async as well if the original call from client to server is async?

@coldfix

This comment has been minimized.

Collaborator

coldfix commented Jul 3, 2017

Can you give a full example?

@power-zhy

This comment has been minimized.

power-zhy commented Jul 23, 2017

Sorry for being late, here is one simple example.
Server:

import rpyc
from rpyc.utils.server import ThreadedServer

class MyService(rpyc.Service):
	def exposed_fun(self, id, data):
		print("Exposed fun called with [{}].".format(id))
		return data.x() + data.y()

if __name__ == "__main__":
	server = ThreadedServer(MyService, port = 12345)
	server.start()

Client:

import rpyc

class MyStruct(object):
	def __init__(self, id, x, y):
		self.id = id
		self.x = x
		self.y = y
	
	def exposed_x(self):
		print("Getting x from [{}], value is {}.".format(self.id, self.x))
		return self.x
	
	def exposed_y(self):
		print("Getting y from [{}], value is {}.".format(self.id, self.y))
		return self.y


conn = rpyc.connect("localhost", 12345)
remote_fun = rpyc.async(conn.root.fun)
res1 = remote_fun("aa", MyStruct("aa", 1,2))
res2 = remote_fun("bb", MyStruct("bb", 3,4))
print(res1.value, res2.value)
input()

After running, the results are:
Server:

Exposed fun called with [bb].
Exposed fun called with [aa].

Client:

Getting x from [bb], value is 3.
Getting y from [bb], value is 4.
Getting x from [aa], value is 1.
Getting y from [aa], value is 2.
3 7

In client side we call remote function with id 'aa' first, but server got 'bb' first, which is reversed.

@coldfix

This comment has been minimized.

Collaborator

coldfix commented Jul 24, 2017

Hi,

thanks a lot. I agree with your analysis, with the additional note that this phenomenon occurs already without even accessing the netref:

# Client
import rpyc
class X(object):
    pass
conn = rpyc.connect("localhost", 12345)
remote_fun = rpyc.async(conn.root.fun)
res1 = remote_fun("aa", X())
res2 = remote_fun("bb", X())
print(res1.value, res2.value)

# Server
import rpyc.utils.server
class MyService(rpyc.Service):
    def exposed_fun(self, id, data):
        print(id)
rpyc.utils.server.ThreadedServer(MyService, port = 12345).start()

Before starting to execute the function, the server sends a synchronous INSPECT request back in order to create a netref proxy for the given class instance during the unboxing of the function arguments. Then it is as you say: the sync request enters another serve loop that catches the second async call request which is then handled in the inner loop and so on.

There is no easy way to prevent this from happening. The server has to handle incoming requests while waiting for the reply for the sync request to avoid dead-locking in case incoming requests are a precondition to complete the sync request. There is no easy way to know which incoming requests have to be handled.

Possible Workaround 1: We could more often preserve the call order by patching rpyc to unbox the function arguments asynchronously and schedule the method call itself as a callback of the unboxing. However, this could have performance implications.

Anyway, async does not guarantee execution order and as soon as you so much as look at the netref proxy all bets are off. So I'm not sure this is worth the effort...

Possible workaround 2: (without modifying rpyc code) Publish all involved classes to the server before executing the async requests:

conn = rpyc.connect("localhost", 12345)

# Cache X on the server side:
conn.ping(X())

# Now proceed as before
remote_fun = rpyc.async(conn.root.fun)
res1 = remote_fun("aa", X())
res2 = remote_fun("bb", X())
print(res1.value, res2.value)

However, neither of these workarounds is sufficient to (a) guarantee given call order and (b) especially not if using the netref proxy in any way during the async method call.

Possible Workaround 3: As it stands, I think the only possible fix is to make sure that the server function is not called with a netref proxy but rather with a true local object as you suggested, i.e. along the lines of:

from rpyc.utils.classic import deliver
res1 = remote_fun("aa", deliver(X()))
res2 = remote_fun("bb", deliver(X()))
@coldfix

This comment has been minimized.

Collaborator

coldfix commented Jul 25, 2017

Thinking about it, by adding a background reactor thread (#84) it should be possible to achieve the expected execution order. I might make a pass at it in a few weeks.

coldfix added a commit that referenced this issue Jul 26, 2017

coldfix added a commit that referenced this issue Jul 26, 2017

Release rpyc 3.4.3
- Add missing endpoints config in ThreadPoolServer (#222)
- Fix jython support (#156,#171)
- Improve documentation (#158,#185,#189,#198 and more)
@power-zhy

This comment has been minimized.

power-zhy commented Aug 7, 2017

Thanks for the detail reply!
Using deliver is a genius idea :)

@coldfix

This comment has been minimized.

Collaborator

coldfix commented May 20, 2018

Closing this for now. To summarize:

  • async doesn't guarantee execution order. If that is required, you will either need synchronous calls, or your own reordering mechanism on the server.
  • deliver will probably work most of the time
  • background reactor thread is on ice for now, instead consider gevent which might more often result in the original execution order (but definitely no guarantees)

@coldfix coldfix closed this May 20, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment