Async requests in reversed order

[aviralg]

I am making multiple async requests in a loop. What I observe is that these requests are serviced in batches and in a particular batch all the requests are served in reverse order.

Here is a code snippet -

def _run_simulation(self, callback):
    async_callback = rpyc.async(callback)
    x = 0
    while x < 1000:
        async_callback({ "runtime" : time.time()})
        time.sleep(0.5)
        x += 1

Is it possible to ensure that the requests are somehow serviced in correct order ?
I am aware of the weakref issue and holding the async_callback reference as well.

[power-zhy]

I hit the same problem too, and I found it only occurs when both async call and by-reference parameter is used, and two async request should be send simultaneously. (Here by-reference means rpyc server would call back to fetch data inside the parameter, see _box function in protocol.py for more detail)
One temporary solution which I used is, use pickle module to pack the parameter to a string so that it would be transferred by-value instead of by-reference.

I wonder if it can be easily solved in rpyc, the problem is, when server trying to send GETATTR call back to client to fetch specific data in parameters, it launch a sync call and start waiting, then it receives another async request instead of the sync response, which gives the second async request a higher order.
When the server side trying to send GETATTR calls, can it use async as well if the original call from client to server is async?

[coldfix]

Can you give a full example?

[power-zhy]

Sorry for being late, here is one simple example.
Server:

import rpyc
from rpyc.utils.server import ThreadedServer

class MyService(rpyc.Service):
	def exposed_fun(self, id, data):
		print("Exposed fun called with [{}].".format(id))
		return data.x() + data.y()

if __name__ == "__main__":
	server = ThreadedServer(MyService, port = 12345)
	server.start()

Client:

import rpyc

class MyStruct(object):
	def __init__(self, id, x, y):
		self.id = id
		self.x = x
		self.y = y
	
	def exposed_x(self):
		print("Getting x from [{}], value is {}.".format(self.id, self.x))
		return self.x
	
	def exposed_y(self):
		print("Getting y from [{}], value is {}.".format(self.id, self.y))
		return self.y


conn = rpyc.connect("localhost", 12345)
remote_fun = rpyc.async(conn.root.fun)
res1 = remote_fun("aa", MyStruct("aa", 1,2))
res2 = remote_fun("bb", MyStruct("bb", 3,4))
print(res1.value, res2.value)
input()

After running, the results are:
Server:

Exposed fun called with [bb].
Exposed fun called with [aa].

Client:

Getting x from [bb], value is 3.
Getting y from [bb], value is 4.
Getting x from [aa], value is 1.
Getting y from [aa], value is 2.
3 7

In client side we call remote function with id 'aa' first, but server got 'bb' first, which is reversed.

[coldfix]

Hi,

thanks a lot. I agree with your analysis, with the additional note that this phenomenon occurs already without even accessing the netref:

# Client
import rpyc
class X(object):
    pass
conn = rpyc.connect("localhost", 12345)
remote_fun = rpyc.async(conn.root.fun)
res1 = remote_fun("aa", X())
res2 = remote_fun("bb", X())
print(res1.value, res2.value)

# Server
import rpyc.utils.server
class MyService(rpyc.Service):
    def exposed_fun(self, id, data):
        print(id)
rpyc.utils.server.ThreadedServer(MyService, port = 12345).start()

Before starting to execute the function, the server sends a synchronous INSPECT request back in order to create a netref proxy for the given class instance during the unboxing of the function arguments. Then it is as you say: the sync request enters another serve loop that catches the second async call request which is then handled in the inner loop and so on.

There is no easy way to prevent this from happening. The server has to handle incoming requests while waiting for the reply for the sync request to avoid dead-locking in case incoming requests are a precondition to complete the sync request. There is no easy way to know which incoming requests have to be handled.

Possible Workaround 1: We could more often preserve the call order by patching rpyc to unbox the function arguments asynchronously and schedule the method call itself as a callback of the unboxing. However, this could have performance implications.

Anyway, async does not guarantee execution order and as soon as you so much as look at the netref proxy all bets are off. So I'm not sure this is worth the effort...

Possible workaround 2: (without modifying rpyc code) Publish all involved classes to the server before executing the async requests:

conn = rpyc.connect("localhost", 12345)

# Cache X on the server side:
conn.ping(X())

# Now proceed as before
remote_fun = rpyc.async(conn.root.fun)
res1 = remote_fun("aa", X())
res2 = remote_fun("bb", X())
print(res1.value, res2.value)

However, neither of these workarounds is sufficient to (a) guarantee given call order and (b) especially not if using the netref proxy in any way during the async method call.

Possible Workaround 3: As it stands, I think the only possible fix is to make sure that the server function is not called with a netref proxy but rather with a true local object as you suggested, i.e. along the lines of:

from rpyc.utils.classic import deliver
res1 = remote_fun("aa", deliver(X()))
res2 = remote_fun("bb", deliver(X()))

[coldfix]

Thinking about it, by adding a background reactor thread (#84) it should be possible to achieve the expected execution order. I might make a pass at it in a few weeks.

[power-zhy]

Thanks for the detail reply!
Using deliver is a genius idea :)

[coldfix]

Closing this for now. To summarize:

• async doesn't guarantee execution order. If that is required, you will either need synchronous calls, or your own reordering mechanism on the server.
• deliver will probably work most of the time
• background reactor thread is on ice for now, instead consider gevent which might more often result in the original execution order (but definitely no guarantees)