Large list of dicts transmit inefficiently as netref

[artemtyupov]

I have list of dictionaries 150000 size.
looping through this netref list on client side very slow.
with using multiprocessing.Pool() performance is better, but very slow too:

• without netref time is 25 seconds (with come logic in loop)
• with netref time is 271 seconds (wth same logic in loop)

What can be done to increase speed?

[comrumino]

This is a little tricky because there are a few approaches. The most performant choice would be to transfer the dictionaries as a brine-able type (e.g. json.dumps the whole list to string and send that). The most likely problem is that you have to construct a large number of netrefs and intrinsically send stuff many times. Without knowing your design requirements, it is not easy to propose the best solution for you.

I could see if there is an "easy" win in optimizing the protocol—things that could be cached better. However, having an example to mock types and any nesting of objects you are doing would be helpful.

[artemtyupov]

Thank you for answer.
Im resolve the problem with use copy.deepcopy on client side.
Timing of this - 2 seconds, and then loop over default list of dicts)