You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"I think the right solution is to look through the IR for the function when it's serialized, and send over all its dependencies. This could also potentially let us remove the localize_vars hack. We can also keep track of which items from Main we've sent to which processors and avoid re-sending, which would be a helpful optimization."
This will help in removing a major source of confusion in serialized closures which capture global variables vs local ones.
The text was updated successfully, but these errors were encountered:
We can also keep track of which items from Main we've sent to which processors and avoid re-sending
Some thoughts/questions:
Doing this should do away with the need for @everywhere for function definitions, since they will be automatically sent from the master.
Limit the automatic sending to calls from the master only? Worker-worker calls will assume that the definitions have been defined on the target process?
Should modules also be automatically loaded on the target worker if referenced during a deserialization? This would require deserialization to read the complete current message (possible as we now have a message boundary), pause the deserialization, load the missing module over the same stream and then continue the deserialization from the temporary buffer. Will do away with the need for @everywhere using - required modules are loaded on demand.
Copying @JeffBezanson 's comment from another issue - #15451 (comment)
"I think the right solution is to look through the IR for the function when it's serialized, and send over all its dependencies. This could also potentially let us remove the localize_vars hack. We can also keep track of which items from Main we've sent to which processors and avoid re-sending, which would be a helpful optimization."
This will help in removing a major source of confusion in serialized closures which capture global variables vs local ones.
The text was updated successfully, but these errors were encountered: