put(RemoteRef(), obj) semantics are troublesome

[timholy]

I expected put to work like push!, and so had code like this:

getnextimg = put(RemoteRef(), frameindex -> img[:,:,frameindex])

However, unlike push!, this doesn't return the container (the RemoteRef), it returns the object. Since we can't currently pass anonymous functions between processes, this caused some hard-to-decipher hangs and error messages.

Is it conceivable to change it to put! and have it return the RemoteRef?

[JeffBezanson]

I assume the problem with anonymous functions is #5775? (since we can pass anonymous functions between processes that have the same top-level environment)

[JeffBezanson]

Also, I don't fully understand the problem. Is the return value of put getting sent somewhere? This shouldn't be necessary, since communication can be done via the RemoteRef that put is writing to, or else via a remotecall.

[timholy]

I assume the problem with anonymous functions is #5775?

Yes indeed, sorry I didn't spell that out.

Is the return value of put getting sent somewhere?

Yes, I was including the remoteref as an argument to a function. To work around #5775, my current strategy (which doesn't require any modifications to Julia base) is to define

eval(Main, :(@everywhere resolvefunction(r::RemoteRef, args...) = fetch(r)(args...) ))
eval(Main, :(@everywhere resolvefunction(ex::Union(Expr,Symbol), args...) = eval(ex)(args...) ))

and then I can call functions in "foreign" modules in other processes as

remotecall(pid, Main.resolvefunction, rr, args...)
remotecall(pid, Main.resolvefunction, :(SomeModule.thefunctionIreallymeantocall), args...)

where rr is a "packed" anonymous function.

[timholy]

Is the return value of put getting sent somewhere? This shouldn't be necessary, since communication can be done via the RemoteRef that put is writing to, or else via a remotecall.

Reading this comment over again, let me make sure I'm being clear:

val = put(RemoteRef(), obj)

returns obj. So the sequence is:

• A "blank" RemoteRef gets created
• obj gets written to the RemoteRef
• put returns obj and assigns it to val. This means there are no references to the RemoteRef, and it disappears (eventually to be gced).

I assumed that val was the RemoteRef, but instead it turned out I was trying to pass obj (which failed catastrophically but without a very helpful error message).

[JeffBezanson]

Ah, ok. Changing it would probably be fine; the return value is not even documented.

[timholy]

Great. And how do you feel about changing put to put!?

[simonster]

Re: naming, see also #5511

[timholy]

@simonster, thanks for the link (I just haven't paid enough attention to everything going on...)

To avoid a troublesome merge for either Amit or myself, I'll wait until there's a verdict on #5757 before renaming put to put! and fixing the return type (assuming it still needs fixing at that point).

[timholy]

Thanks, Jeff. I didn't want to mess Amit up, but I agree this was good to fix. Sorry this fell on you.