Parallel: output of %px and %px${suffix} is inconsistent

[twmr]

%px print a outputs the content of a, whereas %pxroot print a outputs async.execute . The %pxroot magic was created with dv.activate(suffix='root'). The outputs are inconsistent. Is this desired ?

[minrk]

Yes, this is expected. The initial view created for magics has the non-default initial state block=True. If you set dvtmp.block = dvroot.block = True, then they will be blocking as well, or do %pxconfig --no-block, then they will match in the other direction.

[twmr]

What do you mean by initial view? The one associated with the return value of the first invocation of Client() ?

Is it possible to block the Client() function s.t. it blocks until the cluster is available? Until now I have observed 3 different return types/values of the Client function: one where len(c) has the desired length, one where the length is 0 and one where an exception is thrown due to a missing json file.

[minrk]

What do you mean by initial view? The one associated with the return value of the first invocation of Client() ?

When you create a client for the first time, it does this:

view = rc.direct_view(targets='all')
view.block = True
view.activate()

The result is a blocking view always mapping to all engines, which can be a function of time.

Is it possible to block the Client() function s.t. it blocks until the cluster is available?

Sure. To wait for the connection file:

from IPython.utils.path import get_security_file
# 100 * 0.1 = 10s timeout
for i in range(100):
    try:
        cf = get_security_file('ipcontroller-client.json')
    except IOError:
        time.sleep(0.1)
    else:
        break

To wait for a specific number of engines:

while len(rc) < expected_engines:
    time.sleep(0.1)
    rc.spin()

It makes sense for the first case to be builtin to the Client itself (engines already do this). It does not make sense for the second, because a cluster with zero engines is perfectly valid, and there is no mechanism for the client to know how many engines to expect beyond you just telling it, and it is trivial for you to do this waiting yourself. I could add a wait_for_engines(n=1) method, with the two-line loop above. But this certainly would not be called from the Client constructor.

[twmr]

Thx, your get_security_file() code does the trick! I would be +1 for a builtin version.