Multiprocessing semantics in parallel.view.map

[hgrecco]

I have a script that is executed using either

• the builtin map
• using multiprocessing map
• IPython view.map.

This is very convenient because I can easily debug and test scripts, as well as run the same script unchanged in different installations.

However, the semantics of map is different. While in multiprocessing a generator is accepted, in IPython is not (fails when it tries to take the length of the generator). Insider multiprocessing, the following code converts a generator to a list:

if not hasattr(iterable, '__len__'):
    iterable = list(iterable)

I do not really like this behavior (Explicit is better than implicit), but having the same semantics is convenient.

So, my questions are:
1.- Is it possible to have a map that does not require to know in advance the number of tasks?
2.- If not, what do you think about using the same semantics that multiprocessing?

[minrk]

Matching multiprocessing should be trivial, failing to cast to a list might be tricky, but I will have a look.

[minrk]

closed by #3282