Implements all-pairs shortest paths in parallel

[jfinkels]

This commit modifies the implementation of the unweighted all-pairs
shortest paths to use the multiprocessing module.

This pull request is intended to experiment with employing parallelism directly within NetworkX. See issue #356.

[jfinkels]

The tests fail because I used the multiprocessing.Pool class as a context manager, but Pool did not become a context manager until Python 3.3. PyPy3 also fails because on Travis it only supports Python 3.2. I'm sure it's no problem to adapt this to work without the context manager, but I'd like to get feedback on this approach before continuing work on this branch.

[progval]

It would also be nice to have it work without multiprocessing, as it does not work on all platforms, like Jython, or Linux distributions without the right permissions on /dev/shm.

[jfinkels]

The Jython website seems to indicate that there is a Jython 2.7.0 final release that supports Python 2.7, which should include multiprocessing, is that correct?

Does the implementation of multiprocessing always use the shared memory at /dev/shm on Linux systems?

[progval]

The Jython website seems to indicate that there is a Jython 2.7.0 final release that supports Python 2.7, which should include multiprocessing, is that correct?

I don't think so. Jython provides “real” threads in the threading module, so multiprocessing is irrelevant.

Does the implementation of multiprocessing always use the shared memory at /dev/shm on Linux systems?

I don't know, but I have received several bug reports related to this for one of my projects.

[jfinkels]

I don't think so. Jython provides “real” threads in the threading module, so multiprocessing is irrelevant.

I wouldn't say it's irrelevant, I'm sure there are situations in which I might want to have multiple processes performing some work instead of multiple threads. Also, for the sake of maintainability and portability, if I want to write a program that runs on both CPython and Jython (and whatever other Python implementation you like), I might want to just write my program so that it uses multiprocessing because the API is simple, assuming each Python implementation meets the same API (as is the case with me writing this code for NetworkX right now :D).

I don't know, but I have received several bug reports related to this for one of my projects.

Hm, interesting. That's valuable feedback, thanks for forwarding it.

[hagberg]

Maybe this could be an example? So far we are not committing to any particular parallel programming model for networkx.

[MridulS]

To be discussed further in #4064 and #6306