#3439: use multiprocessing

[jazzmuesli]

A very early draft, please criticise and suggest something better.

[dschult]

This is a potentially important step toward parallel work using NetworkX. Decisions made here can affect a lot of code long term. So it is good to implement for one function and see what issues it brings up. I can imagine this becoming a large support issue if we implement it for many functions. There are so many ways to set up parallel systems: synch vs asynch, one machine vs remote connections, etc.

I wonder if we could start by making this an example. Then people don't expect it to "just work" in the way they want it to. They expect to have to switch it from Pool.map to Pool.starmap and they have the code easily accessible to take it and change as needed. (people are reluctant to play with code from the package in a way they aren't for example code.)

What do you think? Am I just acting like an old fuddy duddy? Is this type of usage easy enough now?

[suuuehgi]

I had a quick test on a 8-core machine and there seems to be a big overhead.

I need at least a graph with a couple of thousand nodes and edges to be on par with networkx.all_pairs_shortest_path_length. Below it's considerably slower, above it get's more "efficient".

On a MultiDiGraph with 4201 nodes and 9683 edges it's just two times as fast.

Can this be improved?

[MridulS]

Thanks for the PR @jazzmuesli! We should be using #6306 and #4064 for further discussions about this :)