WIP: Parallel docs build

[mdboom]

This is a follow-on to #5589, that enables parallel doc building for all extensions. This does seem to result in the docs actually being built in parallel -- though only using 186% of CPU time on average on a Core i7. However, on my machine, it hangs indefinitely with the message "waiting for workers..." at the end, so it's still not a viable solution. However, I'm putting this up here in case anyone else has any ideas.

Cc: @jschuller, @jenshnielsen, @birkenfeld

[QuLogic]

It looks like the same metadata fix needs to be applied to doc/sphinxext/github.py if possible.

[birkenfeld]

Might look into this when I have time.

also cc @shimizukawa

[mdboom]

Thanks, @birkenfeld. matplotlib's docs are enormous (though build time is mostly due to plotting, not text), but any additional performance we can get out of Sphinx is much appreciated.

[mdboom]

Also, @birkenfeld: Thanks for making Sphinx!

[birkenfeld]

I merged a pull request that rewrites parallel handling to be less complex yesterday (into both master and stable branches). Can you try again?

[mdboom]

@birkenfeld: Indeed that helps considerably, and it's nice to have the feedback about which extensions are preventing parallel reading.

With the changes in this PR, plus manually hacking the IPython and Numpy sphinx extensions we use (which I'll create PRs for as well), I was able to get the doc build time down from 8:58 to 4:27 on a Core i7 quad-core. Not bad at all.

[mdboom]

See ipython/ipython#9098. Numpydoc already has the necessary changes in master.

[jenshnielsen]

It's probably a good idea to watch out for IPython 4.1 beta announcements and make sure that is backported if needed/possible. I will try to remember that. There was some talk about that on the ipython mailing list before christmas

[QuLogic]

IPython 4.1 is out now, right?

[jenshnielsen]

I have tried it (on https://github.com/jenshnielsen/matplotlib/tree/parallel-docs2) Unfortunatly it suffers from some of the same issues as running the tests in parallel and often fails

[tacaswell]

This seems to be working for me (with numpydoc master). Looks like it will itermittently saturate 8 cores. It gets a full rebuild down to 2:24 and a small-change rebuild to under a few seconds. I'll open a new PR sometime tonight.

[jenshnielsen]

Last time I tried this it resulted in lots if intermittent build failures on Travis so I never opened a new pr

[QuLogic]

This was applied with #6993.