Try at parallel gallery generation

[jschueller]

A naive attempt at #25:

• Does not work
• Only attempts to parallelize generation one directory at a time
• Hard-wired number of processes
• It's something

[larsoner]

Thinking about this a bit more, I'd expect this not to work for (at least) the matplotlib, mayavi, and pyvista scrapers because these are all global-state based. And then there will be tricky interactions with reset_modules, which also by default does global state stuff at least for matplotlib. So I'm not sure this will ever work at least for the majority of our users :(

[larsoner]

I'll copy that over to the issue so more people can tell me how I'm wrong :)

[jschueller]

maybe the sphinx folks (@samdoran or @tk0miya) could give hints, or sklearn (@Titan-C @GaelVaroquaux) would be interested in speeding up their builds too ?

[tk0miya]

It seems this handler is invoked on the bootstrap process of Sphinx (on builder-inited event). So there is no special support from Sphinx framework.

[jschueller]

indeed, we're looking at parallelizing the gallery jobs at the extension level