Suggestion: extract the module list from the Sphinx object inventory

[ncoghlan]

This is a neat project, but rather than scraping the ToC HTML (which offers no backwards compatibility guarantees, and hence may break your page scanning in future versions), you may want to read the full module details from the automatically generated Sphinx object inventory ("objects.inv") instead. That's a standard Sphinx feature that allows intersphinx to link between projects based on object references rather than direct URLs.

The following script provides an example of reading a Sphinx inventory file directly and printing out all the contents: https://gist.github.com/epc/4118456

It may also be worth filing an RFE with the intersphinx project to either break the inventory reading out to an independent module, or at least to provide a public API that encapsulates the task of retrieving and parsing an object inventory independently of Sphinx itself.

/cc @birkenfeld

[jackmaney]

Thank you for the suggestion! I wasn't aware of objects.inv files before. After digging around and figuring out how to get the objects.inv file from the Python docs, I think I know how to proceed. I'll incorporate bits of your script and do an upgrade probably sometime this weekend.

[jackmaney]

• Grab all objects.inv files for the relevant versions of Python (2.6-7, 3.2-4) (eg http://docs.python.org/<version>/objects.inv) and store them in a cache subdirectory within the repo.
  • Use @ncoghlan's script to parse the inventory files and output to HTML, putting the HTML files in a sub-directory of the cache folder
  • Consider auto-updating the cache if any of the files within are more than N days old (N == 60, maybe?).
• Use Beautiful Soup to parse the cached HTML, grab all of the module names, and stuff them into lists for each version.
  • The objects file doesn't contain metadata for the modules (whether or not they're deprecated, etc). Should this be scraped separately, or just scrapped altogether?

[birkenfeld]

The read_inventory functions in sphinx/ext/intersphinx.py are usable standalone. The fetch_inventory function needs an app object with some general features (sourcedir attribute and warn method).

[jackmaney]

I started a new branch to work on this. It's not quite done, yet. I may have to end up scrapping support for 2.6 (since 2.6's inventory file isn't zlibbed, which is what read_inventory_v2 expects....unless that's what read_inventory is for?).

[birkenfeld]

Exactly; the latter decides which version the inventory is and uses _v1 or _v2 accordingly.

[ncoghlan]

A note regarding credit: I didn't write the gist I linked to, @epc did that.

Thank you for considering the change!

[epc]

Feel free to use/abuse the script from that gist. It was my second python script of any consequence so be wary (I know of one minor error which I'll fix shortly). Was used by the @mongodb/docs team but I think they've changed their process since I left @mongodb.

[jackmaney]

Once I saw that I could use a simple dummy object for app, I went ahead and used fetch_inventory. Thank you all for your help!