Example for using v2.X with Read the Docs

[matthewfeickert]

Hi. Thanks for making sphinxcontrib-bibtex! My team and I are working on updating to the v2.0.0 release but as with v2.0 Sphinx must be run twice to generate the bibliography:

It is good to be aware that the extension stores all citation information in a bibtex.json file in the source folder. If it does not exist, the file will be created on your first sphinx build, and you will have to rerun the build to make use of it. The file is automatically kept up to date, with a warning whenever you need to rerun the build (i.e. whenever your citations change). The file can be stored in version control if you do not want your users to have to run sphinx twice when they clone your project, or when you need your documentation to be built on an external service which only runs sphinx once.

It isn't exactly clear how one would accomplish this for a testing system like Read The Docs, which we use to test the rendering of our docs for each PR. It would be great if possible if an example could be added to the documentation that highlighted possible solutions to this.

[choldgraf]

Maybe a more general approach would be document if / how it is possible to generate the cache and use it in a single Sphinx pass. Requiring to manually run Sphinx twice is also going to be a problem for Jupyter Book as well :-/

e.g., could you somehow init the cache manually as a part of your conf.py file via this function? https://sphinxcontrib-bibtex.readthedocs.io/en/latest/api/interface.html#sphinxcontrib.bibtex.init_bibtex_cache

[mcmtroffaes]

That's exactly what the extension already does. Quoting from the documentation:

The file [bibtex.json] can be stored in version control if you do not want your users to have to run sphinx twice when they clone your project, or when you need your documentation to be built on an external service which only runs sphinx once.

Does this help?

[mcmtroffaes]

If you need examples, there's plenty in the regression tests. Most tests store the json in version control, so they only need to run sphinx once. There's a few tests where that's not the case if they explicitly need to test the json generation. You can find them here:

https://github.com/mcmtroffaes/sphinxcontrib-bibtex/tree/develop/test/roots/

[choldgraf]

Sorry I should have added "without committing the auto-generated file" - as a general strategy we are trying not to require users to package build assets with their repositories and only include user-edited files. I don't think it is a deal breaker, but I'm just wondering if there's a way to do it

[mcmtroffaes]

No, not at the moment unfortunately. I understand it's not ideal to have a generated file in version control. That's the price for wanting to run sphinx just once: the extension needs this information to resolve references.

I take from this issue that the documentation may not be 100% clear. If anyone sees a better way of phrasing it, I'm open to suggestions.

[matthewfeickert]

Sorry I should have added "without committing the auto-generated file" - as a general strategy we are trying not to require users to package build assets with their repositories and only include user-edited files.

Yeah, same. Having to maintain the bib files and then ensure in CI that they produce a bibtex.json the same as the one in version control is ideally something that we'd like to avoid having to do and have contributors have to understand and check.

This was my motivation for asking about examples of telling RTD to run Sphinx twice. I'm not sure if this is possible, but if it isn't then I think it would be very helpful to make a large notice in the docs that if you want to use RTD then you must version bibtex.json.

(Thanks for the very fast responses btw!)

[chrisjsewell]

Eurgh, I honestly think this is going to be a big deal breaker for a lot of people looking to use this package.
I'm sure there is method in the madness though lol. Could you explain briefly why this bibtex.json is now required (I assume to do with parallel builds), and why there is no way to achieve this in a single run of sphinx?

[mcmtroffaes]

Thanks for all the feedback, everyone. I deeply appreciate that! I realize 2.0.0 is a bit of a double edged sword: one the one hand it fixes some very long standing issues (unresolved citations, parallel builds), but on the other hand it now needs to either run sphinx twice or keep a generated file in version control.

The reason the extra file is required is to ensure that the bibliography directives can generate the correct citation nodes even if the citing document has not yet been processed, as sphinx processes documents out of order.

There might be other ways to achieve this for instance by postponing citation nodes to be generated to a much later stage when all documents have already been read and processed. However to do that, the extension must do all the heavy lifting to resolve all references appropriately, basically duplicating what sphinx already does for regular citations. I currently don't understand enough of sphinx to code this with confidence, and so far, I've preferred not to duplicate anything that sphinx can already can do for me. So I've leveraged the existing sphinx citation system rather than built my own one from scratch and basically reinvent the wheel.

You can find a bit more discussion about this from the sphinx devs at this thread sphinx-doc/sphinx#6393 (comment) although I'm not sure how that specific suggestion would work with multiple bibliography directives, besides being potentially quite inefficient in case of large bib files with only few citations. Maybe something else with pending_xref or SphinxPostTransform could work too.

For now, an alternative is to use the new footcite and footbibliography system if you're ok with references being part of the footnotes, and if you're ok to have your reference details duplicated across documents. The json is only used for regular citations. If you don't use regular citations a single build will work fine.

[chrisjsewell]

thanks @mcmtroffaes, yeh I'll try to have a think about it a some point. We've pinned jupyter-book to v1 for now, and I feel we will be motivated to help come up with a simpler approach, otherwise we will have more people complaining than already are lol (jupyter-book/jupyter-book#1137)

[mcmtroffaes]

Sure, and this is a major update for a reason. Pinning to sphinxcontrib-bibtex<2.0.0 makes sense. I hope I can eventually make some updates to make the new system work without needing to use a generated file, but it may take some time.

[chrisjsewell]

Oh absolutely, thanks for putting in the time for the improvements 🙏

[RobertoBagnara]

I thought putting bibtex.json in the source directory was a decent workaround, but it is not in my case. I use the same sources to build multiple configurations and each new build results in an unwanted update of the bibtex.json (which was build using the maximal configuration, i.e., the one with all citations included). So I am glad an alternative is materializing.

[mcmtroffaes]

Glad to hear it's appreciated! It's just in develop, will be releasing later today as soon as I have the documentation and release notes finished.

[choldgraf]

@mcmtroffaes I have been enjoying watching your code marathon these past few days :-)

[mcmtroffaes]

The post-transform branch is now included with the 2.1.0 release, so closing as sphinx no longer needs to run twice.