Manubot cite: ability to output formatted bibliography

[dhimmel]

Suggested by @slochower in #42 (comment):

A super handy addition would be an option to output a formatted citation using the built-in style.csl. One challenge is that we currently leverage Pandoc to tie together the JSON for the references and the CSL. I suppose we could make a wrapper that writes the JSON to a file, writes a temporary Markdown file with the citation ids, calls Pandoc to render the temporary file, and then prints the rendered Markdown to stdout. That may be too clumsy to implement, though.

Brilliant! I agree this would be super useful, for cases ranging prototyping to just wanted to generate a reference.

One option would be to use citeproc-py. I am also guessing there is a pandoc / pandoc-citeproc solution that is more direct than having to mess with creating a source markdown file. However, that would create a non-python dependency, which may be difficult for many casual users.

[slochower]

One option would be to use citeproc-py.

Yes, maybe! It sounds better than making temporary files, but we ought to check out the non-compatible things and see if they're relevant.

I am also guessing there is a pandoc / pandoc-citeproc solution...

Not sure about that, but maybe. A two minute search didn't come up with anything relevant.

[dhimmel]

Another option would be to use the JavaScript parser and call it from python using something like https://github.com/stefano/pyduktape.

[dhimmel]

Paging the pros @rmzelle @jgm @Juris-M @brechtm: We are looking to convert CSL JSON Data to a formatted bibliography from a Python program. Basically we're creating a command-line interface where a user would type:

manubot cite \
  --csl=https://github.com/greenelab/manubot-rootstock/raw/master/build/assets/style.csl \
  doi:10.1098/rsif.2017.0387 pmid:29424689 pmcid:PMC5640425 arxiv:1806.05726

Then the program would output something like:

1. Opportunities and obstacles for deep learning in biology and medicine
Travers Ching, Daniel S. Himmelstein, Brett K. Beaulieu-Jones, Alexandr A. Kalinin, Brian T. Do, Gregory P. Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M. Hoffman, … Casey S. Greene
Journal of The Royal Society Interface (2018-04) https://doi.org/gddkhn

2. Sci-Hub provides access to nearly all scholarly literature.
Daniel S Himmelstein, Ariel Rodriguez Romero, Jacob G Levernier, Thomas Anthony Munro, Stephen Reid McLaughlin, Bastian Greshake Tzovaras, Casey S Greene
eLife (2018-03-01) https://www.ncbi.nlm.nih.gov/pubmed/29424689

3. Systematic integration of biomedical knowledge prioritizes drugs for repurposing
Daniel Scott Himmelstein, Antoine Lizee, Christine Hessler, Leo Brueggeman, Sabrina L Chen, Dexter Hadley, Ari Green, Pouya Khankhanian, Sergio E Baranzini
eLife (2017) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5640425/

4. Generalization of the Fermi Pseudopotential
Trang T. Le, Zach Osman, D. K. Watson, Martin Dunn, B. A. McKinney
arXiv (2018-06-14) https://arxiv.org/abs/1806.05726v1

Any idea what the best solution is that can combine CSL JSON Data + CSL XML Style and output a bibliography? Note we don't actually care about rendering the citations in the text, just the reference list.

[jgm]

If you wanted to use pandoc (and pandoc-citeproc) for this, you could. Pandoc will accept a bibliography in CSL JSON format or in bibtex format (which you might more easily get from online services) A pandoc markdown document containing just # --- nocite: '@*' bibliography: yourbib.bib csl: yourcsl.csl ... will generate citations for everything in the bibliography. (Note that you need to use `--filter pandoc-citeproc` on the command line.) With pandoc you can get the output in many different formats, including 'plain'. Your only tasks would be: (1) generating the bibliography from the arguments to your cli program, and (2) stripping out the bibliography part of pandoc's output (since you don't want the citations themselves). This can be done easily using a pandoc filter or lua-filter.

[brechtm]

For a Python project, citeproc-py would indeed be the obvious choice, of course. It should handle your use case just fine. There is no documentation, but you can basically copy the code from the examples.

Note that citeproc-py does depend on libxml, if you are concerned about non-pure-Python dependencies. Removing that dependency (citeproc-py/citeproc-py#35) has been suggested, but unlikely to materialize any time soon since I don't have time to work on citeproc-py and no one has stepped up as a maintainer. This also means that development has stagnated. Considering that, it may be wiser to choose pandoc. But since your implementation will just be a few lines of code, there's little risk involved.

[dhimmel]

@jgm and @brechtm thanks for the guidance. We ended up using Pandoc rather than citeproc-py, because of:

• the more active maintenance
• outputting to many formats (JATS, HTML, plain text, markdown)
• for consistency with our use of pandoc elsewhere in the Manubot system