Create a pandoc-manubot-cite filter for pandoc #99

dhimmel · 2019-03-08T23:37:53Z

Create a pandoc filter providing manubot's cite-by-ID functionality.

Todo:

logging exception to nonzero exit status (off by default)
caching (off by default)
export references.json to file in addition to pandoc metadata?

dhimmel · 2019-03-11T19:47:46Z

As of 87d1e3c, I've implemented a proof of concept pandoc filter that extracts citationIds from pandoc's AST (abstract syntax tree) and replaces them with manubot citation_ids. The primary remaining step would be to generate CSL and store it in the appropriate pandoc metadata fields.

Since it seems feasible to create this filter, I thought I'd open the idea to discussion of its merits. The main benefits are:

the filter could massively increase the userbase of manubot's cite-by-id functionality.
the filter would use pandoc's parsing of documents to extract citations, helping fix bugs like how currently citations in code elements are modified (see Do not extract or replace citation strings inside code or code_blocks #13).

The downsides are:

Increased support burden. More research is needed how the plugin will interact with common pandoc commands, especially those using bibtex citations.
Currently, we use a less stringent method for matching citations than pandoc:

manubot/manubot/cite/util.py

Lines 18 to 35 in 81394c0

    
           """ 
        
           Regex to extract citations. 
        
           Same rules as pandoc, except more permissive in the following ways: 
        
           1. the final character can be a slash because many URLs end in a slash. 
        
           2. underscores are allowed in internal characters because URLs, DOIs, and 
        
              citation tags often contain underscores. 
        
           If a citation string does not match this regex, it can be substituted for a 
        
           tag that does, as defined in citation-tags.tsv. 
        
           https://github.com/greenelab/manubot-rootstock/issues/2#issuecomment-312153192 
        
           Prototyped at https://regex101.com/r/s3Asz3/2 
        
           """ 
        
           citation_pattern = re.compile( 
        
               r'(?<!\w)@[a-zA-Z0-9][\w:.#$%&\-+?<>~/]*[a-zA-Z0-9/]')

If we want to switch from using manubot process to using a pandoc filter for citation-by-id for manubot manuscripts, then we will break some existing citation strings. See also discussion at manubot/rootstock#2 (comment), where @tarleb initially suggested using a pandoc filter for this purpose. One aspect will be how often are persistent identifiers invalid pandoc citations due to forbidden characters.

dhimmel · 2019-03-11T20:42:05Z

manubot/pandoc_filter/cite.py

+    """
+    parser = argparse.ArgumentParser(description='Pandoc filter for citation by persistent identifier')
+    parser.add_argument('target_format')
+    parser.add_argument('--pandocversion', help='The pandoc version.')


@tomduck, I copied this --pandocversion argument from pandoc-fignos. Do you know what it's for and whether it's needed? It doesn't seem to me that pandoc 2.5 supplies this argument to filters.

slochower · 2019-03-12T21:47:54Z

manubot/pandoc_filter/cite.py

+  --filter=pandoc-manubot-cite \
+  --filter pandoc-citeproc \
+  manubot/pandoc_filter/tests/input-with-cites.md
+


Is there a difference between --filter= and --filter (with a space)?

Address failure in https://github.com/manubot/manubot/pull/99/checks?check_run_id=382074412#step:8:102

dhimmel · 2020-01-09T20:14:40Z

@agitter would be great for you to take a look at this. Would like to move forward with merging.

agitter · 2020-01-09T20:54:59Z

Okay, I'll try to make time Friday or Saturday.

agitter

Everything looks good to me at a high level. I don't have much knowledge about panflute and its data structures so some of the details of the AST processing are still opaque to me. Many of my questions/comments are seeking to confirm I understand the processing correctly.

README.md

agitter · 2020-01-11T12:44:59Z

README.md

+specifying --filter=pandoc-manubot-cite.
+
+positional arguments:
+  target_format


What are the valid values? It would be helpful even if this has a help message that says "any valid Pandoc format" or something brief.

README.md

agitter · 2020-01-11T13:28:38Z

manubot/pandoc/tests/test_cite_filter/input.md

+---
+# yaml_metadata_block with pandoc metadata
+citekey-aliases:
+  tag:meta-review: url:https://greenelab.github.io/meta-review/


Is this tag:meta-review: syntax temporary to be backwards compatible? Or is this how you envision using tags/aliases in the future?

Backwards compatible syntax that is required until we implement #192. After we implement #192 then you will no longer need to prefix with tag:. But for now it's needed.

manubot/pandoc/tests/test_cite_filter/output.txt

manubot/pandoc/tests/test_cite_filter/input.md

agitter · 2020-01-11T13:33:38Z

manubot/pandoc/tests/test_cite_filter/input.md

+
+Citing pre-defined citekey aliases [@tag:meta-review; @tag:deep-review].
+
+Citing @raw:dongbo-conversation whose metadata is already in pandoc's reference metadata.


Do we see this being used? Is the idea that a valid Pandoc filter can't ignore valid reference metadata if someone were to use it outside the typical Manbot workflow?

The idea is that we should support the standard pandoc methods for defining reference metadata in addition to any manubot magic. That means we need to support metadata.bibliography (paths to bibliography files) and metadata.references (CSL JSON style reference metadata), since these are the standard pandoc ways to specify reference metadata.

agitter · 2020-01-11T13:37:40Z

manubot/process/tests/test_process_command.py

@@ -27,14 +30,18 @@ def test_example_manuscript(manuscript):
        "--output-directory",
        str(manuscript_dir.joinpath("output")),
    ]
+    if skip_citations:
+        args.append("--skip-citations")


Do you want to test that skip citations actually skips the citation processing?

Not really because I view manubot process as a command we're phasing out and test takes time to write. Relying on manubot/rootstock#300 to test --skip-citations actually works.

--skip-citations is working in https://github.com/manubot/rootstock/runs/387673500 / manubot/rootstock@49dcfcd which relies on citation-tags.tsv.

Co-Authored-By: Anthony Gitter <agitter@users.noreply.github.com>

dhimmel · 2020-01-13T22:16:29Z

be5bc4e is working. see https://travis-ci.com/manubot/rootstock/builds/144231641#L855

dhimmel · 2020-01-14T14:39:39Z

@agitter I'm going to merge and address additional feedback in subsequent PRs.

agitter · 2020-01-14T14:54:52Z

I'm going to merge and address additional feedback in subsequent PRs.

Sounds good to me. I'll look over your responses and new commits here later in the week.

dhimmel added 2 commits March 8, 2019 18:35

DRAFT: add in-progress pandoc filter

bce6da7

Filter: extract & replace citationIds

87d1e3c

dhimmel force-pushed the pandoc-filter branch from 20e9297 to 87d1e3c Compare March 11, 2019 19:29

Enable pandoc-manubot-cite CLI filter

bd177e3

dhimmel commented Mar 11, 2019

View reviewed changes

dhimmel added 5 commits March 11, 2019 16:58

In progress CSL metadata support

979e1fd

Update process.util for additional modularity

c5dfe2a

Fix tests

8c46652

pandoc-citeproc load_bibliography function

093e3b9

Slightly improved docs

68b6814

slochower reviewed Mar 12, 2019

View reviewed changes

dhimmel added 2 commits March 13, 2019 15:03

Set standard_citation for csl_items

d5d80d6

Move csl_item_set_standard_citation to global workflow

fa4198f

dhimmel force-pushed the pandoc-filter branch from cb8bce6 to fa4198f Compare March 13, 2019 22:31

dhimmel mentioned this pull request Mar 15, 2019

Expand flexibility of manual citations #100

Closed

Reformat bibliography.json

b09d6c4

dhimmel force-pushed the pandoc-filter branch from 5431650 to b09d6c4 Compare March 15, 2019 20:06

dhimmel added 3 commits March 15, 2019 17:18

Rename submodule: pandoc_filter to pandoc

b5524e8

Fix test failure

8c9bd65

Split bib2json to bibliography module

0f8c510

dhimmel mentioned this pull request Mar 23, 2019

build: common intermediate JSON for all pandoc outputs manubot/rootstock#196

Closed

dhimmel added 2 commits March 26, 2019 14:07

load_bibliography docstring

515799b

fixup! Split bib2json to bibliography module

b2c505a

dhimmel force-pushed the pandoc-filter branch from c196751 to b2c505a Compare March 26, 2019 18:18

process.load_bibliography

c14bcab

dhimmel mentioned this pull request Mar 27, 2019

Future Manubot enhancements (address review 2-4) greenelab/meta-review#179

Merged

dhimmel added 3 commits March 27, 2019 13:26

Enable multiple manual-reference files for manubot process

5da0bdc

Add nbib bibliography test

ad9f9db

Specify encoding for subprocess.run of pandoc-citeproc

1c365d2

dhimmel force-pushed the pandoc-filter branch from d6a8475 to d056f81 Compare January 7, 2020 16:01

Support metadata.manubot variables including fail-on-errors

36b073b

dhimmel force-pushed the pandoc-filter branch from b3169f8 to 36b073b Compare January 7, 2020 20:20

dhimmel added 3 commits January 7, 2020 15:23

set log-level to INFO for test

7fe6498

Merge branch 'master' into pandoc-filter

b7061ea

make requests-cache-path directory if needed

1da787a

dhimmel force-pushed the pandoc-filter branch 2 times, most recently from c818c12 to 6c02ac7 Compare January 9, 2020 18:23

Support manubot.output-bibliography path

5ac5c85

dhimmel force-pushed the pandoc-filter branch from 6c02ac7 to 5ac5c85 Compare January 9, 2020 19:37

pytest.mark.pandoc_version_sensitive

c79274a

Address failure in https://github.com/manubot/manubot/pull/99/checks?check_run_id=382074412#step:8:102

agitter requested changes Jan 11, 2020

View reviewed changes

dhimmel and others added 8 commits January 13, 2020 14:01

process: support citation-tags.tsv with --skip-citations

36eeed1

Apply suggestions from code review

2e68295

Co-Authored-By: Anthony Gitter <agitter@users.noreply.github.com>

fixup! process: support citation-tags.tsv with --skip-citations

4bdbb49

Improve documentaiton & revisions based on review.

bc7d280

--skip-citations, support manual-references*.* glob

230fdd0

set pandoc metadata variables for pandoc-manubot-cite

619c35f

warn if --skip-citations and citation-tags.tsv

be5bc4e

fixup! warn if --skip-citations and citation-tags.tsv

e20a3d8

manubot-output-citekeys

0aff258

dhimmel requested a review from agitter January 13, 2020 22:37

dhimmel merged commit 5ac49d7 into manubot:master Jan 14, 2020

dhimmel deleted the pandoc-filter branch January 14, 2020 14:47

dhimmel mentioned this pull request Jan 14, 2020

Follow up tweaks to pandoc-manubot-cite #194

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a pandoc-manubot-cite filter for pandoc #99

Create a pandoc-manubot-cite filter for pandoc #99

dhimmel commented Mar 8, 2019 •

edited

Loading

dhimmel commented Mar 11, 2019 •

edited

Loading

dhimmel Mar 11, 2019

slochower Mar 12, 2019

dhimmel Mar 13, 2019

dhimmel commented Jan 9, 2020

agitter commented Jan 9, 2020

agitter left a comment

agitter Jan 11, 2020

agitter Jan 11, 2020

dhimmel Jan 13, 2020

agitter Jan 11, 2020

dhimmel Jan 13, 2020

agitter Jan 11, 2020

dhimmel Jan 13, 2020

dhimmel Jan 13, 2020

dhimmel commented Jan 13, 2020

dhimmel commented Jan 14, 2020

agitter commented Jan 14, 2020


		Citing pre-defined citekey aliases [@tag:meta-review; @tag:deep-review].

		Citing @raw:dongbo-conversation whose metadata is already in pandoc's reference metadata.

Create a pandoc-manubot-cite filter for pandoc #99

Create a pandoc-manubot-cite filter for pandoc #99

Conversation

dhimmel commented Mar 8, 2019 • edited Loading

dhimmel commented Mar 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhimmel commented Jan 9, 2020

agitter commented Jan 9, 2020

agitter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhimmel commented Jan 13, 2020

dhimmel commented Jan 14, 2020

agitter commented Jan 14, 2020

dhimmel commented Mar 8, 2019 •

edited

Loading

dhimmel commented Mar 11, 2019 •

edited

Loading