CSL Data: empty date-parts in issued causing pandoc-citeproc to crash

[dhimmel]

Some Crossref DOI records have their issued date-parts set to null, such as 10.22541/au.149693987.70506124. In these situations, DOI Content Negotiation for CSL JSON returns

    "issued": {
      "date-parts": [
        [
          null
        ]
      ]
    },

In the past, before we pruned invalid CSL using the CSL Data JSON Schema, we addressed this case with the following:

https://github.com/greenelab/manubot/blob/693fbb7758b5922add30ecaa6e30acb98426a977/manubot/cite/citeproc.py#L88-L95

Hence, we'd remove issued if the first element of the date-parts list in Python was None. In #49, we switched to using the JSON Schema to remove invalid fields and removed custom CSL fixing logic. Our hope was that CSL that passed the JSON Schema would be compliant with pandoc-citeproc.

The JSON Schema currently specifies elements in date-parts arrays must be strings or number (excludes null), but does not specify minItems of 1. Hence, our CSL Data pruning removes the null item but keeps the empty list:

    "issued": {
      "date-parts": [
        []
      ]
    },

This causes pandoc-citeproc to crash (as currently happening with manubot cite --render doi:10.22541/au.149693987.70506124 and greenelab/meta-review#101 (comment)):

Error parsing references: Could not parse RefDate
Error running filter pandoc-citeproc:
Filter returned error status 1

This issue was discovered in greenelab/meta-review#101 (comment). The corresponding WIP PR to fix it is #65 (pending a solution).

[dhimmel]

Solutions

This issue touches on many projects, and there are many possible solutions that should be explored concurrently:

1. We (Manubot) should consider checking for empty date-parts arrays and removing their parent object. CC @agitter
2. pandoc-citeproc should consider accepting empty date-parts and treating it the same as if the field were not supplied at all. CC @jgm
3. the JSON Schema for CSL Data should consider setting minItems: 1 for the date-parts array. CC @rmzelle.
4. @CrossRef should consider not returning an issued object when it's missing rather than returning invalid CSL. Refs Invalid types in CSL JSON items returned by DOI content negotiation CrossRef/rest-api-doc#187. CC @kjw @gbilder
5. Authorea should set their @CrossRef issued metadata, see tweet. This is only a partial solution as many publishers probably have this issue.

[agitter]

We (Manubot) should consider checking for empty date-parts arrays and removing their parent object.

This makes sense to me as a temporary workaround. Our short term goal is to be able to upgrade manuscripts to use Manubot v0.2.0.

In the longer term, one or more of the other solutions (especially 2-4) would be helpful so that we don't have to treat this as a special case.

[dhimmel]

We've addressed this issue for Manubot in #65 / 4e6a0f6. @jgm has updated pandoc-citeproc to not crash when encountering empty date-parts (thanks!) in jgm/pandoc-citeproc@bde816c. I opened a PR to update the CSL JSON schema in citation-style-language/schema#158. So overall things are looking good... hopefully Crossref & authorea will implement solutions 4 & 5, but that is not required for Manubot to work.