＠Add pandoc-style citations (#612)

* ＠Add pandoc-style citations
* ＆ Update `&` --> `&` in citation-js-utils
* 📖 Document `@author2023` citation syntax

.changeset/cyan-ligers-hope.md

@@ -0,0 +1,6 @@
+---
+'markdown-it-myst': patch
+'myst-parser': patch
+---
+
+Add column information to citations and roles

.changeset/fresh-rivers-think.md

@@ -0,0 +1,5 @@
+---
+'myst-spec-ext': patch
+---
+
+Add partial to Cite node to allow for year or author only.

.changeset/old-guests-live.md

@@ -0,0 +1,5 @@
+---
+'citation-js-utils': patch
+---
+
+Replace & with & in rendered html

.changeset/purple-apples-happen.md

@@ -0,0 +1,5 @@
+---
+'myst-transforms': patch
+---
+
+Search for unmatched citations and use them as cross references or warn.

.changeset/rare-snakes-sin.md

@@ -0,0 +1,5 @@
+---
+'myst-cli': patch
+---
+
+Do not add unknown citations to the bibliography.

.changeset/slimy-books-argue.md

@@ -0,0 +1,5 @@
+---
+'citation-js-utils': patch
+---
+
+Support partial author or year

docs/citations.md

@@ -4,9 +4,7 @@ description: Add academic citations to your documents easily, have hover-referen
 thumbnail: ./thumbnails/citations.png
 ---
 
-Citations automatically show up in your site, including a references section at the bottom of the page. These citations are able to be clicked on to see more information, like the abstract. There are two different ways to add citations to your documents: (1) adding a markdown link to a [DOI](wiki:Digital_object_identifier); and (2) by adding a bibtex file, which can be exported from any reference manager, and adding a `cite` role to your content.
-
-+++
+Citations automatically show up in your site, including a references section at the bottom of the page. These citations are able to be clicked on to see more information, like the abstract. There are two different ways to add citations to your documents: (1) adding a markdown link to a [DOI](wiki:Digital_object_identifier); and (2) by adding a BibTeX file, which can be exported from any reference manager, and adding a `cite` role to your content.
 
 (doi-links)=
 
@@ -24,13 +22,57 @@ which will insert the citation text in the correct format (e.g. adding an italic
 
 Providing your DOIs as full links has the advantage that on other rendering platforms (e.g. GitHub), your citation will still be shown as a link. If you have many citations, however, this will slow down the build process as the citation information is fetched dynamically.
 
-+++
-
 ## Including BibTeX
 
 A standard way of including references for $\LaTeX$ is using <wiki:BibTeX>, you can include a `*.bib` file or files in the same directory as your content directory for the project. These will provide the reference keys for that project.
 
-To create a citation in Markdown, use either a parenthetical or textual citation:
+If you want to explicitly reference which BibTeX files to use, as well as what order to resolve them in, you can use the `bibliography` field in your frontmatter, which is a string array of local or remote files. This will load the files in order specified.
+
+```yaml
+bibliography:
+  - my_references.bib
+  - https://example.com/my/remote/bibtex.bib
+```
+
+The remote BibTeX can be helpful for working with reference managers that support remote links to your references.
+
+## Markdown Citations
+
+You can add citations to any BibTeX entry using the citation key preceded by an `@`, for example, `@author2023`.
+This syntax follows the [pandoc citation syntax](https://pandoc.org/MANUAL.html#citation-syntax). Multiple citations can be grouped together with square brackets, separated with semi-colons. It is also possible to add a prefix or suffix to parenthetical citations, for example, `[e.g. @author2023, chap. 3; @author1995]`. To add a suffix to a narrative citation, follow the citation with the suffix in square brackets, for example, `@author2023 [chap. 3]`. As with a link to a DOI, you can also use the DOI directly instead of the BibTeX key.
+
+```{list-table} Examples of Markdown citations
+:header-rows: 1
+:name: table-pandoc-citations
+* - Markdown
+  - Rendered
+  - Explanation
+* - `@cockett2015`
+  - @cockett2015
+  - Narrative citation
+* - `[@cockett2015]`
+  - [@cockett2015]
+  - Parenthetical citation
+* - `[@cockett2015; @heagy2017]`
+  - [@cockett2015; @heagy2017]
+  - Multiple parenthetical citations
+* - `[-@cockett2015]`
+  - [-@cockett2015]
+  - Show citation year
+* - `[e.g. @cockett2015, pg. 22]`
+  - [e.g. @cockett2015, pg. 22]
+  - Prefix and suffix
+* - `@cockett2015 [pg. 22]`
+  - @cockett2015 [pg. 22]
+  - Suffix for narrative citations
+* - `@10.1093/nar/22.22.4673`
+  - @10.1093/nar/22.22.4673
+  - Citation using a DOI directly
+```
+
+## Citation Roles
+
+MyST also provides a number of roles for compatibility with Sphinx and JupyterBook. To create a citation role in Markdown, use either a parenthetical or textual citation:
 
 ```md
 This is a parenthetical citation {cite:p}`cockett2015`.
@@ -47,15 +89,3 @@ This will be a citation: {cite}`10.1093/nar/22.22.4673`.
 ```
 
 This will show as: {cite}`10.1093/nar/22.22.4673`.
-
-## Specififying BibTeX
-
-If you want to explicitly reference which bibtex files to use, as well as what order to resolve them in, you can use the `bibliography` field in your frontmatter, which is a string array of local or remote files. This will load the files in order specified.
-
-```yaml
-bibliography:
-  - my_references.bib
-  - https://example.com/my/remote/bibtex.bib
-```
-
-The remote bibtex can be helpful for working with reference managers that support remote links to your references.

docs/frontmatter.md

@@ -173,6 +173,8 @@ banner: banner.png
 Example of a banner in a site using the `article-theme`.
 :::
 
+(authors)=
+
 ## Authors
 
 The `authors` field is a list of `author` objects. Available fields in the author object are:

packages/citation-js-utils/src/index.ts

@@ -39,7 +39,7 @@ export function createSanitizer() {
 function cleanRef(citation: string) {
   const sanitizer = createSanitizer();
   const cleanHtml = sanitizer.cleanCitationHtml(citation).trim();
-  return cleanHtml.replace(/^1\./g, '').trim();
+  return cleanHtml.replace(/^1\./g, '').replace(/&/g, '&').trim();
 }
 
 // eslint-disable-next-line @typescript-eslint/no-unused-vars
@@ -76,7 +76,13 @@ export function getInlineCitation(data: CitationJson, kind: InlineCite, opts?: I
   const year = data.issued?.['date-parts']?.[0]?.[0];
   const prefix = opts?.prefix ? `${opts.prefix} ` : '';
   const suffix = opts?.suffix ? `, ${opts.suffix}` : '';
-  const yearPart = kind === InlineCite.t ? ` (${year}${suffix})` : `, ${year}${suffix}`;
+  let yearPart = kind === InlineCite.t ? ` (${year}${suffix})` : `, ${year}${suffix}`;
+
+  if (opts?.partial === 'author') yearPart = '';
+  if (opts?.partial === 'year') {
+    const onlyYear = kind === InlineCite.t ? `(${year}${suffix})` : `${year}${suffix}`;
+    return [{ type: 'text', value: onlyYear }];
+  }
 
   if (!authors || authors.length === 0) {
     const text = data.publisher || data.title;
@@ -101,7 +107,7 @@ export function getInlineCitation(data: CitationJson, kind: InlineCite, opts?: I
   throw new Error('Unknown number of authors for citation');
 }
 
-export type InlineOptions = { prefix?: string; suffix?: string };
+export type InlineOptions = { prefix?: string; suffix?: string; partial?: 'author' | 'year' };
 
 export type CitationRenderer = Record<
   string,

packages/citation-js-utils/tests/fixtures.ts

@@ -25,9 +25,9 @@ export const FORMATED_CONTENT_VANCOUVER =
 
 // sanitized
 export const TEST_APA_HTML =
-  'Cockett, R., Kang, S., Heagy, L. J., Pidlisecky, A., &amp; Oldenburg, D. W. (2015). SimPEG: An open source framework for simulation and gradient based parameter estimation in geophysical applications. <i>Computers &amp; Geosciences</i>, <i>85</i>, 142–154. <a target="_blank" rel="noreferrer" href="https://doi.org/10.1016/j.cageo.2015.09.015">10.1016/j.cageo.2015.09.015</a>';
+  'Cockett, R., Kang, S., Heagy, L. J., Pidlisecky, A., & Oldenburg, D. W. (2015). SimPEG: An open source framework for simulation and gradient based parameter estimation in geophysical applications. <i>Computers & Geosciences</i>, <i>85</i>, 142–154. <a target="_blank" rel="noreferrer" href="https://doi.org/10.1016/j.cageo.2015.09.015">10.1016/j.cageo.2015.09.015</a>';
 export const TEST_VANCOUVER_HTML =
-  'Cockett R, Kang S, Heagy LJ, Pidlisecky A, Oldenburg DW. SimPEG: An open source framework for simulation and gradient based parameter estimation in geophysical applications. Computers &amp; Geosciences [Internet]. 2015 Dec;85:142–54. Available from: <a target="_blank" rel="noreferrer" href="https://doi.org/10.1016/j.cageo.2015.09.015">10.1016/j.cageo.2015.09.015</a>';
+  'Cockett R, Kang S, Heagy LJ, Pidlisecky A, Oldenburg DW. SimPEG: An open source framework for simulation and gradient based parameter estimation in geophysical applications. Computers & Geosciences [Internet]. 2015 Dec;85:142–54. Available from: <a target="_blank" rel="noreferrer" href="https://doi.org/10.1016/j.cageo.2015.09.015">10.1016/j.cageo.2015.09.015</a>';
 // straight outta citation-js
 export const TEST_DATA_HTML_DIRTY =
   'Cockett, R., Kang, S., Heagy, L. J., Pidlisecky, A., &#38; Oldenburg, D. W. (2015). SimPEG: An open source framework for simulation and gradient based parameter estimation in geophysical applications. <i>Computers &#38; Geosciences</i>, <i>85</i>, 142–154. <a href="https://doi.org/10.1016/j.cageo.2015.09.015" target="_blank">10.1016/j.cageo.2015.09.015</a>';

packages/markdown-it-myst/src/citations.ts

@@ -0,0 +1,122 @@
+/**
+ * @license
+ * citations.js file is adapted from `markdown-it-citations` (Sept 2023)
+ * MIT License - Martin Ring <@martinring>
+ * https://github.com/martinring/markdown-it-citations
+ */
+
+import type { PluginWithOptions } from 'markdown-it';
+import type Token from 'markdown-it/lib/token.js';
+
+type CiteKind = 'narrative' | 'parenthetical';
+type CitePartial = 'author' | 'year';
+
+export interface Citation {
+  label: string;
+  kind: CiteKind;
+  partial?: CitePartial;
+  prefix?: Token[];
+  suffix?: Token[];
+}
+
+export const citationsPlugin: PluginWithOptions = (md) => {
+  const regexes = {
+    citation: /^([^^-]|[^^].+?)?(-)?@([\w][\w:.#$%&\-+?<>~/]*)(.+)?$/,
+    inText: /^@([\w][\w:.#$%&\-+?<>~/]*)(\s*)(\[)?/,
+    allowedBefore: /^[^a-zA-Z.0-9]$/,
+  };
+
+  md.inline.ruler.after('emphasis', 'citation', (state, silent) => {
+    // const max = state.posMax;
+    const char = state.src.charCodeAt(state.pos);
+    if (
+      char == 0x40 /* @ */ &&
+      (state.pos == 0 || regexes.allowedBefore.test(state.src.slice(state.pos - 1, state.pos)))
+    ) {
+      // in-text
+      const match = state.src.slice(state.pos).match(regexes.inText);
+      if (match) {
+        const citation: Citation = {
+          label: match[1],
+          kind: 'narrative',
+        };
+        let token: Token | undefined;
+        if (!silent) {
+          token = state.push('cite', 'cite', 0);
+          token.meta = citation;
+          (token as any).col = [state.pos];
+        }
+        if (match[3]) {
+          // suffix is there
+          const suffixStart = state.pos + match[0].length;
+          const suffixEnd = state.md.helpers.parseLinkLabel(state, suffixStart);
+          const charAfter = state.src.codePointAt(suffixEnd + 1);
+          if (suffixEnd > 0 && charAfter != 0x28 && charAfter != 0x5b /* ( or [ */) {
+            const suffix = state.src.slice(suffixStart, suffixEnd);
+            citation.suffix = state.md.parseInline(suffix, state.env);
+            state.pos += match[0].length + suffixEnd - suffixStart + 1;
+            if (token) token.content = match[0] + suffix + ']';
+          } else {
+            state.pos += match[0].length - match[2].length - match[3].length;
+            if (token) {
+              token.content = match[0];
+              (token as any).col.push(state.pos);
+            }
+          }
+        } else {
+          state.pos += match[0].length - match[2].length;
+          if (token) {
+            token.content = match[0];
+            (token as any).col.push(state.pos);
+          }
+        }
+        return true;
+      }
+    } else if (char == 0x5b /* [ */) {
+      const end = state.md.helpers.parseLinkLabel(state, state.pos);
+      const charAfter = state.src.codePointAt(end + 1);
+      if (end > 0 && charAfter != 0x28 && charAfter != 0x5b) {
+        const str = state.src.slice(state.pos + 1, end);
+        const parts = str.split(';').map((x) => x.match(regexes.citation));
+        if (parts.indexOf(null) >= 0) return false;
+        let curCol = state.pos + 1;
+        const cites = (parts as RegExpMatchArray[]).map(
+          (x): Citation & { content: string; col: number[] } => {
+            // remove the punctuation from the suffix if it exists
+            const suffix = x[4] ? x[4].trim().replace(/^,[\s]*/, '') : undefined;
+            const colEnd = curCol + x[0].length;
+            const meta = {
+              label: x[3],
+              kind: 'parenthetical' as CiteKind,
+              prefix: x[1]?.trim() ? state.md.parseInline(x[1]?.trim(), state.env) : undefined,
+              suffix: suffix ? state.md.parseInline(suffix, state.env) : undefined,
+              partial: x[2] ? ('year' as CitePartial) : undefined,
+              content: x[0].trim(),
+              col: [curCol, colEnd],
+            };
+            curCol = colEnd + 1;
+            return meta;
+          },
+        );
+        if (!silent) {
+          const token = state.push('cite_group_open', 'span', 1);
+          token.content = '[';
+          token.meta = { kind: 'parenthetical' };
+          cites.forEach((citation) => {
+            const { content, col, ...meta } = citation;
+            const cite = state.push('cite', 'cite', 0);
+            cite.content = content;
+            (cite as any).col = col;
+            cite.meta = meta;
+          });
+          const close = state.push('cite_group_close', 'span', -1);
+          close.content = ']';
+        }
+        state.pos = end + 1;
+        return true;
+      }
+      return false;
+    }
+    return false;
+  });
+};

packages/markdown-it-myst/src/index.ts

@@ -1,9 +1,9 @@
 import type MarkdownIt from 'markdown-it/lib';
 import { rolePlugin } from './roles.js';
 import { directivePlugin } from './directives.js';
+import { citationsPlugin } from './citations.js';
 
-export { rolePlugin };
-export { directivePlugin };
+export { rolePlugin, directivePlugin, citationsPlugin };
 
 /**
  * A markdown-it plugin for parsing MyST roles and directives to structured data

packages/markdown-it-myst/src/roles.ts

@@ -31,13 +31,13 @@ function roleRule(state: StateInline, silent: boolean): boolean {
   const match = ROLE_PATTERN.exec(state.src.slice(state.pos));
   if (match == null) return false;
   const [str, name, , content] = match;
-  state.pos += str.length;
-
   if (!silent) {
     const token = state.push('role', '', 0);
     token.meta = { name };
     token.content = content;
+    (token as any).col = [state.pos, state.pos + str.length];
   }
+  state.pos += str.length;
   return true;
 }
 
@@ -63,13 +63,14 @@ function runRoles(state: StateCore): boolean {
         if (child.type === 'role') {
           try {
             const { map } = token;
-            const { content } = child;
+            const { content, col } = child as any;
             const roleOpen = new state.Token('parsed_role_open', '', 1);
             roleOpen.content = content;
             roleOpen.hidden = true;
             roleOpen.info = child.meta.name;
             roleOpen.block = false;
             roleOpen.map = map;
+            (roleOpen as any).col = col;
             const contentTokens = roleContentToTokens(content, map ? map[0] : 0, state);
             const roleClose = new state.Token('parsed_role_close', '', -1);
             roleClose.block = false;