jgm / pandoc

Add --citeproc (-C) option to trigger built-in citation processing. It is no longer necessary to use the external pandoc-citeproc filter. --citeproc behaves like a filter and can be positioned relative to other filters as they appear on the command line.

The new built-in citation processing uses the citeproc library, a reimplementation of CSL citation processing that fixes many of the shortcomings of pandoc-citeproc. In general, citation processing should work much the same as it did with pandoc-citeproc, but with greater fidelity to CSL styles and better performance. (The tests from the pandoc-citeproc package have been carried over to pandoc.) The following differences should be noted:

• At this point, only some of the writers (HTML, ms, LaTeX) properly interpret CSL display styles. You should get decent output in all formats (at least as good as with pandoc-citeproc), but indentation and block-alignment may not be right.

• pandoc-citeproc searches the ~/.csl directory for .csl styles. Pandoc instead searches the csl subdirectory of the pandoc user data directory (e.g., ~/.pandoc/csl or ~/.local/share/pandoc/csl). Users who already keep CSL styles in ~/.csl may find it convenient to add a symlink.

• Some of the bibliography formats supported by pandoc-citeproc (via hs-bibutils) are no longer supported: Copac, EndNote, ISI, MEDLINE, MODS, and RIS. If you use one of these formats, you may use the bibutils utility to convert to BibLaTeX. We now support only BibTeX, BibLaTeX, CSL JSON, and pandoc’s YAML/Markdown analogue of CSL JSON.

• pandoc-citeproc would always retrieve the independent parent of a dependent style by doing an HTTP request. pandoc will now first seek the independent parent locally (in the resource path or in the csl subdirectory of the pandoc user data directory) before resorting to HTTP. In addition, you may omit the .csl extension, e.g. --csl zoology.

• Using the --bibliography option (or including bibliography in YAML metadata) no longer triggers citation processing implicitly: one must always use the --citeproc option if citation processing is wanted.

Add csljson as and input and output format. This allows pandoc to convert between csljson and other bibliography formats (e.g. -f csljson -t markdown -s or -f bibtex -t csljson), and to generate formatted versions of CSL JSON bibliographies (e.g., pandoc -f csljson --citeproc pl.json -o pl.pdf).

Added bibtex, biblatex as input formats. This allows pandoc to convert between BibLaTeX and BibTeX and other bibliography formats, and to generated formatted versions of BibTeX/BibLaTeX bibliographies (e.g., pandoc -f biblatex --citeproc pl.bib -o pl.pdf).

Raise informative errors when YAML metadata parsing fails (#6730). Previously the command would succeed, returning empty metadata, with no errors or warnings.

Sort languages in --list-highlight-languages output (#6718, Albert Krewinkel). Languages were previously sorted by their long name, which leads to unexpected results).

Add CSS to default HTML template (#6601, Mauro Bieg). This greatly improves the default typography in pandoc’s HTML output. The CSS is sensitive to a number of variables (e.g. mainfont, fontsize, linestretch): see the manual for details. To restore the earlier, more spartan output, you can disable this with -M document-css=false.

Support --toc-depth option for ODT writer (#6696, niszet).

Fix issues with Windows UNC paths with some options (#5127).

Remove fenced_code_blocks and backtick_code_blocks from allowed commonmark and gfm extensions. These shouldn’t really be counted as extensions, because they can’t be disabled in commonmark. Adjust markdown writer to check for the commonmark variant in addition to extensions.

Add these extensions to gfm and commonmark: fenced_code_blocks, backtick_code_blocks, fenced_code_attributes. These can’t really be disabled in the reader, but they need to be enabled in the writer or we just get indented code.

Make sure proper set of extensions is recognized for commonmark_x.

Allow gfm_auto_identifiers, ascii_identifiers extensions for docx.

Markdown reader:

• Add Maybe FilePath parameter to yamlToMeta [API change].
• Export yamlToRefs [API change], a version of yamlToMeta specialized to references.
• Set citationNoteNum accurately in citations.
• Revise abbreviation support. Don’t insert a nonbreaking space after a potential abbreviation if it comes right before a note or citation. This causes problems for citeproc’s moving of note citations.

LaTeX reader:

• Support missing siunitx commands (#6658).
• Support squared, cubed, tothe in siunitx (#6657).
• Improved uncertainty handling in slunitx.
• Factored out siunitx stuff into separate unexposed module.
• Fix improper empty cell filtering (#6689, Christian Despres).
• Fix parsing of “show name” in \newtheorem (#6734). Previously we were just treating it as a string and ignoring accents and formatting.
• Prevent wrong nesting of \multirow and \multicolumn table cells (#6603, Laurent P. René de Cotret).
• Table cell parser not consuming spaces correctly (#6596, Laurent P. René de Cotret).
• Change SIRange to SIrange (#6617, Emerson Harkin).
• Allow blank lines inside \author (#6324).

DocBook reader:

• Don’t squelch space at end of emphasis and other inline elements; instead, move it outside the element (#6719).
• Implement table cell alignment (#6698, Nils Carlson).
• Implement column span support for tables (#6492, Nils Carlson).
• Update list of block level tags (#6610).

JATS reader:

• Don’t squelch space at end of emphasis and other inline elements; instead, move it outside the element (#6719).

RST reader:

• Apply .. class:: directly to following Header rather than creating a surrounding Div (#6699).

Docx reader:

• Allow empty dates in comments and tracked changes (#6726, Diego Balseiro).

Markdown writer:

• Be less aggressive about using quotes for YAML values, allowing e.g. a quotation mark or bracket as long as it’s not at the beginning of the line.
• Use double quotes for YAML metadata (#6727).
• Sort YAML metadata keys in Markdown output case-insensitive.

Asciidoc writer:

• Support asciidoctor’s block figures (#6538, argent0).

LaTeX writer:

• Fix spacing issue with list in definition list. When a list occurs at the beginning of a definition list definition, it can start on the same line as the label, which looks bad. Fix that by starting such lists with an \item[].

HTML writer:

• Support intermediate table headers (#5314, Albert Krewinkel).
• Support attributes on all table elements (Albert Krewinkel).
• Render table footers if present (#6314, Albert Krewinkel).
• Fix addition of doc-biblioentry role.
• Support colspans and rowspans in HTML tables (#6312, Albert Krewinkel).

ICML writer:

• Support internal document links (#5541, Leonard Rosenthol).
• Changed default link state to invisible (#6676, Leonard Rosenthol).

Docx writer:

• Better handle list items whose contents are lists (#5948, Michael Hoffmann). If the first element of a bulleted or ordered list is another list, then that first item would previously disappear if the target format is docx.
• Separate adjacent tables (#4315). Word combines adjacent tables, so to prevent this we insert an empty paragraph between two adjacent tables.

Org writer:

• Don’t force blank line after headings (#6554).

OpenDocument writer:

• Implement table cell alignment (#6700 Nils Carson, Mauro Bieg).
• New table cell support with row and column spans (#6682, Nils Carson).
• Syntax highlighting for inline code (#6711, niszet).

Add Text.Pandoc.Citeproc module, exporting processCitations [API change]. This depends on several other, unexported modules under Text.Pandoc.Citeproc.

Add module Text.Pandoc.Writers.CslJson, exporting writeCslJson. [API change]

Add module Text.Pandoc.Readers.CslJson, exporting readCslJson. [API change]

Add module Text.Pandoc.Readers.BibTeX, exporting readBibTeX and readBibLaTeX. [API change]

Text.Pandoc.Filter: Add CiteprocFilter constructor to Filter. [API change] This runs the processCitations transformation. We need to treat it like a filter so it can be placed in the sequence of filter runs (after some, before others). In FromYAML, this is parsed from citeproc or {type: citeproc}, so this special filter may be specified either way in a defaults file (or by citeproc: true, though this gives no control of positioning relative to other filters).

Add new exported module Text.Pandoc.Writers.AnnotatedTable [API change] (#6655, Christian Despres). This module (which should generally be imported qualified to avoid name conflicts) provides a Table type that mirrors the structure of a pandoc Table, but with added inferred information so that the writers do not have to lay out tables themselves. The toTable and fromTable functions convert between an annotated Table and a regular pandoc Table. In addition to producing a Table with coherent and well-formed annotations, the toTable function also normalizes its input table like the table builder does. Tests ensure that toTable normalizes tables exactly like the table builder, and that its annotations are coherent.

Text.Pandoc.Logging:

• Remove unused CouldNotParseYamlMetadata constructor for LogMessage [API change].
• Add CiteprocWarning constructor to LogMessage [API change].

Text.Pandoc.Readers.Metadata: export yamlBsToRefs [API change]. These allow specifying an id filter so we parse only references that are used in the document.

Text.Pandoc.Parsing:

• Export ParseError [API change].
• Add stateInNote and stateNoteNumber to ParserState [API change]. These are used to populate note numbers for citation processing.

Fix apparent typos in sample.lua (#6729, William Lupton). Also make the writer less aggressive in escaping quotes.

Text.Pandoc.Options:

• defaultMathJaxURL: use tex-chtml-full instead of tex-mml-chtml (#6599, Kolen Cheung). This drops the MathML support (which we don’t need for HTML math rendering) and includes the full JavaScript, which makes it possible to use --self-contained (though there may still be issues if the required math fonts aren’t available). This change should also reduce latency in pages with lots of formulas.
• Add /tex-chtml-full.js to defaultMathJaxURL (#6593) Previously we added this in processing command line options, but not in processing defaults files, which was inconsistent.

epub.css: Fix cover page selectors and add note explaining their use (#6649, a-vrma).

Add data files needed for Text.Pandoc.Citeproc: these include default.csl in the data directory and a citeproc directory that is only used at compile-time for biblatex localizations. Note that we’ve added file-embed as a mandatory rather than a conditional depedency, because of the biblatex localization files.

Lua filters:

• Add SimpleTable for backwards compatibility (#6575, Albert Krewinkel). A new type SimpleTable is made available to Lua filters. It is similar to the Table type in pandoc versions before 2.10; conversion functions from and to the new Table type are provided. Old filters using tables now require minimal changes and can use, e.g.,

  if PANDOC_VERSION > {2,10,1} then
    pandoc.Table = pandoc.SimpleTable
  end
  

  and

  function Table (tbl)
    tbl = pandoc.utils.to_simple_table(tbl)
    …
    return pandoc.utils.from_simple_table(tbl)
  end
  

  to work with the current pandoc version.

• Make attr argument optional in Table constructor (Albert Krewinkel). This changes the Lua API. It is highly unlikely for this change to affect existing filters, since the documentation for the new Table constructor (and type) was incomplete and partly wrong before. The Lua API is now more consistent, as all constructors for elements with attributes now take attributes as the last parameter.

MANUAL.txt:

• Add a dedicated Citations section which consolidates the information the manual used to contain about citation processing, and incorporates some information formerly found in the pandoc-citeproc man page.
• Add note about lualatex using selnolig.
• Remove duplicate seriespage (#6568, Blake Eryx).
• Remove lists of support extensions for markdown variants (#6604). Instead, offer the advice to use --list-extensions=FORMAT.
• Fix position of attributes in header (Albert Krewinkel).
• Delete obsolete section on compact and loose lists (#6684).

doc/lua-filters.md:

• Add info on how to debug Lua filters (#6732, Ian Max Andolina).
• Document Underline type and constructor (Albert Krewinkel).
• Document body field (Albert Krewinkel).
• Add missing header attribute
• Add missing Link.title field (Albert Krewinkel).
• Make the setting-the-date example conditional (the-solipsist). This makes the example a bit more realistic.
• Remove outdated link table example.

doc/org.md:

• Add section on tables (Albert Krewinkel).
• Add section on handling of unknown directives (Albert Krewinkel).

CONTRIBUTING.md: fix typo (#6584, Dmitry Volodin).

Use golden test framework for command tests. This means that --accept can be used to update expected output.

Use the smart extension when generating pandoc’s man page (#6613).

Release-candidate: don’t build windows i386. So far we haven’t been able to figure out how to get stack to use a 32-bit ghc.

Use null instead of deprecated Builder.isNull.

Makefile:

• Fix macospkg target to fetch target from S3 artifacts.
• Fix pandoc-templates target to include all partials.

Remove duplicated dependency in pandoc.cabal (#6591, Felix Yan).

Sort build depends in pandoc.cabal alphabetically (#6691, Albert Krewinkel).

Add .travis.yml for macos release candidate build (#6622). We need to build the release candidate on Travis rather than GitHub actions, because GH has macos 10.15, and binaries compiled on that OS will not work with 10.13. This build is only triggered on rc/* branches.

Remove instructions for building pandoc-citeproc from CI and release binary build instructions. We will no longer distribute pandoc-citeproc.

Fix math rendering in trypandoc (this broke after commit d8ad766).

Use latest versions of skylighting, commonmark (#6589), comonmark-extensions, commonmark-pandoc, texmath.

Relax version bounds for hslua, hslua-module-text, bytestring.

Use released pandoc-types 1.22. This changes the JSON encoding slightly for the new table types introduced in 1.21, so they’re more consistent with the rest. Developers of libraries for pandoc filters will want to take note.

Fix hlint suggestions, update hlint.yaml (#6680, Christian Despres).

Code cleanup (#6678, Joseph C. Sible).

Add haddocks to functions in Text.Pandoc.Writers.Shared (Albert Krewinkel).

Remove duplicate tshow definition.

Linux release candidate build: use ghc-musl container. This simplifies our build process (over using a customized alpine container).

Add commonmark_x output format. This is commonmark with a number of useful pandoc extensions enabled.

Many more extensions now work with commonmark and gfm.

Add generic attributes extension. This allows attributes to be added to any block or inline element in a uniform way. Since the Pandoc AST doesn’t include attributes on each element type, the attributes will sometimes be added by creating a surrounding Div or Span container. Currently this extension is only compatible with the commonmark and gfm readers.

To add an attribute to a block-level element, e.g. a paragraph, put it before the block:

{#mypara}
This is a paragraph.

Multiple attributes may be used and will be combined:

{#mypara}
{.blue .warning key="val"}
This is a paragraph.

To add an attribute to an inline-level element, put it immediately after the element:

*emphasized text*{.special}

Support --number-sections for docx output (#1413).

LaTeX reader:

• Support \SIRange reader (#6418, Emerson Harkin).
• Support table col-span and row-span (#6311, Laurent P. René de Cotret). Supports \multirow and \multicolumn.
• Support amsthm: \newtheorem, \theoremstyle, and theorem and proof environments, including labels and references. The only thing that is unsupported is the second optional argument, which causes numbering to be reset after the specified series is incremented.
• Moved some code to T.P.LaTeX.Parsing. We need to reduce the size of the LaTeX reader to ease compilation on resource-limited systems.

RST reader:

• Fix csv tables with multiline cells (#6549).
• Fix spurious newlines in some attributes from directives.
• Avoid extra newline in included code blocks.

Commonmark reader:

• Switch from cmark-gfm to commonmark-hs for commonmark and gfm parsing. This avoids depending on a C library and allows us to support more pandoc extensions for commonmark and gfm.

DocBook reader:

• Parse releaseinfo as metadata (#6542).

Docx reader:

• Only use bCs/iCs on runs with rtl or cs property (#6514, Nikolay Yakimov).
• Code cleanup/refactoring (Nikolay Yakimov).

Org reader (Albert Krewinkel):

• Respect export setting which disables entities MathML-like entities, e.g., \alpha, can be disabled with the #+OPTION: e:nil export setting (Albert Krewinkel).
• Respect export setting disabling footnotes. Footnotes can be removed from the final document with the #+OPTION: f:nil export setting.
• Respect tables-excluding export setting. Tables can be removed from the final document with the #+OPTION: |:nil export setting.

Markdown writer:

• Move asciify out of escapeString. Otherwise unsmartify doesn’t catch quotes that have already been turned to entities.
• Add writeCommonmark (new exported function, API change).
• Use unicode super/subscript characters when possible if the superscript or subscript extension or raw_html aren’t available.
• Render caption as following paragraph when table_caption extension is not enabled.
• Use numerical labels for reference links that are longer than 999 characters or contain square brackets, for conformity with commonmark (#6560).

Commonmark writer:

• Instead of using cmark-gfm, use writeCommonmark from the Markdown writer. This function calls the markdown writer with appropriate extensions and a few small modifications (e.g. not requiring backslashes before spaces inside super/subscripts). With this change comonmark and gfm output can be used with a wider selection of extensions.

Jira writer: keep image caption as alt attribute (#6529, Albert Krewinkel).

HTML writer:

• Improve alt-text/caption handling for HTML5 (#6491, Albert Krewinkel). Screen readers read an image’s alt attribute and the figure caption, both of which come from the same source in pandoc. The figure caption is hidden from screen readers with the aria-hidden attribute. This improves accessibility. For HTML4, where aria-hidden is not allowed, pandoc still uses an empty alt attribute to avoid duplicate contents.

Ms writer:

• Fix code highlighting with blank lines. Previously blank lines were simply omitted from highligted code.
• Escape starting periods in ms writer code blocks (#6505, Michael Hoffmann). If a line of ms code block output starts with a period (.), it should be prepended by \& so that it is not interpreted as a roff command.

Text.Pandoc.Extensions:

• Add raw_markdown extension (which only affects ipynb input).

• Trim down githubMarkdownExtensions. Previously it included all of the following, which make sense for the legacy markdown_github but not for gfm, since they are part of base commonmark and thus can’t be turned off in gfm:

  • Ext_all_symbols_escapable
  • Ext_backtick_code_blocks
  • Ext_fenced_code_blocks
  • Ext_space_in_atx_header
  • Ext_intraword_underscores
  • Ext_lists_without_preceding_blankline
  • Ext_shortcut_reference_links

  These have been removed from githubMarkdownExtensions, though they’re still turned on for legacy markdown_github.

• Add Ext_attributes constructor for Extension [API change].

LaTeX template: use selnolig to selectively suppress ligatures with lualatex (#6534).

Benchmark bytestring readers (Nikolay Yakimov).

Documentation:

• Update using-the-pandoc-api.md (favonia).
• Fix Typos in lua-filters.md (tajmone).
• Rewrite Raw HTML/TeX section in MANUAL.txt to avoid duplicate headings for the extensions.
• Fix typo in MANUAL.txt (Benjamin Wuethrich).
• Remove duplicate ‘titlepage’ in MANUAL.txt (Blake Eryx).
• CONTRIBUTING.md: Advertise the official nightlies in GitHub actions. Replaces #6500, thanks to @ickc.

Use pandoc-types 1.21. This adds two things:

• A native Underline constructor for Inline (#6277, Vaibhav Sagar).
• More expressive types for tables (#1024, Christian Despres). Tables can now take attributes; and rowspans and colspans, column headers, multiple row headers, table head and foot can all be represented. (Note, however, that reader and writer support for these features is still lacking, so most users won’t see any differences in table conversion yet. These changes just lay the foundation for further improvements.)

Support new Underline element in readers and writers (#6277, Vaibhav Sagar).

Support new Table type (Christian Despres). The Builder.simpleTable now only adds a row to the TableHead when the given header row is not null. This uncovered an inconsistency in the readers: some would unconditionally emit a header filled with empty cells, even if the header was not present. Now every reader has the conditional behaviour. Only the XWiki writer depended on the header row being always present; it now pads its head as necessary.

Add an option to disable certificate validation (#6156, Cédric Couralet, Cécile Chemin, Juliette Fourcot). This commit adds the option --no-check-certificate, which disables certificate checking when resources are fetched by HTTP.

Unify defaults and markdown metadata parsers (#6328, Nikolay Yakimov). Clean up code in Text.Pandoc.Readers.Metadata and properly handle errors in yamlToMeta. This fixes parsing of Boolean fields in metadata withinin defaults files and reduces code duplication.

Docbook reader:

• Implement <procedure> (#6442, Mathieu Boespflug).
• Implement <phrase> (#6438, Mathieu Boespflug).
• Treat envar and systemitem like code (#6435, Mathieu Boespflug).
• Implement <replaceable> (#6437, Mathieu Boespflug)
• Map <simplesect> to unnumbered section (#6436, Mathieu Boespflug).

JATS reader:

• Handle “label” element in section title (#6288).
• Parse abstract element into metadata field of same name (#6480, Albert Krewinkel).

Jira reader (Albert Krewinkel):

• Resolve multiple parsing problems, including issues with empty table cells, faulty recognition of closing emphasis characters, and parsing of image attributes (#6212, #6219, #6220).
• Two consecutive markup chars are now parsed verbatim (#6343); styled text must not be empty.
• Newlines are no longer allowed within styled text (#6325).
• Links to anchors are now parsed as links (#6407).
• Retain image attributes (#6234). Jira images attributes as in !image.jpg|align=right! are retained as key-value pairs. Thumbnail images, such as !example.gif|thumbnail!, are marked by a thumbnail class in their attributes.
• Use Underline for inserted text (#6237). Previously, the span was marked with the non-standard class inserted.
• Improve icon conversion for (/), (x), (!), (?) (+), (-), (off), (*). (#6236, #6264).
• Support citations, attachment links, and user links (#6231, #6238, #6239).
• Resolve parsing issues of blockquote, color (#6233, #6235).

HTML reader:

• Parse attributes into table attributes.
• Support <bdo> (#5794, Tristan de Cacqueray).
• Add summary to list of block-level HTML tags (#6385). This improves support for summary/details inside Markdown. NOTE: you need to include a blank line before the closing </details>, if you want the last part of the content to be parsed as a paragraph.
• Fix parsing unclosed th elements in a table (#6247).

Commonmark reader: Implement implicit_figures extension (#6350).

Markdown Reader:

• Fix inline code in lists (#6284, Nikolay Yakimov). Previously inline code containing list markers was sometimes parsed incorrectly.
• Don’t require blank line after grid table (#6481). This allows grid tables to be enclosed in fenced divs with no intervening blank lines.

LaTeX reader:

• Don’t parse beyond \end{document} (#6380). This required some internal changes to \subfile handling.
• Better handling of \lettrine. SmallCaps instead of Span for the part after the initial capital. Ensure that both arguments are parsed, so that in Markdown both are treated as raw LateX. (Closes #6258.)

Org reader (Albert Krewinkel):

• Recognize images with uppercase extensions (#6472).
• Keep unknown keyword lines as raw org. The lines of unknown keywords, like #+SOMEWORD: value are no longer read as metadata, but kept as raw org blocks. This ensures that more information is retained when round-tripping org-mode files; additionally, this change makes it possible to support non-standard org extensions via filters.
• Unify keyword handling. Handling of export settings and other keywords (like #+LINK) has been combined and unified.
• Support LATEX_HEADER_EXTRA and HTML_HEAD_EXTRA settings. These export settings are treated like their non-extra counterparts, i.e., the values are added to the header-includes metadata list.
• Allow multiple #+SUBTITLE export settings. The values of all lines are read as inlines and collected in the subtitle metadata field.
• Read #+INSTITUTE values as text with markup. The value is stored in the institute metadata field and used in the default beamer presentation template.
• The behavior of the #+AUTHOR and #+KEYWORD export settings has changed: Org now allows multiple such lines and adds a space between the contents of each line. Pandoc now always parses these settings as meta inlines; setting values are no longer treated as comma-separated lists. Note that a Lua filter can be used to restore the previous behavior.
• Read description lines as inlines (#6485). #+DESCRIPTION lines are now treated as text with markup. If multiple such lines are given, then all lines are read and separated by soft linebreaks.
• Honor tex export option (#4070). The tex export option can be set with #+OPTION: tex:nil and allows three settings: t (the default) causes LaTeX fragments to be parsed as TeX or added as raw TeX. nil removes all LaTeX fragments from the document. verbatim treats LaTeX as text.

RST reader:

• Pass arbitrary attributes through in code blocks (#6465). Exceptions: name (which becomes the id), class (which becomes the classes), and number-lines (which is treated specially to fit with pandoc highlighting).
• Handle date:: directive (#6276).

Textile reader: support pre. for code blocks (#6454).

Ipynb reader:

• Handle application/pdf output as image (#6430).
• Properly handle image/svg+xml as an image (#6430).

Docx reader:

• Distinguish between docx parsing and docx container unpacking errors.

MediaWiki reader:

• Fix gfm_auto_identifiers so that - is not replaced by _ (#6335).

Vimwiki reader:

• Add nested syntax highlighting (#6256, Vlad Hanciuta). Nested syntaxes are specified like this:

  {{{sql
  SELECT * FROM table
  }}}
  

  The preformatted code block parser has been extended to check if the first attribute of the block is not a key=value pair, and in that case it will be considered as a class.

Jira writer (Albert Krewinkel):

• Always escape braces (#6478). Braces are now always escaped, even within words or when surrounded by whitespace. Jira and Confluence treat braces specially.
• Convert Underline to inserted text (+inserted+).
• Add image attributes (#6234). Image attributes are added to the output as image parameters. If the image has a class “thumbnail”, then a thumbnail image is generated; all other attributes are discarded in this case.

LaTeX writer:

• Ensure that -M csquotes works even in fragment mode (#6265).
• Escape ^ specially for listings (#6460).
• Create hypertarget for links with identifier (#6360).
• Distinguish between single and double quotes when using enquote package (#6457, dbecher-ito).
• Add support for customizable alignment of columns in beamer (#6331, andrebauer).
• Add support for customizable alignment of columns in beamer (#4805, #4150, andrebauer).

HTML writer:

• Use CSS in favor of <br> for display math (#6372) Some CSS to ensure that display math is displayed centered and on a new line is now included in the default HTML-based templates; this may be overridden if the user wants a different behavior.

Org writer:

• Clean-up Div handling (Albert Krewinkel).

Docx writer:

• Enable column and row bands for tables (#6371). This change will not have any effect with the default style. However, it enables users to use a style (via a reference.docx) that turns on row and/or column bands.

OpenDocument (and ODT) writer:

• Add custom-style “Abstract” in metadata abstract. This ensures that the abstract is rendered with style Abstract.
• Enable custom-style attribute on a Div. This allows you to apply a custom style to contained paragraphs.

DocBook writer:

• Add id of figure to enclosed image.
• Add personname element to docbook author (#6244).

FB2 writer:

• Properly handle cover-image containing spaces (#6391).

Markdown writer:

• Ensure consistent padding for pipe tables (#6240).
• Avoid unnecessary escapes before intraword _ when intraword_underscores extension is enabled (#6296).

RST writer:

• Properly handle images with same alt text (#6194). Previously we created duplicate references for these in rendering RST.

AsciiDoc writer:

• Add blank line after Div (#6308).

Haddock Writer:

• Support Haddock tables (Joe Hermaszewski). See this PR on Haddock for details on the table format: haskell/haddock#718.

PowerPoint writer (Jesse Rosenthal):

• Write math input verbatim in speaker notes (#6301). OMML in speaker notes would lead to corrupt PowerPoint output. We now output the OMML verbatim as LaTeX in the speaker notes.

LaTeX template: Make polyglossia package options list-aware (#6444, Frederik Elwert).

Reveal.js template:

• Update template for reveal.js 4.0.0 (#6390, Salim B).
• Update template with newly available options (#6347, Jake Zimmerman).
• Use CDN version of revealjs v4 by default (#6408).

opendocument template: Add abstract and subtitle to opendocument template (#6369).

reference.odt: clean up styles. Add Abstract. Change Author, Date to centered paragraphs with no character styling.

epub.css: wrap overlong lines in highlighted code blocks (#6242). This fixes a problem in iBooks v2.4 with our earlier horizontally scrolling code blocks. The problem seems to be a bug in iBooks, not pandoc, but since iBooks is a major target we’re changing pandoc’s default behavior so that pandoc-produced epubs work on that platform.

Text.Pandoc.PDF:

• Use --enable-local-file-access in invoking wkhtmltopdf (#6474). wkhtmltopdf changed in recent versions to require this for access to local files. This fixes PDF via HTML5 with --css.
• Send verbose output to stderr, not stdout (#6483).

Text.Pandoc.MIME: Fix MIME type for TrueType fonts in EPUBs (#6464, Michael Reed).

Text.Pandoc.Shared:

• makeSections: omit number attribute when unnumbered class is present (#6339). Previously the attribute was included but given an empty value, and this caused the table of contents creation functions in Text.Pandoc.Writers.Shared to think these items had numbers, which meant that they were included in the TOC even if the unlisted class was used.
• Deprecate underlineSpan in Shared in favor of Text.Pandoc.Builder.underline (Vaibhav Sagar).
• renderTags': use self-closing tag for col element (#6295).

Text.Pandoc.UUID: Fix getRandomUUID, which previously would return the same value twice in a row. Make getRandomUUID polymorphic in PandocMonad. Remove getUUID (#6228, Joseph C. Sible).

Text.Pandoc.Class: Generalize PandocIO functions to MonadIO.

Fixed Katex standalone script (#6399, Lucas Escot). Global macros are now persistent when using the HTML Writer with the --katex option.

Lua subsystem (Albert Krewinkel):

• Use new type PandocLua for all pandoc Lua operations (API change). The new type PandocLua is an instance of the PandocMonad typeclass and can thus be used in a way similar to PandocIO.
• Use PandocError for exceptions in Lua subsystem (API change). The PandocError type is used throughout the Lua subsystem. All Lua functions throw an exception of this type if an error occurs. The LuaException type is removed and no longer exported from Text.Pandoc.Lua. In its place, a new constructor PandocLuaError is added to PandocError.

Lua filters: improve error messages for failing filters (#6332, Albert Krewinkel). Print the Lua error properly instead of displaying their show string.

Use latest skylighting. This fixes a bug with lua multiline comments (and may improve handling of other syntaxes as well). IT also adds aria-hidden="true" to the empty a elements, which helps people who use screen readers.

Use latest texmath.

Require latest doctemplates 0.8.2. This adds support for template pipes first, rest, last, allbutlast.

Revert 0e48a02 and dependency on base-noprelude, which hasn’t been updated for ghc 8.10 (see #6187).

Dependency adjustments:

• Allow haddock-library 1.9.x.
• Allow hslua 1.1 (#6243, Felix Yan).
• Allow base64-bytestring 1.1.
• Use latest jira-wiki-markup.
• Allow http-client 0.7.
• Allow tasty 1.3.x.
• Allow aeson 1.5 (#6400, Felix Yan).
• Remove unused dependency vector (#6462, Laurent P. René de Cotret).
• Bump QuickCheck upper bound.

Significant code cleanup and simplification (Joseph C. Sible, #6223, #6209, #6225, #6229, #6226, #6340).

Remove unnecessary hlint ignores (#6341, Joseph C. Sible).

Remove obsolete RelaxedPolyRec extension (#6487, Nikolay Yakimov).

trypandoc improvements (Mike Tzou):

• Add standalone option to the command text (#6210).
• Update third party libraries.

MANUAL.txt:

• Clarify template partial naming (#6476, Mauro Bieg).
• Describe jira as “Jira/Confluence wiki markup” (#6351, Albert Krewinkel). In the past, Jira’s wiki markup was also used by – and could be imported into – Atlassian Confluence.
• Add link to print-css.rocks (#6272, Mauro Bieg).
• Clarify pipe table column width adjustment (#6254).
• Fix ATX header syntax.
• Fix misleading note about image size conversions (#6353).
• Update links to reveal.js documentation (#6386, Salim B).
• Separate adjacent verbatim code blocks (#6307, tom-audm).

org.md:

• Document behavior of smart extension (#4387, Albert Krewinkel).
• Describe all supported export options in detail.

lua-filters.md:

• Fix description of BulletList Lua type (Levi Gruspe).
• Use pandoc.system module in TikZ example (Albert Krewinkel). Showcase temporary directory handling with with_temporary_directory and with_working_directory.

INSTALL.md: fix FreeBSD port link (#6422, Mo). The FreeBSD port was renamed from pandoc to hs-pandoc in 2010. The old pandoc port is still at version 1.5.1.1

Propagate (DY)LD_LIBRARY_PATH in tests (#6376, Lila).

Bump cabal-version to 2.2 (#6377).

Make it possible to compile using Stack on NixOS (#6439, Mathieu Boespflug).

CI action to check for commit messsage length (Nikolay Yakimov, #6398).

Markdown reader: Fix table alignment when heading begins with t (#6153). Due to a typo (t instead of \t) we were center aligning column headings that begin with a lowercase t!

Text.Pandoc.Readers.Roff:

• Fix parsing of \. in man/ms readers (#6175). Previously due to a typo it was being parsed as `.
• Fix parsing of \' in man/ms readers (#6175). It was being parsed as a backtick.

Jira reader (Albert Krewinkel):

• Fix parsing of tables without preceding blankline (#6198). A bug was fixed which caused faulty parsing if a table was not preceded by a newline and the first table cell had no space after the initial | characters.
• Fix parsing of strikeout, emphasis (#6196). A bug was fixed which caused non-emphasized text containing digits and/or non-special symbols (like dots) to sometimes be parsed incorrectly.
• Support colored inline text, indented lists (#6183, #6184).

Ms writer:

• Fix definition lists so indent even when paragraph indent is set to 0 (as is the default). Also ensure indent for display math that falls back to TeX.
• Use .QS/.QE instead of .RS/.RE for block quotes.

EPUB writer: fix regression on detection of front/back/bodymatter (#6170). This bug caused sections with epub:type dedication to be misplaced in bodymatter instead of frontmatter as specified in the manual. The same problem would affect other epub:types. The pattern matching needed to be changed with the use of makeSection.

AsciiDoc writer: remove redundant otherwise guard in inlineToAsciiDoc (#6146, Ryan Scott).

Text.Pandoc.Class:

• Fix missing import when data files are not embedded (Albert Krewinkel).
• Subdivide Text.Pandoc.Class into small unexported modules and ensure that all functions have Haddock documentation (#6106, Albert Krewinkel).
• Finer grained imports of Text.Pandoc.Class submodules (#6203, Albert Krewinkel).

Text.Pandoc.XML: Add to list of HTML5 attributes: allow, autocapitalize, decoding, enterkeyhint, imagesizes, imagesrcset, loading.

Use implicit Prelude (#6187, Albert Krewinkel). The previous behavior was introduced as a fix for #4464. It seems that this change alone did not fix the issue, and stack ghci and cabal repl only work with GHC 8.4.1 or newer, as no custom Prelude is loaded for these versions. Given this, it seems cleaner to revert to the implicit Prelude.

Always use custom prelude (#6187, Albert Krewinkel). Previously, the custom prelude was used only with older GHC versions, as a workaround for problems with ghci. The ghci problems are resolved by replacing package base with base-noprelude, allowing for consistent use of the custom prelude across all GHC versions.

Remove outdated checks for no longer supported base versions (Albert Krewinkel).

PDF via wkhtmltopdf: put user-specified options last (#6171). Certain options (e.g. cover) need to come after flags on the command line.

Text.Pandoc.App: set resource path at the beginning so it can affect things like include-in-header (#5982).

Change macOS release candidate CI process so that notarized packages can be produced (#6169).

Make MANUAL more explicit about nonbreaking space handling by all_symbols_escapable (#6154, Fabien Schurter).

trypandoc (Mike Tzou):

• Add checkbox for standalone option (#6189).
• Use strict mode for JavaScript code (#6188).
• Fetch resources over https (#6188).
• Remove unnecessary attributes on style, script elements (#6188).

Use details tag to make GitHub releases changelog collapsible.

Update filter code in doc/filters.md so it works with latest pandoc (#6185).

linux/Dockerfile: upgrade to alpine 3.11 (#6180, Albert Krewinkel). This is used to build the static linux binaries.

Add csv as an input format (#6100). The CSV table is converted into a pandoc simple table. A new module Text.Pandoc.Readers.CSV exports readCSV [API change].

Introduce new format variants for JATS writer (#6014, Albert Krewinkel):

• jats_archiving for the “Archiving and Interchange Tag Set”,
• jats_publishing for the “Journal Publishing Tag Set”, and
• jats_articleauthoring for the “Article Authoring Tag Set.”

The jats output format is now an alias for jats_archiving. The module Text.Pandoc.Writers.JATS now exports writeJatsArchiving, writeJatsPublishing, and writeJatsArticleAuthoring, as well as the legacy writeJATS [API change].

--defaults: Support bibliography and csl fields. Move addMeta from Text.Pandoc.App.CommandLineOptions to Text.Pandoc.App.Opt (internal change).

Add timing info for filters in --verbose mode (#6112). When verbose mode is specified (verbosity == INFO), print a notice when running a filter and when a filter completes (including timing).

LaTeX reader:

• Allow & in LaTeX citation keys (#6110).
• Improve caption and label parsing.
• Don’t emit empty Span elements for labels.
• Put tables with labels in a surrounding Div.
• Resolve \ref to table numbers (#6137).
• Skip comments in more places where this is needed (#6114).
• Allow beamer overlays for all commands in all raw tex (#6043). This affects parsing of raw tex in LaTeX and in Markdown and other formats.
• Improve parsing of raw environments (#6034). If parsing fails in a raw environment (e.g. due to special characters like unescaped _), try again as a verbatim environment, which is less sensitive to special characters. This allows us to capture special environments that change catcodes as raw tex when -f latex+raw_tex is used.

RST reader:

• Add highlight directive (#6140, Lucas Escot).

MediaWiki writer:

• Prevent triple [[[ which confuses MediaWiki (#6119).

HTML reader:

• Don’t parse data-id as id attribute. And similarly don’t parse any data-X as X when X is a valid HTML attribute.

Org reader:

• Simplify parsing of sub- and superscripts (#6127, Albert Krewinkel). Speeds up parsing of single-word, markup-less sub- and superscripts.

LaTeX writer:

• Group biblatex citations even with prefix and suffix (#5849, Ethan Riley). Previously biblatex citations were only grouped if there was no prefix. This patch allows them to be grouped in subgroups split by prefixes and suffixes, which allows better citation sorting.
• Fix regression in handling of columns in beamer slides (#6033). Columns in title slides were causing problems with slide division.
• Fix duplicate frame classes in LaTeX/Beamer output (#6107).

HTML writer:

• Fix duplicate attributes on headings (#6062), regression from 2.7.x.
• Fix --number-offset with HTML TOC. Eventually it would be worth adding a parameter to makeSections so this could be done at that level; then it would also affect other writers that construct TOC manually.
• reveal.js: restore old behavior for 2D nesting (#6032). The fix to #6030 actually changed behavior, so that the 2D nesting occurred at slide level N-1 and N, instead of at the top-level section. This commit restores the v2.7.3 behavior. If there are more than 2 levels, the top level is horizontal and the rest are collapsed to vertical.
• reveal.js: ensure that pauses work even in title slides (#5819).

Markdown writer:

• Fix regression: spurious dots in markdown_mmd metadata output (#6133).

Docx writer:

• Fix regression with Compact style on tight lists (#6072). Starting in 2.8, the docx writer no longer distinguishes between tight and loose lists, since the Compact style is omitted. This is a side-effect of the fix to #5670, as explained in the changelog. This patch fixes the problem by extending the exception currently offered to Plain blocks inside tables to Plain blocks inside list items.

Jira writer:

• Fix output of table headers (Albert Krewinkel, #6035).

Add Text.Pandoc.Image with unexported svgToPng.

Text.Pandoc.XML: Export html5Attributes, html4Attributes, rdfaAttributes (formerly unexported in Text.Pandoc.Writers.HTML). [API change]

Text.Pandoc.Shared: Export a new function findM (#6125, Joseph C. Sible).

Text.Pandoc.Logging: Add RunningFilter, FilterCompleted constructors to LogMessage [API change].

Text.Pandoc.CSV: fix bug in CSV parser; previously an extra blank record would sometimes be inserted at the end.

LaTeX template: add space option to xeCJK with PassOptionsToPackage (#6002). Otherwise we can get a clash with documentclasses that already load the package.

Lua filters:

• Allow filtering of element lists (#6038, Albert Krewinkel). Lists of Inline and Block elements can now be filtered via Inlines and Blocks functions, respectively. This is helpful if a filter conversion depends on the order of elements rather than a single element. For example, the following filter can be used to remove all spaces before a citation:

  function isSpaceBeforeCite (spc, cite)
    return spc and spc.t == 'Space'
     and cite and cite.t == 'Cite'
  end
  
  function Inlines (inlines)
    for i = #inlines-1,1,-1 do
      if isSpaceBeforeCite(inlines[i], inlines[i+1]) then
        inlines:remove(i)
      end
    end
    return inlines
  end
  

• Add methods insert, remove, and sort to pandoc.List (Albert Krewinkel). Example of use:

  local numbers = pandoc.List {2, 3, 1}
  numbers:sort()     -- numbers is now {1, 2, 3}
  

• Make pandoc.List a callable constructor (Albert Krewinkel). It is now possible to construct a new List via pandoc.List() instead of pandoc.List:new().

• Add tests for pandoc.List module (Albert Krewinkel).

Text.Pandoc.App.CommandLineOptions: Change setVariable to use Text instead of String. This avoids some unnecessary unpacking.

Use versioned directory for windows release zipfile. Also remove old make-windows-installer.bat, superseded by GitHub actions workflow, and modify pandoc.wxs for new paths.

Extensive code cleanup (#6141, #6128, #6129, #6130, #6123, #6105, 6102, #6117, #6124, #6115, #6116, #6111, Joseph C. Sible).

Fix hlint warnings (Albert Krewinkel).

Use latest doclayout, doctemplates (#6031). The new version of doclayout fixes a memory leak that affected --include-in-header with large files (and possibly other cases involving extremely long lines).

Use latest texmath.

Use latest skylighting and fix test suite (#6086).

sample.lua: Fix typo in descriptive comments (#6136, Caleb Maclennan). Fix typo in error message (#6135).

Add Docker and GH Actions instructions/links to INSTALL.md.

Update filter documentation (#6065). Improve cabal v2 instructions. Remove example using pandoc API directly (we have other docs for that and it was outdated).

Lua filter docs:

• Cross-link constructors and types (Albert Krewinkel). Thanks to @bpj for the idea.
• Sort pandoc.List methods alphabetically (Albert Krewinkel).
• Unify, fix anchors and internal links (#6061, Albert Krewinkel). Links and anchors now follow consistent conventions, like lowercase-only anchor names. This breaks some links to specific sections in the document, but will make it much easier to link documentation in the future.
• Clarify filter function execution order (#6059, Albert Krewinkel).

In docs, update URLs and use https: wherever possible (#6090, Salim B).