convert: read the .ipynb converted files as commonmark_x

[amaier-obs]

Bug description

Quarto 1.2.335 on Mac OS X 12.6.1 seems to render a list only, if there is an empty line before the list. So the following quarto file test.qmd will not render the list

---
title: "Introduction to markdown lists"
subtitle: ""
author: Mark Down
date: 2023-01-02
---

## Introduction to lists

This is an example of a list
- first item  
- second item
- third item

To make this example work as expected ( see babelmark ), one has to add an empty line before the list like

This is an example of a list

- first item  
- second item
- third item

One could argue this is just a documentation bug, because it is not mentioned in https://quarto.org/docs/authoring/markdown-basics.html#lists .

However it leads to inconsistent behaviour when rendering iPython notebooks with quarto. The following notebook test.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "title: \"Introduction to markdown lists\"\n",
    "subtitle: \"\"\n",
    "author: Mark Down\n",
    "date: 2023-01-02\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to lists\n",
    "\n",
    "This is an example of a list\n",
    "- first item  \n",
    "- second item\n",
    "- thrid item"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

will render the list correctly when using

jupyter nbconvert --execute --to html test.ipynb

However it will not render the list when using

quarto render test.ipynb --execute

This inconsistent behaviour is annoying for users, because it means one has to change the markdown dialect inside jupyter notebooks when one wants to use quarto for rendering.

I understand quarto is using pandoc under the hood, so this behaviour probably cannot be changed easily. But maybe an option in the preamble could be given when rendering jupyter notebooks, to let quarto select the correct renderer for a given jupyter notebook.

P.S.

❯ quarto check

[✓] Checking Quarto installation......OK
      Version: 1.2.335
      Path: /Applications/quarto/bin

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.10.6
      Path: /Users/<username>/.local/share/virtualenvs/test2-O5T_d_dR/bin/python3
      Jupyter: 5.3.0
      Kernels: python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.2.1
      Path: /opt/homebrew/Cellar/r/4.2.1_2/lib/R
      LibPaths:
        - /opt/homebrew/Cellar/r/4.2.1_2/lib/R/library
      rmarkdown: (None)

      The rmarkdown package is not available in this R installation.
      Install with install.packages("rmarkdown")

Checklist

• Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
• Please format your issue so it is easier for us to read the bug report.
• Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• Please document the operating system you're running. If on Linux, please provide the specific distribution.
• Please provide the output of quarto check so we know which version of quarto and its dependencies you're running.

[cscheid]

To be clear, this is not a quarto bug; that list behavior is part of Pandoc markdown, and we're not going to deviate from a well-specified markdown parser because it becomes impossible to be consistent.

With that said, it would be nice if we could do better. I'm reopening this as an enhancement, and I'm specifically wondering if we could add an mdlint pass or something similar to the resulting markdown, to find potential issues.

With that said, it might make sense for us to eventually move to commonmark as the "blessed" input, or at least to document it better as an option.

(Thanks for the link to babelmark too, that's a very useful resource!)

[amaier-obs]

I believe one of the issues here is that quarto claims

"Quarto can render Jupyter notebooks represented as plain text (.qmd) or as a normal notebook file (.ipynb)." (https://quarto.org/docs/tools/jupyter-lab.html#overview)

But on the other side the documentation notes that

Any valid Pandoc markdown syntax can be included in markdown cells. (https://quarto.org/docs/tools/jupyter-lab.html#markdown-cells)

But a normal jupyter notebook file doesn't use pandoc markdown. It uses marked.js(+ MathJax) which supports

• CommonMark.
• GitHub Flavored Markdown

see https://nbformat.readthedocs.io/en/latest/markup.html#what-flavor-of-markdown-does-the-notebook-format-support

I think it should be made clear in the documentation, that quarto does not support normal jupyter notebooks, but only a special kind of notebook using pandoc flavored markdown, which one cannot render in the default jupyter notebook or jupyter lab interface.

[cscheid]

What we're going to do instead is we'll read the .ipynb converted files as commonmark_x, which means that this will be supported correctly. We'll do that for 1.4 because it is a behavior change and we don't want to risk major regressions at this point in 1.3 release.