Jupyter engine with Julia kernel does select text/latex output for `format: md` which leads to no cell output in rendered document

[sethaxen]

On Quarto v1.3.433, I tried rendering a .qmd file with a cell block that outputs a DataFrames.DataFrame object in Julia. When rendering an HTML or PDF file, the DataFrame is rendered as a table, but when rendering to markdown, it is not rendered at all. There's a minimal repo demonstrating this here: https://github.com/sethaxen/quarto_dataframes_jl_demo

[cderv]

I can reproduce this.

Somehow, when the first table is outputing a LaTeX table in the intermediate .md file pass to Pandoc

The following shows a table when rendered to PDF or HTML but nothing when rendered to markdown

::: {.cell execution_count=2}
``` {.julia .cell-code}
using DataFrames
df = DataFrame(:x => randn(10), :y => randn(10))
```

::: {.cell-output .cell-output-display execution_count=3}
\begin{tabular}{r|cc}
	& x & y\\
	\hline
	& Float64 & Float64\\
	\hline
	1 & 0.0387945 & 0.728907 \\
	2 & 0.0640609 & -0.0620356 \\
	3 & 0.0834239 & 0.209232 \\
	4 & 1.46305 & -0.305209 \\
	5 & 0.883393 & 0.61293 \\
	6 & -1.27005 & 0.167557 \\
	7 & 0.469263 & 0.873955 \\
	8 & 0.398598 & -0.243128 \\
	9 & -0.210688 & 0.131962 \\
	10 & 1.57329 & -1.30073 \\
\end{tabular}

:::
:::

This is why it gets ignored when converting to a Markdown output with Pandoc... This is suprising 🤔

[cderv]

So this the information we get from the Jupyter rendering

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "a6f5d7b4",
   "metadata": {},
   "source": [
    "---\n",
    "title: \"DataFrames rendering test\"\n",
    "keep-md: true\n",
    "keep-ipynb: true\n",
    "format: \n",
    "  md:\n",
    "    output-file: test-md.md\n",
    "  html: default\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f96792b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m\u001b[1m  Activating\u001b[22m\u001b[39m project at `C:\\Users\\chris\\Documents\\DEV_OTHER\\DEMOS\\test-quarto`\n"
     ]
    }
   ],
   "source": [
    "#| output: false\n",
    "#| echo: false\n",
    "using Pkg\n",
    "Pkg.activate(\".\")\n",
    "Pkg.instantiate()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f64c9a77",
   "metadata": {},
   "source": [
    "The following shows a table when rendered to PDF or HTML but nothing when rendered to markdown\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f3ee4485",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div style = \"float: left;\"><span>10×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">x</th><th style = \"text-align: left;\">y</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">0.662702</td><td style = \"text-align: right;\">0.451484</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">-1.54828</td><td style = \"text-align: right;\">-1.25003</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">-0.0443339</td><td style = \"text-align: right;\">-1.39289</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: right;\">-1.55107</td><td style = \"text-align: right;\">-1.9189</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: right;\">0.0755979</td><td style = \"text-align: right;\">-0.58877</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">6</td><td style = \"text-align: right;\">0.748819</td><td style = \"text-align: right;\">-1.87185</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">7</td><td style = \"text-align: right;\">-0.281013</td><td style = \"text-align: right;\">1.54457</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">8</td><td style = \"text-align: right;\">-0.680722</td><td style = \"text-align: right;\">0.664824</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">9</td><td style = \"text-align: right;\">-1.20119</td><td style = \"text-align: right;\">2.64156</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">10</td><td style = \"text-align: right;\">-0.161478</td><td style = \"text-align: right;\">1.11157</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& x & y\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.662702 & 0.451484 \\\\\n",
       "\t2 & -1.54828 & -1.25003 \\\\\n",
       "\t3 & -0.0443339 & -1.39289 \\\\\n",
       "\t4 & -1.55107 & -1.9189 \\\\\n",
       "\t5 & 0.0755979 & -0.58877 \\\\\n",
       "\t6 & 0.748819 & -1.87185 \\\\\n",
       "\t7 & -0.281013 & 1.54457 \\\\\n",
       "\t8 & -0.680722 & 0.664824 \\\\\n",
       "\t9 & -1.20119 & 2.64156 \\\\\n",
       "\t10 & -0.161478 & 1.11157 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m10×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m x          \u001b[0m\u001b[1m y         \u001b[0m\n",
       "     │\u001b[90m Float64    \u001b[0m\u001b[90m Float64   \u001b[0m\n",
       "─────┼───────────────────────\n",
       "   1 │  0.662702    0.451484\n",
       "   2 │ -1.54828    -1.25003\n",
       "   3 │ -0.0443339  -1.39289\n",
       "   4 │ -1.55107    -1.9189\n",
       "   5 │  0.0755979  -0.58877\n",
       "   6 │  0.748819   -1.87185\n",
       "   7 │ -0.281013    1.54457\n",
       "   8 │ -0.680722    0.664824\n",
       "   9 │ -1.20119     2.64156\n",
       "  10 │ -0.161478    1.11157"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using DataFrames\n",
    "df = DataFrame(:x => randn(10), :y => randn(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f6bb391",
   "metadata": {},
   "source": [
    "But there is a custom `show` method for HTML outputs:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6805350a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"<div><div style = \\\"float: left;\\\"><span>10×2 DataFrame</span></div><div style = \\\"clear: both;\\\"></div></div><div class = \\\"data-frame\\\" style = \\\"overflow-x: scroll;\\\"><table class = \\\"data-frame\\\" style = \\\"margin-bottom: 6px;\\\"><thead><tr class = \\\"header\\\"><th class = \\\"rowNumber\" ⋯ 1943 bytes ⋯ \"ht;\\\">-1.20119</td><td style = \\\"text-align: right;\\\">2.64156</td></tr><tr><td class = \\\"rowNumber\\\" style = \\\"font-weight: bold; text-align: right;\\\">10</td><td style = \\\"text-align: right;\\\">-0.161478</td><td style = \\\"text-align: right;\\\">1.11157</td></tr></tbody></table></div>\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sprint(show, \"text/html\", df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62ee20cf",
   "metadata": {},
   "source": [
    "In fact, if we just wrap the `DataFrame` with an object that has a custom HTML `show` method, it renders in markdown just fine:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8c684f8f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div style = \"float: left;\"><span>10×2 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">x</th><th style = \"text-align: left;\">y</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">0.662702</td><td style = \"text-align: right;\">0.451484</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">-1.54828</td><td style = \"text-align: right;\">-1.25003</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">-0.0443339</td><td style = \"text-align: right;\">-1.39289</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: right;\">-1.55107</td><td style = \"text-align: right;\">-1.9189</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: right;\">0.0755979</td><td style = \"text-align: right;\">-0.58877</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">6</td><td style = \"text-align: right;\">0.748819</td><td style = \"text-align: right;\">-1.87185</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">7</td><td style = \"text-align: right;\">-0.281013</td><td style = \"text-align: right;\">1.54457</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">8</td><td style = \"text-align: right;\">-0.680722</td><td style = \"text-align: right;\">0.664824</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">9</td><td style = \"text-align: right;\">-1.20119</td><td style = \"text-align: right;\">2.64156</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">10</td><td style = \"text-align: right;\">-0.161478</td><td style = \"text-align: right;\">1.11157</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "DataFrameWrapper(\u001b[1m10×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m x          \u001b[0m\u001b[1m y         \u001b[0m\n",
       "     │\u001b[90m Float64    \u001b[0m\u001b[90m Float64   \u001b[0m\n",
       "─────┼───────────────────────\n",
       "   1 │  0.662702    0.451484\n",
       "   2 │ -1.54828    -1.25003\n",
       "   3 │ -0.0443339  -1.39289\n",
       "   4 │ -1.55107    -1.9189\n",
       "   5 │  0.0755979  -0.58877\n",
       "   6 │  0.748819   -1.87185\n",
       "   7 │ -0.281013    1.54457\n",
       "   8 │ -0.680722    0.664824\n",
       "   9 │ -1.20119     2.64156\n",
       "  10 │ -0.161478    1.11157)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "struct DataFrameWrapper\n",
    "    df\n",
    "end\n",
    "\n",
    "Base.show(io::IO, mime::MIME\"text/html\", w::DataFrameWrapper) = show(io, mime, w.df)\n",
    "\n",
    "DataFrameWrapper(df)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.7.3",
   "language": "julia",
   "name": "julia-1.7"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}

We can see that the first cell will offer a three representation: html, plain and latex. No text/markdown output.

And then we have specific process that handle some raw LaTeX in that case are they are expected to be Math.

Here is what happens

We are getting the cell content from the above in

quarto-cli/src/core/jupyter/jupyter.ts

Lines 1166 to 1170 in 7277a8f

	async function mdFromCodeCell(
	cell: JupyterCellWithOptions,
	cellIndex: number,
	options: JupyterToMarkdownOptions,
	) {

In that process we do the following with the outputs

quarto-cli/src/core/jupyter/jupyter.ts

Lines 1193 to 1199 in 7277a8f

	}).map((output) => {
	// convert text/latex math to markdown as appropriate
	if (!options.toLatex && isDisplayData(output) && output.data[kTextLatex]) {
	return displayDataWithMarkdownMath(output);
	} else {
	return output;
	}

So we are calling displayDataWithMarkdownMath() on this toLatex output, and in there we select the output.data["text/latex"] explicitely

quarto-cli/src/core/jupyter/display-data.ts

Lines 122 to 132 in 7277a8f

	export function displayDataWithMarkdownMath(output: JupyterOutputDisplayData) {
	if (Array.isArray(output.data[kTextLatex]) && !output.data[kTextMarkdown]) {
	const latex = output.data[kTextLatex] as string[];
	if (displayDataLatexIsMath(latex)) {
	output = ld.cloneDeep(output);
	output.data[kTextMarkdown] = output.data[kTextLatex];
	return output;
	}
	}
	return output;
	}

For the last code cell you have, there is only text/html and text/plain, so there is no LaTeX table inserted and so you see the right table result.

I don't know enough about Jupyter, Julia and nbConvert to understand what happens. It seems though that

df = DataFrame(:x => randn(10), :y => randn(10))

won't output text/markdown to include.

Though I guess we should probably not choose the text/latex part, but either the plain or html.

To note that if we do force HTML then we correctly select the HTML version of the results

format: 
   md:
      prefer-html: true

@cscheid @dragonstyle you know Quarto internals for Jupyter a bit better than me. Do you have more insight on what we should do here ? if we do need to do something unless we consider this to be a Julia DataFrames output problem. But really surprising that we select the text/latex data for this output.

Hope it helps

[sethaxen]

To note that if we do force HTML then we correctly select the HTML version of the results

format: 
   md:
      prefer-html: true

Is there a way to force this for a single cell instead of the whole document?

[cderv]

Is there a way to force this for a single cell instead of the whole document?

Unfortunately no. It was not meant for this specific cell usage. IMO this is a side effect that it helps with this issue. It should be solved differently by correctly handling the Julia outputs for markdown format.

[sethaxen]

Hi, any updates on what the solution for this might be?

[mcanouil]

As you can see, no new informations.
When there is, the team will communicate.