Process ANSI escape code in non-HTML format for Jupyter errors

[lballabio]

Bug description

When using the PDF format for a book project, the generated LaTeX file contains ANSI color characters that cause the document compilation to fail.

A search brings up #7813, which looks like it should fix the problem, but that's not the case.

Steps to reproduce

Here is a minimal YML configuration:

project:
  type: book

execute:
  enabled: true

book:
  title: "A reproducible example"
  chapters:
    - index.qmd

format:
  pdf:
    documentclass: book

and here is the sample input file:

# An example

This outputs an ANSI-colorized traceback:

```{python}
# | error: true

1 + "2"
```

Expected behavior

ANSI color characters should be stripped or translated so that the LaTeX file can compile and produce a PDF document.

Actual behavior

The LaTeX compilation fails wih the following error:

ERROR: 
compilation failed- error
Text line contains an invalid character.
l.220 ^^[
         [0;31m-------------------------------------------------------------... 

Your environment

Mac OS Sonoma 14.5, Python 3.12.4, a fresh virtual environment with only Jupyter and its dependencies installed.

Quarto check output

Quarto 1.5.55
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.2.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
      Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.5.55
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: v2024.07.03
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /Users/lballabio/Library/TinyTeX/bin/universal-darwin
      Version: 2024

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.12.4
      Path: /Users/lballabio/Downloads/test/.venv/bin/python3
      Jupyter: 5.7.2
      Kernels: python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

[prs513rosewood]

I just encountered the same error with the following install (on openSUSE Tumbleweed):

Quarto 1.5.55
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.2.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
      Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.5.55
      Path: /opt/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: Installation From Path
      Path: /opt/texlive/bin/x86_64-linux
      Version: 2024

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.11.9
      Path: /home/$USER/.local/share/venv/discret/bin/python3
      Jupyter: 5.7.1
      Kernels: python3, python3.11

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

[cderv]

It seems indeed #7813 is not enough. It will handle only some handling for html output content having ANSI Escape

quarto-cli/src/core/jupyter/jupyter.ts

Lines 1790 to 1803 in 0d87b08

	async function mdOutputError(
	output: JupyterOutputError,
	options: JupyterToMarkdownOptions,
	) {
	const traceback = output.traceback.join("\n");
	if (
	!options.toHtml ||
	(!hasAnsiEscapeCodes(output.evalue) && !hasAnsiEscapeCodes(traceback))
	) {
	if (output.traceback.length > 0) {
	return mdCodeOutput([
	output.ename + ": " + output.evalue + "\n" + traceback,
	]);
	} else {

This means ANSI will be kept in PDF output by Quarto. Here is the .md

# An example

This outputs an ANSI-colorized traceback:

::: {.cell execution_count=1}
``` {.python .cell-code}
1 + "2"
```

::: {.cell-output .cell-output-error}
```
TypeError: unsupported operand type(s) for +: 'int' and 'str'
[1;31m---------------------------------------------------------------------------[0m
[1;31mTypeError[0m                                 Traceback (most recent call last)
Cell [1;32mIn[1], line 1[0m
[1;32m----> 1[0m [38;5;241;43m1[39;49m[43m [49m[38;5;241;43m+[39;49m[43m [49m[38;5;124;43m"[39;49m[38;5;124;43m2[39;49m[38;5;124;43m"[39;49m

[1;31mTypeError[0m: unsupported operand type(s) for +: 'int' and 'str'
```
:::
:::

So ANSI are not stripped nor handle in Jupyter output when inside non-HTML format. If we can, we should probably try to strip them out.

Otherwise, we could need a way to configure Jupyter execution for Python cells to not use ANSI in output when we know we'll be producing a non-HTML format.

[cderv]

Prior and related work / ideas:

• Suggestion in When an error is rendered in the output, the ascii escape codes should be stripped or processed to HTML #4204 (comment) to have a new option to control this behavior (e.g. process-ansi-escapes: strip|yes|no)

• Knitr engine related issue: knitr: process ANSI color codes #5746. Usually in R ecosystem, the functions in packages are themselves opting out colored output when inside knitting context. Though, in this specific issue, a package did not have handling for it. So stripping it would have worked.

So this is another case for handling ANSI in all our engine

[cscheid]

This is fixed in 1.6: