Julia Makie rendering plots to incorrect IO with display and PDF output format

[sceptri]

Bug description

When using layout-* option with Julia and Makie.jl, only for HTML output format the layout-plots are rendered correctly. (because the layout-* option, one needs to pass the plots using display instead of just fig at the end)

For PDF output format, Julia opens separate windows containing PNG images of the layout-figures. Those images are then not included in the final PDF.

Steps to reproduce

This example should demonstrate the bug

---
title: "Broken PDF"
author: "Štěpán Zapadlo"
date: last-modified

format:
  html:
    code-fold: false
  pdf:
    documentclass: scrartcl
  
jupyter: julia-1.9
---

This is a test `.qmd` to show the bug. Add Julia packages 
```{julia}
using Pkg; Pkg.add("CairoMakie")
using CairoMakie
```
this also works fine
```{julia}
fig = Figure(); ax = Axis(fig[1,1]);

lines!(ax, 1:10, 1:10)

fig
```
and now a layout-plot, which works only in HTML. When rendering PDF, the display will open (on my computer) to PNG images with the figures that do not appear in the PDF
```{julia}
#| layout-ncol: 2

fig1 = Figure(); ax1 = Axis(fig1[1,1]);
fig2 = Figure(); ax2 = Axis(fig2[1,1]);

lines!(ax1, 1:10, (1:10).^2)
lines!(ax2, 1:10, (1:10).^(1/2))

display(fig1); display(fig2)
nothing
```

Expected behavior

In the PDF, the layout plots will be present (thus rendered to the correct IO)

Actual behavior

The images are not in the PDF, but rendered to a different (probably the same as REPL) output

Your environment

• terminal command: quarto render <file>.qmd --to pdf
• Quarto: 1.3.361
• OS: Linux Open Suse Tumbleweed
• Julia: 1.9.3

Quarto check output

[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.1: OK
      Dart Sass version 1.55.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.3.361
      Path: /home/sceptri/Apps/quarto-1.3.361/bin

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.11.5
      Path: /usr/bin/python3
      Jupyter: 5.3.1
      Kernels: python3, julia-1.6, julia-1.8, julia-1.9, jupyter_matlab_kernel, python3.11, python3.9, python3.10

(|) Checking Jupyter engine render....0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.3.1
      Path: /usr/lib64/R
      LibPaths:
        - /home/sceptri/R/x86_64-suse-linux-gnu-library/4.3
        - /usr/lib64/R/library
      knitr: 1.44
      rmarkdown: (None)

      The rmarkdown package is not available in this R installation.
      Install with install.packages("rmarkdown")

[cscheid]

and now a layout-plot, which works only in HTML. When rendering PDF, the display will open (on my computer) to PNG images with the figures that do not appear in the PDF

Can you type quarto convert FILE.qmd and then execute this file on jupyter lab? What's happening is probably that Makie needs to be configured to emit these as Jupyter output.

[cscheid]

I also don't understand why you need display() to work with layout. I think that's part of the problem. I'm far from a Julia or Makie expert, but I have to assume it is possible for the library to emit more than one Plot output in a single cell.

[cscheid]

In any case, this isn't really a quarto bug; if display() opens a new window, there's nothing we can do about it.

I couldn't figure out how to convince Makie to generate two plots in a single cell, so I just did this instead:

::: {layout-ncol="2"}

```{julia}
#| echo: false
fig1 = Figure(); ax1 = Axis(fig1[1,1]);
lines!(ax1, 1:10, (1:10).^2)
fig1
```

```{julia}
#| echo: false
fig2 = Figure(); ax2 = Axis(fig2[1,1]);
lines!(ax2, 1:10, (1:10).^(1/2))
fig2
```

:::

That works:

[sceptri]

Honestly, I would argue that this is a Quarto bug. Using display for Julia subfigures is recommended in the docs. What's more, I would expect both HTML and PDF outputs to be (reasonably) equivalent. More so when considering this is the recommended approach.

Although one could use the solution provided by @cscheid, the subfigure captions would be all wrong. I know that I can solve this using multiple axes in the Makie figure, but I hoped to use the responsive capabilities of Quarto.

So my question is why does it work in HTML and not in PDF... Is there a difference in how the notebook is processed/rendered? And if so, where to look?

What I discovered is that when one uses PDF output and Makie.inline!(true) to force the inline output of Makie figures, it raises a warning that there is no display to show the figure... So I would bet it is an issue of Makie configuration.

Warning:
  Makie.inline!(do_inline) was set to true, but we didn't detect a display that can show
  so we aren't inlining the plot and try to show the plot in a window.
  If this wasn't set on purpose, call `Makie.inline!()` to restore the default.
@ Makie ~/.julia/packages/Makie/Ylh0P/src/display.jl:152

Also, what is the difference between returning fig in the execution cell and using display? Can't some other closer alternative be used? The display function documentation hints at show(stdout, "text/plain", fig), but I couldn't quite get it to work.

The .qmd converted into .ipynb runs just fine and puts those figures into a column.

(I included both .qmd and .ipynb as a .zip

index.zip

[cscheid]

Honestly, I would argue that this is a Quarto bug. Using display for Julia subfigures is recommended in the docs.

That's the recommendation for Plots: see the code example that follows. You're using Makie. I don't know the internals of either library, but I would be wary of expecting them to behave internally.

Although one could use the solution provided by @cscheid, the subfigure captions would be all wrong.

Your message didn't specify you wanted captions and subcaptions. They work just fine with 1.4's new syntax:

::: {#fig-1 layout-ncol="2"}

```{julia}
#| echo: false
#| label: fig-1a
#| fig-cap: Subcaption 1
fig1 = Figure(); ax1 = Axis(fig1[1,1]);
lines!(ax1, 1:10, (1:10).^2)
fig1
```

```{julia}
#| echo: false
#| label: fig-1b
#| fig-cap: Subcaption 2
fig2 = Figure(); ax2 = Axis(fig2[1,1]);
lines!(ax2, 1:10, (1:10).^(1/2))
fig2
```

Overall caption

:::

See @fig-1, @fig-1a, and @fig-1b.

This syntax exists precisely so that we can guarantee uniform behavior under third-party libraries that might not emit multiple outputs in a single cell. You can also use it to create figures whose contents are code, or text, or an iframe, etc.

What I discovered is that when one uses PDF output and Makie.inline!(true) to force the inline output of Makie figures, it raises a warning that there is no display to show the figure... So I would bet it is an issue of Makie configuration.

We try to guess some common library setups and create the right configurations, but I think you can reasonably agree that it's a generally impossible task to guess at every possible library and configure them automatically. I agree with you that this is an issue of Makie configuration, but that means that you can configure that on your side.

So my question is why does it work in HTML and not in PDF... Is there a difference in how the notebook is processed/rendered? And if so, where to look?

Also, what is the difference between returning fig in the execution cell and using display?

If I understand your question correctly, that's not exactly a Quarto question, that's a Jupyter+Makie question.

To investigate these issues, it's helpful to look at the intermediate notebook file generated by nbclient, the library we use to execute Jupyter notebooks. You can get that output by adding keep-ipynb: true to your document metadata.

[sceptri]

Your message didn't specify you wanted captions and subcaptions. They work just fine with 1.4's new syntax:

Yeah, that's true, sorry! I didn't consider them relevant or necessary for the MWE. And it's totally awesome that it can done via multiple execution cells in Quarto 1.4! I imagine it must be very hard to ensure compatibility across multiple libraries and languages. Well done!

If I understand your question correctly, that's not exactly a Quarto question, that's a Jupyter+Makie question.

Yeah, that's true I guess. Sorry!

To investigate these issues, it's helpful to look at the intermediate notebook file generated by nbclient, the library we use to execute Jupyter notebooks. You can get that output by adding keep-ipynb: true to your document metadata.

I might do that, thank you. It is always quite complicated to see the entire pipeline/process at first with new software.

All in all, I hope I wasn't too entitled or anything. I am very thankful for this level of immediate support/bugfix and for the awesome work, you (all) do at Quarto. Thanks!

[cscheid]

I have good news! This is, at least partially, a thing we can fix with Quarto.

Here's a direct workaround for you:

```{julia}
CairoMakie.activate!(type = "png")
```

We try to set figure formats in libraries, including CairoMakie, and in PDF format, this is clearly interacting badly with display(). The line above should make your original code work, even though it'll use PNG for the image outputs (instead of a vector format like PDF). I don't know how to make CairoMakie's display emit a PDF to the notebook, though.

I think what we'll do is we won't change the default figure format to PDF in CairoMakie, since that's clearly going to break things.

[sceptri]

This is awesome! Thank you for your hard work and dedication, @cscheid!

[sceptri]

Just as a last thing to add, setting type to SVG (CairoMakie.activate!(type = "svg")) also works. So we can actually keep a vector graphics format.

[cscheid]

Just as a last thing to add, setting type to SVG (CairoMakie.activate!(type = "svg")) also works. So we can actually keep a vector graphics format.

I wonder if that's wise, though. We recently had a different issue in a different library where rendering to svg caused an out-of-memory error because an upstream library was using a very large number of very small rectangles to render a heatmap.

As quarto becomes more popular and used in broader settings, I unfortunately think that our defaults need to become more conservative, given that support-load ~= p(error) * number-of-users.

[jkrumbiegel]

Warning:
Makie.inline!(do_inline) was set to true, but we didn't detect a display that can show
so we aren't inlining the plot and try to show the plot in a window.
If this wasn't set on purpose, call Makie.inline!() to restore the default.

Could be that this was a bug that I just fixed here MakieOrg/Makie.jl#3403

If the display type that Quarto uses accepts the application/pdf mime type, then it could be that it works with that change. I've actually never tried "inline display" for pdfs because VSCode, Pluto and the like don't support it.