gfm output gets overwritten if other formats are also generated

[mine-cetinkaya-rundel]

Bug description

I have a file with

format: 
  html: default
  pdf: default
  gfm: default

When I run quarto render on it I end up with HTML and PDF files, and the HTML links to a GFM file, but that file doesn't exist.

Your environment

• IDE: RStudio 2023.09.0 Build 380
• OS: Mac 13.5

Quarto check output

Mines-MacBook-Pro:test-website mine$ quarto check
Quarto 1.4.346
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.6: OK
      Dart Sass version 1.55.0: OK
      Deno version 1.33.4: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.4.346
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: v2023.08
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /Users/mine/Library/TinyTeX/bin/universal-darwin
      Version: 2023

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.9.6
      Path: /Library/Developer/CommandLineTools/usr/bin/python3
      Jupyter: 5.3.0
      Kernels: python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.3.1
      Path: /Library/Frameworks/R.framework/Resources
      LibPaths:
        - /Users/mine/Desktop/teaching/Duke/sta113-f23/sta113-f23.github.io/renv/library/R-4.3/aarch64-apple-darwin20
        - /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
      knitr: 1.43
      rmarkdown: 2.24

[✓] Checking Knitr engine render......OK

[cderv]

I believe this is a duplicate of this issue

• Rendering multiple formats not working with markdown variants #1929

Or at least same family as same cause mentioned by @cscheid in #1929 (comment)

This is an unfortunate clash with our intermediate files. It's not a quick fix because a lot of the tools we depend on are the ones doing the overwriting. But you can work around it like so:

---
title: "quarto"
format: 
 commonmark: 
   output-file: out-commonmark.md
 gfm:
   output-file: out-gfm.md
 revealjs: default
---

## Quarto

Current workaround is using output-file, but this means change the gfm output filename for now... not ideal.
Other workaround is using keep-md: true

format: 
  html: default
  pdf: default
  gfm: default
keep-md: true

but gfm needs to be the last format

However, I believe the name conflict may not be the only thing here, because if gfm is the last format listed, the .md file should not be deleted. I believe it is deleted because keep-md is false for the two other format. Doing

format: 
  html:
    keep-md: true
  pdf: 
    keep-md: true
  gfm: default

as the same effect as previous one. So it seems that when rendering to all format, the last keep-md value is not applied.

[cderv]

Additional information from further debug to understand

• Cleanup code is in renderCleanup()
  

  quarto-cli/src/command/render/cleanup.ts

  Lines 43 to 46 in f87976c

  	// cleanup md if necessary
  	if (keepMd && !format.execute[kKeepMd] && keepMd !== output) {
  	removeIfExists(keepMd);
  	}

• When rendering to --to all we pass through this 3 times, and not in the order I though.

  • Last format is called first. This means the gfm format would technically need to be first
  • But in this case, the clash with between gfm format output and html intermediary will happen

  quarto-cli/src/command/render/render-files.ts

  Lines 671 to 683 in f87976c

  	onPostProcess: async (renderedFormats: RenderedFormat[]) => {
  	let completion = renderCompletions.pop();
  	while (completion) {
  	renderedFiles.push(await completion.complete(renderedFormats));
  	completion = renderCompletions.pop();
  	}
  	renderedFiles.reverse();
  	},
  	onComplete: async () => {
  	return {
  	files: await Promise.resolve(renderedFiles),
  	};
  	},

• When gfm is called last, its output index.md is existing, when cleaning step happen. This means the cleaning step for HTML and PDF format really remove the gfm format output.

So I wonder if a fix for this specific issue (before we manage to deal with intermediary conflict in naming) would be

• gfm format should be last
• cleaning step should happen after each rendering and not after all format rendering.

This also means that using keep-md: true in YAML header is also a good solution

[cscheid]

I think the best workaround for now is to use a different output-file name for the gfm output. This is ultimately because quarto doesn't have a mechanism for tracking which formats use which filenames. (This is going to be fixable once we have #6518 taken care of in 1.5)

[cderv]

I do think here keep-md: true is good enough, and allow you to have md file named as input. I believe with gfm you sometime expect specific name like README.md from a .qmd file. So changing output-file would not give you that

Anyhow, both works, and I agree that it may be best to wait for #6518 big work to happen.

Still surprised of the cleaning up after all format renders though, and not between each format. Feels like there could be other conflict, but again #6518 will help.

[cscheid]

I think keep-md: true is only working by accident.

Quarto doesn't know that two separate paths write a .md:

• html and pdf output: .qmd -> .md -> .html
• .qmd -> .md

I wouldn't rely on that.