ToC does not play well when sections are numbered

[astamm]

Bug description

When both toc: true and number-sections: true are set, the section number is added twice in the ToC, resulting in dead links when one clicks on them in the ToC.
Small reproducible example (running under quarto v0.9.566 and RStudio 2022.02.3 Build 492 on macOS Monterey)

---
title: "toc-gfm-numbered-sections"
author: "Aymeric Stamm"
format:
  gfm: default
  html: default
editor: visual
toc: true
number-sections: true
---

## H1

### Sub-H1

## H2

Checklist

• formatted your issue so it is easier for us to read?
• included a minimal, self-contained, and reproducible example?
• documented the quarto version you're running, by providing the output produced by quarto check in a terminal in your issue?
• documented the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• documented which operating system you're running? If on Linux, please provide the specific distribution as well.
• upgraded to the latest version, including your versions of R, the RStudio IDE, and relevant R packages?

[cderv]

Hi @astamm can you reformat your issue correctly ? https://yihui.org/issue/#please-format-your-issue-correctly

I assume you used level-1 headers in the doc but we don't have them, and I assume code chunks have backticks too.
Also I assume the issue is with gfm format, right ? it is working well for HTML ?

Small reproducible example I have

---
title: "toc-gfm-numbered-sections"
format: gfm
toc: true
number-sections: true
---

# H 1

## Sub 1

# H2

will output

toc-gfm-numbered-sections
================

-   <a href="#h-1" id="toc-h-1"><span class="toc-section-number">1</span>
    <span class="header-section-number">1</span>. H 1</a>
    -   <a href="#sub-1" id="toc-sub-1"><span
        class="toc-section-number">1.1</span> <span
        class="header-section-number">1.1</span> Sub 1</a>
-   <a href="#h2" id="toc-h2"><span class="toc-section-number">2</span>
    <span class="header-section-number">2</span>. H2</a>

# <span class="header-section-number">1</span>. H 1

## <span class="header-section-number">1.1</span> Sub 1

# <span class="header-section-number">2</span>. H2

Change in Pandoc 2.18 regarding but probably not directly related

For the record, there was a change in Pandoc 2.18 regarding adding id to TOC elements (jgm/pandoc#7913) and this had the effect on gfm formats to use html for TOC with id attributes.
So we may need to handle that correctly.

This is what we got with bare Pandoc

❯ quarto pandoc --to gfm -f markdown --toc -s
# Head

Content
^Z
-   <a href="#head" id="toc-head">Head</a>

# Head

Content

❯ quarto pandoc --to gfm -f markdown --toc -s --number-sections
# Head

content
^Z
-   <a href="#head" id="toc-head"><span class="toc-section-number">1</span>
    Head</a>

# Head

content

Or maybe it is not related directly related, but wanted to share as it impacted us on rmarkdown

[cderv]

In fact it may just be something related to our number section handling

We are adding the class using a Span

quarto-cli/src/resources/filters/crossref/sections.lua

Lines 59 to 62 in 2449933

	el.content:insert(1, pandoc.Span(
	stringToInlines(section),
	pandoc.Attr("", { "header-section-number"})
	))

before

1: Str ". "
2: Str "H"
3: Space
4: Str "1"

after

1: Span ("",["header-section-number"],[]) [Str "1"]
2: Str ". "
3: Str "H"
4: Space
5: Str "1"

So it gets doubled down by Pandoc which will also add a Span for the numbering section even when numbering is not working on headers https://github.com/jgm/pandoc/blob/f3cfeba3efc1a94b02040e5b206123f58d08313a/src/Text/Pandoc/Writers/Shared.hs#L444-L447

We need probably need to post process the output to remove what Pandoc is adding (as it is one for HTML document it seems

quarto-cli/src/format/html/format-html.ts

Lines 367 to 375 in 23d0feb

	// remove toc-section-number if we have provided our own section numbers
	const headerSections = doc.querySelectorAll(".header-section-number");
	for (let i = 0; i < headerSections.length; i++) {
	const secNumber = headerSections[i] as Element;
	const prevElement = secNumber.previousElementSibling;
	if (prevElement && prevElement.classList.contains("toc-section-number")) {
	prevElement.remove();
	}
	}

)

Is there any markdown output post processor yet in Quarto ?

[cderv]

But I wonder if it cannot be consider an issue to add a Span with class toc-section-number when --number-sections flag is passed, but the Writer won't be adding any number on the section (like with Markdown writer)

This is what we got with Pandoc 2.17.1.1 before 2.18 change for gfm

> pandoc --to gfm -f markdown --toc -s
# Head

Content
^Z
-   [Head](#head)

# Head

Content

❯ pandoc --to gfm -f markdown --toc -s --number-sections
# Head

content
^Z
-   [<span class="toc-section-number">1</span> Head](#head)

# Head

Content

It will just number the TOC item, not the actual header.

[jjallaire]

There are a handful of underlying issues here. Here's the test document I'll be basing examples on:

---
title: "toc-gfm-numbered-sections"
---

# H

## Sub

# H2

As of Pandoc 2.18 the table of contents in gfm output now uses raw HTML as follows:

pandoc toc.md --to gfm --standalone --toc

-   <a href="#h" id="toc-h">H</a>
    -   <a href="#sub" id="toc-sub">Sub</a>
-   <a href="#h2" id="toc-h2">H2</a>

# H

## Sub

# H2

If this isn't considered desirable then an issue should be filed with Pandoc to make this optional and/or disabled by default (not sure what their position on that would be though).

Numbering of section headings is not formally supported by Pandoc for markdown formats (see the docs here on what formats are supported: https://pandoc.org/MANUAL.html#option--number-sections). However, if you add --number-sections along with --toc for gfm output Pandoc will number the TOC but will not number the section headings:

quarto pandoc toc.md--to gfm --standalone --toc --number-sections

-   <a href="#h" id="toc-h"><span class="toc-section-number">1</span> H</a>
    -   <a href="#sub" id="toc-sub"><span class="toc-section-number">1.1</span>
        Sub</a>
-   <a href="#h2" id="toc-h2"><span class="toc-section-number">2</span>
    H2</a>

# H

## Sub

# H2

The problem reported here (the doubling of the section numbers in the TOC) results from Quarto attempting to number the heading manually, then subsequently Pandoc doing its own numbering in the TOC (and picking up our numbers). I think given this behavior we are out of school doing manual numbering for markdown formats, so I've disabled that (now we behave identically to pandoc). Here is that change: 8875ec6

I think if you want heading numbering to work for Pandoc gfm output that may be a worthwhile thing requesting on the Pandoc repo. Once this works then presumably the doubling of section numbers in the TOC would go away (as there would be an expectation in the Pandoc writer that headings already have a number so an additional one wouldn't be added when building the TOC).

[cderv]

If this isn't considered desirable then an issue should be filed with Pandoc to make this optional and/or disabled by default (not sure what their position on that would be though).

Regarding the raw HTML for TOC, as mentioned this is due to the change in jgm/pandoc#7913 which was intentional. It adds id on TOC element and GFM output can handle that only by writing raw HTML, as it does not support attributes in markdown syntax. I had raised a question about this change in the past which remained unanswered https://groups.google.com/g/pandoc-discuss/c/gQZrKunCvB4/m/j7177M-3BwAJ

I think I will open an issue to bring attention to this (done in jgm/pandoc#8131). I'll report back. cc @tarleb as you may have context.

[astamm]

Hi @astamm can you reformat your issue correctly ? https://yihui.org/issue/#please-format-your-issue-correctly

Thanks for looking into this. @cderv I edited the original post to format the issue correctly.

[cderv]

Just a heads up that jgm/pandoc#8131 has been merged, so this will probably be better in next Pandoc version. This is to follow.