Footnotes in documentation generation seem to be broken by default

[fsh]

Description

Running nim doc to produce HTML documentation on something like

## Given a small[#]_ prime number ...
##
## .. [#] 16 or less unsigned bits

errors out with

Error: unhandled exception: number of footnotes and their references don't match: 1 (lines 7) != 0 (lines ) for auto-numbered footnotes [ERecoverableError]

Similarly,

## Here's another[#foo]_
##
## .. [#foo] footnote2

gives a "broken link" warning, footnote is a regular link pointing to nothing; it was (likely) incorrectly interpreted as a Markdown link? Same goes for [1]_ etc.

I didn't analyze deeply but the problem seems to be caused by roSupportMarkdown (default) getting priority and interpreting either case as a pandoc implicit header ref Markdown link.

Here:

Nim/lib/packages/docutils/rst.nim

Line 1714 in 6505bd3

elif roSupportMarkdown in p.s.options and

and here:

Nim/lib/packages/docutils/rst.nim

Line 1590 in 6505bd3

else: # [description=target]

WORKAROUND: there's a doctype pragma to turn off the roSupportMarkdown default, but it seems rather obscure as I had to look at the compiler code to even know about it; it should probably be mentioned on https://nim-lang.org/docs/docgen.html ? I'm not sure what the purpose is of making this weird rST+Markdown amalgam the default.

However, I still contend something is wrong as currently there seems to be no way to actually use the footnote functionality by default (or I am dumb), nor does the markdown doctype seem to support pandoc style footnotes.

Nim Version

Nim Compiler Version 1.7.3 [Linux: amd64]
Compiled at 2022-10-27
Copyright (c) 2006-2022 by Andreas Rumpf

git hash: 3469f37
active boot switches: -d:release

Current Output

No response

Expected Output

No response

Possible Solution

No response

Additional Information

No response

[Araq]

I'm not sure what the purpose is of making this weird rST+Markdown amalgam the default.

It went like this: "Nim's docgen uses RST, deal with it."

People: "No, omg! We can only understand and use markdown"

Me: "There are thousands of lines already written that successfully use RST and markdown and RST are so similar we can easily support both and make everybody happy"

My analysis might have been off but Markdown also keeps changing and nobody knows what Markdown is, are tables now part of the markdown spec? Who knows, I cannot keep up with this nonsense.

[fsh]

Who knows, I cannot keep up with this nonsense.

Hmm, might I offer the following hypothetical as a diplomatic plan of attack:

1. baseline sanity: Make nim doc actually commit to one or the other by default (whichever covers the most extant Nim docs (RST), or whichever is the most expressive). With the goal (even if not met 100%) being to adhere to some concrete spec of that syntax (which can be linked and referred to). Any weird hybridization or ad-hoc extensions are relegated to other doctypes.

2. usability: Add warnings or make parsing errors output a helpful message about the doctype pragma, so when people use Markdown syntax it will inform them they need to set the correct doctype. (Acknowledging that Markdown is indeed the only markup many people know or want to use, even though I would agree with you that it is not great: ref1 ref2 ref3)

3. extensibility (long term): in some future, make the doctype pragma function as a sort of implicit import (only for nim doc), and then hand over control to that package for the actual parsing, allowing any kind of doctype by way of installing packages for, say, asciidoc, org-mode, etc., without having to maintain this stuff in the compiler. Would need some sort of rough spec on what functions etc such a packages are expected to provide.

[Araq]

I don't like doctype, there should be a single format that works. doctype also affects IDE integration.

What's wrong with "we support our own hybrid format which we claim to be markdown compatible". It's not my fault that markdown is not particularly well thought-out. RST also has plenty of flaws though.

What we have works reasonably well. We should just give it a name like "Nim structured text" so that people don't draw the wrong conclusions.

[fsh]

I don't like doctype, there should be a single format that works.

Fair enough, & the IDE integration point is indeed a good one! Tho then the concrete problem I originally had remains, as there is no way to access the footnote code path / functionality without it? (That I can figure out.) So then Nim Structured Text does not support footnotes and the dead code can be eliminated?

We should just give it a name like "Nim structured text" so that people don't draw the wrong conclusions.

I strongly sympathize with the distaste of implementing a spec you think is badly designed or flawed. Tho I also feel using a custom/homemade/hybrid format will paradoxically lead to a lot more work in the end, as in such cases there's a lot of extra documentation, guides, answers, etc. that need to be written to demarcate the differences and ease users into this new dialect. Following a spec is potentially more work in actual code, but in that case there's an already extant cornucopia of documentation, tutorials, & examples which can be referred to, offloading a lot (if not all) of documentation work. (And I find writing full & complete documentation is just as arduous as coding, if not more-so.)

[Araq]

Tho I also feel using a custom/homemade/hybrid format will paradoxically lead to a lot more work in the end

But that's what we already have and in practice "in the end" people need a preview of how it's rendered regardless if we follow spec X closely as most people cannot keep X in their heads reliably.