Replace hoedown markdown parser with CMark

[redstrate]

This is a draft for patches to replace the old, crusty and buggy hoedown markdown library - with CMark.

Benefits

• Less resource management (no more hoedown.h needed to wrap api calls)
• Already used in production by other big projects like KDE and GNOME
• Maintained

CMark also supports raw HTML blocks, so stuff like mod pages are no longer broken and have to be worked around (#713):

Since we don't need a resource wrapper, and less quirks because hoedown is terrible - it also means less lines of code worrying about markdown rendering :-)

Tasks

• Check everywhere where markdown is shown and make sure there aren't any rendering problems.
• Vendor/import CMark (see question below)
• Remove hoedown vendored source
• Check around to see if I'm missing any quirks that were previously used with hoedown, I think I zapped all of them though.

Questions

• Should we vendor in cmark considering it's in a lot of distribution repositories already (unlike hoedown/sundown?)
  It's a small library anyway, so it doesn't matter either way.
• How should we handle the markdownToHTML function? I haven't touched the codebase in a while, so I'm not sure if you want anything more than a static global function.

[DioEgizio]

Should we vendor in cmark considering it's in a lot of distribution repositories already (unlike hoedown/sundown?)
It's a small library anyway, so it doesn't matter either way.

Do like zlib/quazip/ghc-filesystem etc and check if it's installed in the system, dynamically link if found and if it isn't found or Launcher_FORCE_BUNDLED_LIBS is on statically link

[DioEgizio]

Since modrinth uses GitHub flavoured markdown, might make more sense to use cmark-gfm (GitHub's cmark fork) instead

https://github.com/github/cmark-gfm

[DioEgizio]

Should we vendor in cmark considering it's in a lot of distribution repositories already (unlike hoedown/sundown?)
It's a small library anyway, so it doesn't matter either way.

Do like zlib/quazip/ghc-filesystem etc and check if it's installed in the system, dynamically link if found and if it isn't found or Launcher_FORCE_BUNDLED_LIBS is on statically link

Forgor to say it but cmark-gfm is also commonly packaged (see https://repology.org/project/cmark-gfm/versions ), so yeah do what I said there ^

[Scrumplex]

Should we vendor in cmark considering it's in a lot of distribution repositories already (unlike hoedown/sundown?)
It's a small library anyway, so it doesn't matter either way.

We probably need this at the very least for our MSVC Windows builds.

I have pushed a commit adding support for system installs of CMark as well as falling back to a submodule, if required.

I have also looked at cmark-gfm, but it looks like it doesn't provide CMake files to allow finding it using find_package. Might need to use pkgconfig for that.

[redstrate]

I have also looked at cmark-gfm, but it looks like it doesn't provide CMake files to allow finding it using find_package. Might need to use pkgconfig for that.

Erk, that's annoying :-) But if we want to switch to GFM for whatever reason - it shouldn't change anything API wise so that's something we can put off doing until later. It's not like I encountered anything in the changelogs or mod pages I've viewed that requires GH extensions

[DioEgizio]

I guess not having gfm extensions is not a big issue for now, but flatpak build for some reason is failing

[redstrate]

I guess not having gfm extensions is not a big issue for now, but flatpak build for some reason is failing

The fix should be simple, I just need to add cmark to the flatpak manifest. I'll be adding a commit to fix that :-)

[redstrate]

Also, is there a standard format for writing new license headers? I see a mix of some of the old license headers from the MultiMC/PolyMC days but also SPDX. I'm guessing I should mark it GPLv3 with SPDX.

[redstrate]

I also added an additional fix, I noticed that Modrinth pages specifically has extra line breaks. These are now stripped (Curseforge and other mod platforms I tested don't suffer from this issue):

How it looked before:

[redstrate]

I also tested every place I could (with the exception of the updater, I don't know how to test that easily). The commits are all signed, messages fixed up and ready for review now :-)

[DioEgizio]

just found a mod using gfm stuff

https://modrinth.com/mod/immediatelyfast

uses tables

[redstrate]

just found a mod using gfm stuff

Erk :-) I'll look into changing it to the GFM fork then

[redstrate]

I got around to working on this again, but looking through cmark gfm is nothing but pain.

Not only are they a fork of cmark - that part is fine - but they don't include important commits like github/cmark-gfm@c8d5be9 which fixes CMake find_package as shown above. I could make a PR upstream to fix that though.

Also the availability of cmark-gfm pales in comparison to regular cmark. Not that this is a problem if we include it as a submodule, this only matters for linux packaging.

And to include the relevant GitHub extensions (notably tables) I would have to expand the markdownToHtml function because there's no simple way to add extensions through the cmark API. Not that big of a deal, but annoying.

Okay, so let's just accept all of those problems and just take the plunge:

/usr/include/cmark-gfm-core-extensions.h:10:10: fatal error: config.h: No such file or directory
   10 | #include "config.h" // for bool
      |          ^~~~~~~~~~
compilation terminated.

No way, it includes a private header in a public API, so even if I wanted to include cmark-gfm support I couldn't :-( Oh there's an open PR for that but it hasn't been merged yet. Since there's been no activity, Nix and Ghostwriter had to implement workarounds for this bug, Nix PR here and ghostwriter commit here respectively.

Is there that many mods that use tables in their READMEs? It's a shame that they don't render, but honestly cmark-gfm is a baffling library. We could vendor cmark-gfm and then apply our own patches, but I think that's a bad idea for a lot of reasons (and ideally, this would be fixed upstream).

I think this PR is in a good state right now and we can look at cmark-gfm support later, because with just plain cmark our markdown is much better than before :-) Perfection cannot be the enemy of good, etc

[DioEgizio]

Yeah let's just use normal cmark then

[flowln]

lgtm, thanks!

[DioEgizio]

When you rebase develop with signoff:

[redstrate]

Oops, somehow a bunch of unrelated commits made it's way into here :S