Use custom parser for markdown processing. #198
Conversation
Previously, mdoc used the flexmark library for processing markdown files. Since flexmark is a proper markdown parser, it may not always understand the format of the document that mdoc was processing, for example: * revealjs slides inside HTML files * custom markdown extensions that are not supported by commonmark This commit introduces a new custom parser that only understands the parts of markdown files that mdoc touches: code fences. Since mdoc doesn't need to understand all markdown flavors to process code fences, we can get away with a much simpler parser than flexmark (that understands all of markdown). With this change, it should be possible to use mdoc on any document as long as the code fences start at the beginning of a line with the triple-backtick "scala mdoc" syntax ```` ```scala mdoc println(42) ``` ```` Everything outside the code fences is left unchanged, character-by-character, so the underlying markup language is preserved.
|
There are still several failing test suites |
|
FYI @tpolecat this should unblock some tut users from migrating to mdoc. My understanding is that a common use-case is to run tut on HTML files with Reveal.js slides, is that correct? Are there any test cases I could add to ensure this use-case is covered now? |
|
@olafurpg yes, the remark.js use case is the main thing holding people back. I don't have any specific testcases for it because tut ignores everything outside fenced blocks. This is great! |
|
The only regression from this change is that we no longer expand site variables in the |
| |``` | ||
| | | ||
| |```scala | ||
| |implicit val x: Int = 41 |
There was a problem hiding this comment.
@hmf Can you elaborate on why this test was passing before? The mdoc:reset modifier should reset the scope so the println(x) should not compile. I updated the test case to introduce a new implicit val x value
|
Let's give this a try. The failing tests mostly highlighted a lot of cases where flexmark was introducing blank lines that didn't exist in the original source. |
|
1.4.0-RC2 is out with this change! cc/ @julien-truffaut iirc, you were unable to use mdoc with reveal.js, could you maybe give it a try again with this release? |
|
@olafurpg Thanks let me give it a try |
|
The file still needs to have the |
|
@olafurpg It works! it would be perfect if mdoc could also process html files. Here is an example: fp-tower/scala-intro#1 |
|
Awesome! I'll update it to process |
|
@julien-truffaut 1.4.0-RC3 should process |
|
Awesome. Thanks a lot @olafurpg |
Previously, mdoc used the flexmark library for processing markdown
files. Since flexmark is a proper markdown parser, it may not always
understand the format of the document that mdoc was processing, for
example:
This commit introduces a new custom parser that only understands the
parts of markdown files that mdoc touches: code fences. Since mdoc
doesn't need to understand all markdown flavors to process code fences,
we can get away with a much simpler parser than flexmark.
With this change, it should be possible to use mdoc on any document as
long as the code fences start at the beginning of a line with the
triple-backtick "scala mdoc" syntax
Everything outside the code fences is left unchanged,
character-by-character, so the underlying markup language is
preserved.