The parser does not parse the full ISTQB example document

[Witiko]

Consider the document example-istqb-content.md. The document contains a number of sections, but only the first paragraph of the document is recognized by the parser of Markdown 3.0.0:

$ docker run --rm -it -v "$PWD":/mnt -w /mnt witiko/markdown:3.0.0-0-gdcaea6c-latest markdown-cli example-istqb-content.md
\markdownRendererDocumentBegin
\markdownRendererSectionBegin
\markdownRendererHeadingOne{Fundamentals of Testing}\markdownRendererInterblockSeparator
{}\markdownRendererSectionBegin
\markdownRendererHeadingTwo{Showcase of the LaTeX template for ISTQB documents}\markdownRendererInterblockSeparator
{}\markdownRendererSectionBegin
\markdownRendererHeadingThree{Regular text}\markdownRendererInterblockSeparator
{}Regular parapgraphs and senteces are written without any formatting. New line is depited by two spaces at the end of the line and a new line.\markdownRendererParagraphSeparator
{}
\markdownRendererSectionEnd 
\markdownRendererSectionEnd 
\markdownRendererSectionEnd \markdownRendererDocumentEnd

This issue is likely linked to the addition of TeX-like rich paragraphs in #306. Closing this issue will allow us to continue work on https://github.com/danopolan/istqb_latex/pull/77.

[Witiko]

Here is a minimal working example document example.md, notice the single trailing space at the end of the second paragraph:

Here is the first paragraph.

Here is the second paragraph. 

Here is the third paragraph.

Here is the result of running markdown-cli example.md:

\markdownRendererDocumentBegin
Here is the first paragraph.\markdownRendererParagraphSeparator
{}\markdownRendererDocumentEnd

The expected behavior is that three paragraphs are recognized.

Removing the trailing space produces the correct output:

\markdownRendererDocumentBegin
Here is the first paragraph.\markdownRendererParagraphSeparator
{}Here is the second paragraph.\markdownRendererParagraphSeparator
{}Here is the third paragraph.\markdownRendererDocumentEnd

This issue seems to target the blind spot of our unit tests: When writing inline markdown, no TeX engine other than ConTeXt can recognize trailing spaces. Since the unit tests expect the same results for inline markdown and when inputting markdown documents using the \markdownInput command (or using Lua as in this issue), the behavior of trailing spaces are not tested.

As the next step, we should try to reproduce the error using a commit before merging #306 to see if the error has been introduced by #306.

[Witiko]

Regardless, we may want to add support for YAML blocks at the top of testfiles that would allow us to specify whether a testfile should be processed for a given format or template. For example:

if: format == 'context-mkiv' or template == 'verbatim'  # processed by python's eval() built-in
---
<<<
This is a line of text.␣␣
This is another line of text.
>>>
documentBegin
hardLineBreak
documentEnd

This would allow us to test the behavior of trailing spaces even though they cannot be processed by some formats and templates.

This should not be necessary for this issue, where the trailing spaces at the end of a paragraph should always produce the same output regardless of whether the format/template allows TeX to see the trailing spaces. However, it would be useful for testing hard line breaks.

[Witiko]

As the next step, we should try to reproduce the error using a commit before merging #306 to see if the error has been introduced by #306.

I failed to reproduce the error using the Docker image witiko/markdown:3.0.0-alpha.2-120-gd4acce2-latest from commit d4acce2 immediately before merging #306. Here is the result of running markdown-cli example.md:

Here is the first paragraph.\markdownRendererInterblockSeparator
{}Here is the second paragraph.\markdownRendererInterblockSeparator
{}Here is the third paragraph.\markdownRendererDocumentEnd

This indicates that the error has been introduced by #306.