Fix problems in the markdown -> html pipeline

[koraa]

This fixes two issues with markdown rendering and adds an infrastructure for testing whether for a specific markdown input, the correct output is produced.

Not sure whether test/testHTMLFromMarkdown.js should not be merged into test/testHTML.js; I kind of like having all the markdown tests in their own place, on the other hand a lot of boilerplate is required to put them there…

[koraa]

I am also pondering what would be a good place for a domEquals function? Thinking of extracting one from assertDomEquals and putting those two functions somewhere where all helix code can use it…

[trieloff]

We may want to create a new helix-testutils project. I need to put my condit method for conditional integration testing somewhere, too.

[koraa]

After fixing the missing support for the missing "referenceType" attribute in the mdast json schema, I can't get the example below to work; needs extra investigation but I do not think this is any error we introduced, since this is a highly specific property of the GFM & CommonMark specs (which flavor are we using by the way).
Pondering whether this is worth fixing if it's somewhere deep inside mdast…in general IMHO it's worth fixing those really finicky errors because they make for even weirder edgecases and issues…

As far as I can see, the markdown is already being parsed incorrectly…it parses [link] as a reference and (<foobar) as adjacent text…

Input Markdown

# Foo

Hello World [link](<foobar)

Expected result

(As rendered by marked with GFM enabled and without)

<h1 id="foo">Foo</h1>
<p>Hello World <a href="%3Cfoobar">link</a></p>

Actual Result

<h1>Foo</h1>
<p>Hello World [link](&lt;foobar)</p>

MDAST

{ type: 'root',
  children:
   [ { type: 'heading',
       depth: 1,
       children:
        [ { type: 'text',
            value: 'Foo',
            position:
             Position {
               start: { line: 1, column: 3, offset: 2 },
               end: { line: 1, column: 6, offset: 5 },
               indent: [] } } ],
       position:
        Position {
          start: { line: 1, column: 1, offset: 0 },
          end: { line: 1, column: 6, offset: 5 },
          indent: [] } },
     { type: 'paragraph',
       children:
        [ { type: 'text',
            value: 'Hello World ',
            position:
             Position {
               start: { line: 3, column: 1, offset: 7 },
               end: { line: 3, column: 13, offset: 19 },
               indent: [] } },
          { type: 'linkReference',
            identifier: 'link',
            label: 'link',
            referenceType: 'shortcut',
            children:
             [ { type: 'text',
                 value: 'link',
                 position:
                  Position {
                    start: { line: 3, column: 14, offset: 20 },
                    end: { line: 3, column: 18, offset: 24 },
                    indent: [] } } ],
            position:
             Position {
               start: { line: 3, column: 13, offset: 19 },
               end: { line: 3, column: 19, offset: 25 },
               indent: [] } },
          { type: 'text',
            value: '(<foobar)',
            position:
             Position {
               start: { line: 3, column: 19, offset: 25 },
               end: { line: 3, column: 28, offset: 34 },
               indent: [] } } ],
       position:
        Position {
          start: { line: 3, column: 1, offset: 7 },
          end: { line: 3, column: 28, offset: 34 },
          indent: [] } } ],
  position:
   { start: { line: 1, column: 1, offset: 0 },
     end: { line: 3, column: 28, offset: 34 } } }

HAST

{ type: 'root',
  children:
   [ { type: 'element',
       tagName: 'h1',
       properties: {},
       children:
        [ { type: 'text',
            value: 'Foo',
            position:
             { start: { line: 1, column: 3, offset: 2 },
               end: { line: 1, column: 6, offset: 5 } } } ],
       position:
        { start: { line: 1, column: 1, offset: 0 },
          end: { line: 1, column: 6, offset: 5 } } },
     { type: 'text', value: '\n' },
     { type: 'element',
       tagName: 'p',
       properties: {},
       children:
        [ { type: 'text',
            value: 'Hello World ',
            position:
             { start: { line: 3, column: 1, offset: 7 },
               end: { line: 3, column: 13, offset: 19 } } },
          { type: 'text',
            value: '[link]',
            position:
             { start: { line: 3, column: 14, offset: 20 },
               end: { line: 3, column: 18, offset: 24 } } },
          { type: 'text',
            value: '(<foobar)',
            position:
             { start: { line: 3, column: 19, offset: 25 },
               end: { line: 3, column: 28, offset: 34 } } } ],
       position:
        { start: { line: 3, column: 1, offset: 7 },
          end: { line: 3, column: 28, offset: 34 } } } ],
  position:
   { start: { line: 1, column: 1, offset: 0 },
     end: { line: 3, column: 28, offset: 34 } } }

[codecov]

Codecov Report

Merging #163 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #163   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          39     39           
  Lines         839    833    -6     
  Branches      164    160    -4     
=====================================
- Hits          839    833    -6

Impacted Files	Coverage Δ
src/html/emit-html.js	100% <100%> (ø)	⬆️
src/defaults/html.pipe.js	100% <100%> (ø)	⬆️
src/utils/mdast-to-vdom.js	100% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed30562...a4b33c6. Read the comment docs.

[koraa]

Please do not merge yet; wip

[koraa]

Btw, this PR removes the VDOM.process function; is it used anywhere outside pipe?

[trieloff]

We are using GFM. If you've discovered an issue in the remark tools, file an issue against the project and create a PR. @woorm is generally fast to respond, review, merge and release.

[trieloff]

The idea of this function is to be used in project-code, e.g. if you want to override the way HTML is being generated, you'd be able to register a new handler and then run process in your pre.js

[trieloff]

But I guess just using getDocument is easier.

[koraa]

Should be ready to merge as soon as the branch in helix-shared is merged.
Please merge in separate commits (already squashed into sensible atomic changes)

[kptdobe]

arff... no... too fast...
I use the process method because it returns a node only and not a full document! See https://github.com/kptdobe/helix-sections-playground/blob/master/src/html.pre.js#L50