Colophon - October 1st, 2014

This project was initiated to provide a test suite for Markdown markup, and eventually create a specification from this test results. A part of of the community has started a new endeavor which seems to get traction as CommonMark. We are then closing this project and encourage you to contribute to CommonMark.

The most interesting part of this project would not have been possible without the dedication of Ciro Santilli @cirosantilli. So big applause and thank you to him.

The rest is kept around for archives and references.

Markdown Test Suite

Inspired by questions on W3C Markdown Community Group.

Pull Requests are welcome. See the CONTRIBUTING Guidelines

Design goals

• Comprehensive.
• Small modularized tests.
• Easy to run tests using any programming language. In particular, data representations must have an implementation on all major languages.
• Develop a consensus based markdown specification at markdown-spec.html. Visualize it here.

Test Scripts

Markdown Test Suite already includes tests for many important markdown engines.

To see what the scripts do run:

./cat-all.py -h
./run-tests.py -h

To configure the scripts do:

cp config_local.py.example config_local.py

and edit config_local.py. It is already gitignored.

A Vagrantfile is provided with a provision script that installs all installable engines.

Sample output from run-tests.py:

blackfriday   |......FFF..............FF...F....................................F....................................|   0.93s  102    7   6%
gfm           |FF.....F.....F...FFFF..F....F........................FFFF...FF...............FF...F.......F...........| 262.88s  102   20  19%
hoedown       |..............................F.................................................F.....................|   0.36s  102    2   1%
kramdown      |......FFF.....FF.......FF.......FF.FFFFFFFFFFFFF.......................F..............................|  30.69s  102   23  22%
lunamark      |FFFFFF.F.FFFFFFFFFFFF.F.....FFFF..FF............FFFFF..FFFFFFFFFF..............F...FFFFFFFFFFF........|   1.58s  103   53  51%
markdown_pl   |.....................FFFF...............................F...............F.............................|   2.56s  102    6   5%
markdown2     |......................................................................................................|   5.39s  103    0   0%
marked        |..............F.................FFFFFFFFFFFFFFFF......................F.F.............................|   6.22s  102   19  18%
maruku        |......FFF......F.......F.FFF...............................................FF.........................|  37.02s  102   10   9%
md2html       |.........................FFF...F............................................FF........................|   7.42s  102    6   5%
multimarkdown |......FFF....FF...F............FFF.FFFFFFFFFFFFF.....FFFF.........F...................................|   0.58s  102   27  26%
pandoc        |FF...........F.F.FFFF..FFFFF....FF.FFFFFFFFFFFFF.....FFFF.....FF............FFF.FFF..................F|   1.11s  102   41  40%
peg_markdown  |.................................................................F....................................|   0.46s  102    1   0%
rdiscount     |.......F.........................................................F....................................|  25.19s  102    3   2%
redcarpet     |......................................................................................................|  21.49s  102    0   0%
showdown      |.......................................................................F..............................|   6.85s  102    1   0%

Extensions:

blackfriday   |F..|  0.04s    3    1  33%
gfm           |F.|   2.37s    2    1  50%
hoedown       |.|    0.00s    1    0   0%
lunamark      |F.F|  0.05s    3    2  66%
kramdown      |..|   0.62s    2    0   0%
markdown_pl   ||     0.00s    0    0   0%
markdown2     |F.|   0.14s    2    1  50%
marked        |F.|   0.13s    2    1  50%
maruku        |F..|  1.06s    3    1  33%
md2html       |F.|   0.14s    2    0   0%
multimarkdown |F.|   0.01s    2    1  50%
pandoc        |F.|   0.02s    2    1  50%
peg_markdown  |...|  0.02s    3    0   0%
rdiscount     |F..|   0.74s    3    1  33%
redcarpet     |..|   0.42s    2    0   0%
showdown      |F..|  0.20s    3    1  33%

Where F indicates a failing test.

Other Noticeable Test Suites

We haven't been the first test suite effort. Some projects have maintained their own test suite for a long time. Hopefully we can reach a state where people agree on the terms of what should be a good test suite for all developers.

• Original test suite
• PHP markdown test suite:
• Markdown-js test suite
• Python markdown 2 test suite
• multimarkdown test suite
• John MacFarlane's Standard Markdown spec

In addition we should note the wonderful work made by John Mac Farlane. The Web service output the differences in between the different markdown implementations. It helps a lot when searching on the most common output.