Java Implementation of Markdown parser with source level AST. CommonMark 0.27, emulation of: pegdown, kramdown, markdown.pl, MultiMarkdown and HTML to Markdown conversion module.
Java HTML
Pull request Compare This branch is 364 commits ahead, 112 commits behind atlassian:master.
Permalink
Failed to load latest commit information.
.idea rename flexmark-parent to flexmark-all Feb 19, 2017
META-INF add table options, fix ext-gfm-tables and ext-tables Jun 14, 2016
assets/images add include file sample and update readme and wiki Jan 20, 2017
etc rework for maven release Aug 6, 2016
flexmark-all rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-abbreviation rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-anchorlink rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-aside rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-autolink rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-definition rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-emoji rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-escaped-character rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-footnotes rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-gfm-strikethrough rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-gfm-tables rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-gfm-tasklist rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-ins rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-jekyll-front-matter rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-jekyll-tag rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-spec-example rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-superscript rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-tables rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-toc rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-typographic rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-wikilink rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-xwiki-macros rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-yaml-front-matter rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-ext-zzzzzz rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-formatter-test-suite add table formatting Jan 30, 2017
flexmark-formatter rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-html-parser rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-integration-test rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-java-samples rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-jira-converter rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-jira-parser fix #44 Jan 22, 2017
flexmark-pdf-converter rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-profile-pegdown rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-test-suite add HTML to PDF conversion module Feb 14, 2017
flexmark-test-util rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-util rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark-youtrack-converter rename flexmark-parent to flexmark-all Feb 19, 2017
flexmark rename flexmark-parent to flexmark-all Feb 19, 2017
lib add HTML to PDF conversion module Feb 14, 2017
.gitignore update ignore Jan 14, 2017
.gitmodules remove wiki sub-module Sep 22, 2016
.travis.yml change travis yaml jdk to 7 Dec 31, 2016
LICENSE.txt remove auto-inserted copyright from test cases Jun 17, 2016
MarkdownProcessorsEmulation.md update docs for profile config info Feb 20, 2017
README.md update docs for profile config info Feb 20, 2017
VERSION.md update readme and version Feb 19, 2017
central-pom.xml update version to 0.5.0 Sep 24, 2016
codecov.yml update codecov.yml Aug 7, 2016
flexmark-java.iml add jekyll and yaml front matter formatter to enforce these being fir… Jan 28, 2017
pom.xml rename flexmark-parent to flexmark-all Feb 19, 2017
update-plugin.sh cleanup sequence handling impl Dec 20, 2016

README.md

Flexmark Icon Logo flexmark-java

flexmark-java is a Java implementation of CommonMark 0.27 spec parser using the blocks first, inlines after Markdown parsing architecture.

Its strengths are speed, flexibility, Markdown source element based AST with details of the source position down to individual characters of lexemes that make up the element and extensibility.

The API allows granular control of the parsing process and is optimized for parsing with a large number of installed extensions. The parser and extensions come with plenty of options for parser behavior and HTML rendering variations. The end goal is to have the parser and renderer be able to mimic other parsers with great degree of accuracy. This is now partially complete with the implementation of Markdown Processor Family Emulation

Motivation for this project was the need to replace pegdown parser in my Markdown Navigator plugin for JetBrains IDEs. pegdown has a great feature set but its speed in general is less than ideal and for pathological input either hangs or practically hangs during parsing.

Latest Additions

Changes Forced by Downgrade to Java 7

I tried to keep all changes caused by the downgrade in Java language level as mechanical as possible to make it easier to migrate. The biggest change is the lack of lambda syntax and no static extension methods in interfaces.

  • All VISIT_HANDLERS in GroupNameVisitor were moved to GroupNameVisitorExt class. A mechanical search and replace from .VISIT_HANDLERS to Ext.VISIT_HANDLERS should take care of the change.

  • Lack of Lambda syntax will only have an effect if your project language level is downgraded. Otherwise, you can keep using lambda syntax.

Requirements

  • Java 7 or above
  • The core has no dependencies; for extensions, see below

Build status Maven Central status

Changes from commonmark-java project

  • The project is on Maven: com.vladsch.flexmark
  • Java compatibility back to 1.7
  • Android compatibility to be added
  • No attempt is made to keep API backward compatibility to the original project.

    The API is evolving to accommodate new extensions and functionality.

Releases, Bug Fixes, Enhancements and Support

I use flexmark-java as the parser for Markdown Navigator plugin for JetBrains IDEs. I tend to use the latest, unreleased version to fix bugs or get improvements. So if you find a bug that is a show stopper for your project or see a bug in github issues page marked fixed for next release that is affecting your project then please let me know and I may be able to promptly make a new release to address your issue. Otherwise, I will let bug fixes and enhancements accumulate thinking no one is affected by what is already fixed.

Extension points in the API are many and numerous

There are many extension options in the API with their intended use. If your extension lines up with the right API, the task is usually very short and sweet. If your extension uses the API in an unintended fashion or does not follow expected housekeeping protocols, you may find it an uphill battle with a rat's nest of if/else condition handling and fixing one bug only leading to creating another one.

Generally, if it takes more than a few dozen lines to add a simple extension, then either you are going about it wrong or the API is missing an extension point. If you look at all the implemented extensions you will see that most are a few lines of code other than boiler plate dictated by the API. That is the goal for this library: provide an extensible core that makes writing extensions a breeze.

The larger extensions are flexmark-ext-tables and flexmark-ext-spec-example, the meat of both is around 200 lines of code. You can use them as a guide post for size estimating your extension.

My own experience adding extensions shows that sometimes a new type of extension is best addressed with an API enhancement to make its implementation seamless, or a by fixing a bug that was not visible before the extension excersized the API in just the right way. Your intended extension may just be the one requiring such an approach.

Don't hesitate to open an issue if you can't find the answer

The takeaway is: if you want to implement an extension or a feature please don't hesitate to open an issue and I will give you pointers on the best way to go about it. It may save you a lot of time by letting me improve the API to address your extension's needs before you put a lot of fruitless effort into it.

I do ask that you realize that I am chief cook and bottle washer on this project, without an iota of Vulcan Mind Melding skills. I do ask that you describe what you want to implement because I can't read your mind and do some reconaissance background work around the source code and documentation because I cannot transfer what I know to you, without your willing effort.

Consulting is available

If you have a commercial application and don't want to write the extension(s) yourself or want to reduce the time and effort of implementing extensions and integrating flexmark-java, feel free to contact me. I am available on a consulting/contracting basis, All about me.

Pegdown Migration Helper

PegdownOptionsAdapter class converts pegdown Extensions.* flags to flexmark options and extensions list. Pegdown Extensions.java is included for convenience. These are located in flexmark-profile-pegdown module, was added to Maven but you can grab the source from this repo: PegdownOptionsAdapter.java and if you need it Extensions.java and make your own version, modified to your project's needs.

You can pass your extension flags to static PegdownOptionsAdapter.flexmarkOptions(int) or you can instantiate PegdownOptionsAdapter and use convenience methods to set, add and remove extension flags. PegdownOptionsAdapter.getFlexmarkOptions() will return a fresh copy of DataHolder every time with the options reflecting pegdown extension flags.

import com.vladsch.flexmark.html.HtmlRenderer;
import com.vladsch.flexmark.parser.Parser;
import com.vladsch.flexmark.profiles.pegdown.Extensions;
import com.vladsch.flexmark.profiles.pegdown.PegdownOptionsAdapter;
import com.vladsch.flexmark.util.options.DataHolder;

public class PegdownOptions {
    static final DataHolder OPTIONS = PegdownOptionsAdapter.flexmarkOptions(
            Extensions.ALL
    );

    static final Parser PARSER = Parser.builder(OPTIONS).build();
    static final HtmlRenderer RENDERER = HtmlRenderer.builder(OPTIONS).build();

    // use the PARSER to parse and RENDERER to render with pegdown compatibility
}

ℹ️ flexmark-java has many more extensions and configuration options than pegdown in addition to extensions available in pegdown 1.6.0. Available Extensions via PegdownOptionsAdapter

Markdown Processor Emulation

Despite its name, commonmark is neither a superset nor a subset of other markdown flavors. Rather, it proposes a standard, unambiguous syntax specification for the original, "core" Markdown, thus effectively introducing yet another flavor. While flexmark is by default commonmark compliant, its parser can be tweaked in various ways. The sets of tweaks required to emulate the most commonly used markdown parsers around are available in flexmark as ParserEmulationProfiles.

As the name ParserEmulationProfile implies, it's only the parser that is adjusted to the specific markdown flavor. Applying the profile does not add features beyond those available in commonmark. If you want to use flexmark to fully emulate another markdown processor's behavior, you have to adjust the parser and configure the flexmark extensions that provide the additional features available in the parser that you want to emulate.

Latest addition was a rewrite of the list parser to better control emulation of other markdown processors as per Markdown Processors Emulation. Addition of processor presets to emulate specific markdown processing behaviour of these parsers is on a short to do list.

Some emulation families do a better better job of emulating their target than others. Most of the effort was directed at emulating how these processors parse standard Markdown and list related parsing specifically. For processors that extend original Markdown, you will need to add those extensions that are already implemented in flexmark-java to the Parser/Renderer builder options.

Extensions will be modified to include their own presets for specific processor emulation, if that processor has an equivalent extension implemented.

If you find a discrepancy please open an issue so it can be addressed.

Major processor families are implemented and some family members also:

ℹ️ profiles to encapsulate configuration details for variants within the family were added in 0.11.0:

  • CommonMark (default for family): ParserEmulationProfile.COMMONMARK
  • FixedIndent (default for family): ParserEmulationProfile.FIXED_INDENT
  • GitHub Comments (just CommonMark): ParserEmulationProfile.COMMONMARK
  • GitHub Docs: ParserEmulationProfile.GITHUB_DOC
  • Kramdown (default for family): ParserEmulationProfile.KRAMDOWN
  • Markdown.pl (default for family): ParserEmulationProfile.MARKDOWN
  • MultiMarkdown: ParserEmulationProfile.MULTI_MARKDOWN
  • Pegdown, with pegdown extensions use PegdownOptionsAdapter in flexmark-profile-pegdown
  • Pegdown, without pegdown extensions ParserEmulationProfile.PEGDOWN

History and Motivation

flexmark-java is a fork of commonmark-java project, modified to generate an AST which reflects all the elements in the original source, full source position tracking for all elements in the AST and easier JetBrains Open API PsiTree generation.

The API was changed to allow more granular control of the parsing process and optimized for parsing with a large number of installed extensions. The parser and extensions come with many tweaking options for parser behavior and HTML rendering variations. The end goal is to have the parser and renderer be able to mimic other parsers with great degree of accuracy.

Motivation for this was the need to replace pegdown parser in Markdown Navigator plugin. pegdown has a great feature set but its speed in general is less than ideal and for pathological input either hangs or practically hangs during parsing.

commonmark-java has an excellent parsing architecture that is easy to understand and extend. The goal was to ensure that adding source position tracking in the AST would not change the ease of parsing and generating the AST more than absolutely necessary.

Reasons for choosing commonmark-java as the parser are: speed, ease of understanding, ease of extending and speed. More detailed description in Pegdown - Achilles heel of the Markdown Navigator plugin. Now that I have reworked the core and added a few extensions I am extremely satisfied with my choice.

Another goal was to improve the ability of extensions to modify parser behavior so that any dialect of markdown could be implemented through the extension mechanism. An extensible options API was added to allow setting of all options in one place. Parser, renderer and extensions use these options for configuration, including disabling some core block parsers.

This is a work in progress with many API changes. No attempt is made to keep backward API compatibility to the original project and until the feature set is mostly complete, not even to earlier versions of this project.

Feature Comparison

Feature flexmark-java commmonmark-java pegdown
Relative parse time (less is better) ✔️ 1x (1) ✔️ 0.6x to 0.7x (2) 25x average, 20,000x to ∞ for pathological input (3)
All source elements in the AST ✔️ ✔️
AST elements with source position ✔️ ✔️ with some errors and idiosyncrasies
AST can be easily manipulated ✔️ AST post processing is an extension mechanism ✔️ AST post processing is an extension mechanism not an option. No node's parent information, children as List<>.
AST elements have detailed source position for all parts ✔️ only node start/end
Can disable core parsing features ✔️
Core parser implemented via the extension API ✔️ instanceOf tests for specific block parser and node classes core exposes few extension points
Easy to understand and modify parser implementation ✔️ ✔️ one PEG parser with complex interactions (3)
Parsing of block elements is independent from each other ✔️ ✔️ everything in one PEG grammar
Uniform configuration across: parser, renderer and all extensions ✔️ none beyond extension list int bit flags for core, none for extensions
Parsing performance optimized for use with extensions ✔️ parsing performance for core, extensions do what they can performance is not a feature
Feature rich with many configuration options and extensions out of the box ✔️ limited extensions, no options ✔️
Dependency definitions for processors to guarantee the right order of processing ✔️ order specified by extension list ordering, error prone not applicable, core defines where extension processing is added
(1)

flexmark-java pathological input of 100,000 [ parses in 68ms, 100,000 ] in 57ms, 100,000 nested [ ] parse in 55ms

(2)

commonmark-java pathological input of 100,000 [ parses in 30ms, 100,000 ] in 30ms, 100,000 nested [ ] parse in 43ms

(3)

pegdown pathological input of 17 [ parses in 650ms, 18 [ in 1300ms

Progress

  • Parser options, items marked as a task item are to be implemented the rest are complete:
    • Typographic
      • Quotes
      • Smarts
    • GitHub Extensions
      • Fenced code blocks
      • Anchor links for headers with auto id generation
      • Table Spans option to be implemented for tables extension
      • Wiki Links with GitHub and Creole syntax
      • Emoji Shortcuts with use GitHub emoji URL option
    • GitHub Syntax
      • Strikethrough
      • Task Lists
      • No Atx Header Space
      • No Header indents
      • Hard Wraps (achieved with SOFT_BREAK option changed to "<br />")
      • Relaxed HR Rules Option
      • Wiki links
    • Publishing
      • Abbreviations
      • Footnotes
      • Definitions
      • Table of Contents
    • Suppress
      • inline HTML: all, non-comments, comments
      • HTML blocks: all, non-comments, comments
    • Processor Extensions
      • Jekyll front matter
      • Jekyll tag elements, with support for {% include file %}, Include Markdown and HTML File Content
      • GitBook link URL encoding. Not applicable
      • HTML comment nodes: Block and Inline
      • Multi-line Image URLs
      • Spec Example Element
    • Commonmark Syntax suppression
      • Manual loose lists
      • Numbered lists always start with 1.
      • Fixed list item indent, items must be indented by at least 4 spaces
      • Relaxed list start option, allow lists to start when not preceded by a blank line.

I am very pleased with the decision to switch to commonmark-java based parser for my own projects. Even though I had to do major surgery on its innards to get full source position tracking and AST that matches source elements, it is a pleasure to work with and is now a pleasure to extend. If you don't need source level element AST or the rest of what flexmark-java added and CommonMark is your target markdown parser then I encourage you to use commonmark-java as it is an excellent choice for your needs and its performance does not suffer for the overhead of features that you will not use.

Benchmarks

Latest, Jan 28, 2017 flexmark-java 0.13.1, intellij-markdown from CE EAP 2017, commonmark-java 0.8.0:

File commonmark-java flexmark-java intellij-markdown pegdown
README-SLOW 0.420ms 0.812ms 2.027ms 15.483ms
VERSION 0.743ms 1.425ms 4.057ms 42.936ms
commonMarkSpec 31.025ms 44.465ms 600.654ms 575.131ms
markdown_example 8.490ms 10.502ms 223.593ms 983.640ms
spec 4.719ms 6.249ms 35.883ms 307.176ms
table 0.229ms 0.623ms 0.800ms 3.642ms
table-format 1.385ms 2.881ms 4.150ms 23.592ms
wrap 3.804ms 4.589ms 16.609ms 86.383ms

Ratios of above:

File commonmark-java flexmark-java intellij-markdown pegdown
README-SLOW 1.00 1.93 4.83 36.88
VERSION 1.00 1.92 5.46 57.78
commonMarkSpec 1.00 1.43 19.36 18.54
markdown_example 1.00 1.24 26.34 115.86
spec 1.00 1.32 7.60 65.09
table 1.00 2.72 3.49 15.90
table-format 1.00 2.08 3.00 17.03
wrap 1.00 1.21 4.37 22.71
----------- --------- --------- --------- ---------
overall 1.00 1.41 17.47 40.11
File commonmark-java flexmark-java intellij-markdown pegdown
README-SLOW 0.52 1.00 2.50 19.07
VERSION 0.52 1.00 2.85 30.12
commonMarkSpec 0.70 1.00 13.51 12.93
markdown_example 0.81 1.00 21.29 93.66
spec 0.76 1.00 5.74 49.15
table 0.37 1.00 1.28 5.85
table-format 0.48 1.00 1.44 8.19
wrap 0.83 1.00 3.62 18.83
----------- --------- --------- --------- ---------
overall 0.71 1.00 12.41 28.48

Because these two files represent the pathological input for pegdown, I no longer run them as part of the benchmark to prevent skewing of the results. The results are here for posterity.

File commonmark-java flexmark-java intellij-markdown pegdown
hang-pegdown 0.082ms 0.326ms 0.342ms 659.138ms
hang-pegdown2 0.048ms 0.235ms 0.198ms 1312.944ms

Ratios of above:

File commonmark-java flexmark-java intellij-markdown pegdown
hang-pegdown 1.00 3.98 4.17 8048.38
hang-pegdown2 1.00 4.86 4.10 27207.32
----------- --------- --------- --------- ---------
overall 1.00 4.30 4.15 15151.91
File commonmark-java flexmark-java intellij-markdown pegdown
hang-pegdown 0.25 1.00 1.05 2024.27
hang-pegdown2 0.21 1.00 0.84 5594.73
----------- --------- --------- --------- ---------
overall 0.23 1.00 0.96 3519.73
  • VERSION.md is the version log file I use for Markdown Navigator
  • commonMarkSpec.md is a 33k line file used in intellij-markdown test suite for performance evaluation.
  • spec.txt commonmark spec markdown file in the commonmark-java project
  • hang-pegdown.md is a file containing a single line of 17 characters [[[[[[[[[[[[[[[[[ which causes pegdown to go into a hyper-exponential parse time.
  • hang-pegdown2.md a file containing a single line of 18 characters [[[[[[[[[[[[[[[[[[ which causes pegdown to go into a hyper-exponential parse time.
  • wrap.md is a file I was using to test wrap on typing performance only to discover that it has nothing to do with the wrap on typing code when 0.1 seconds is taken by pegdown to parse the file. In the plugin the parsing may happen more than once: syntax highlighter pass, psi tree building pass, external annotator.
  • markdown_example.md a file with 10,000+ lines containing 500kB+ of text.

Contributing

Pull requests, issues and comments welcome 😄. For pull requests:

  • Add tests for new features and bug fixes, preferably in the ast_spec.md format
  • Follow the existing style to make merging easier, as much as possible: 4 space indent, trailing spaces trimmed.

License

Copyright (c) 2015-2016 Atlassian and others.

Copyright (c) 2016, Vladimir Schneider,

BSD (2-clause) licensed, see LICENSE.txt file.