Add tests from Standard Markdown (renamed to CommonMark now)

[neclepsio]

Standard Markdown (www.standardmarkdown.com) CommonMark (http://commonmark.org/) is a proposal for a standard, unambiguous syntax specification for Markdown, along with a suite of comprehensive tests to validate Markdown implementations against this specification.

[dmitshur]

In general, I feel 👍 about the idea. Not sure how it'll play out in practice.

[dmitshur]

In general, I feel 👍 about the idea. Not sure how it'll play out in practice.

[jingweno]

👍

[jingweno]

👍

[buro9]

+1 here too.

[buro9]

+1 here too.

[russross]

Maybe this would be a good opportunity to create a version 2.0 of blackfriday. A few things I'd like:

1. Write a new interface for specifying options. Get rid of the all-caps constants at a minimum, but explore other open-ended ways of configuring the parser and renderer. Generally try to excise the remaining C idioms and make it more Go friendly.
2. Create a new set of convenience methods--like MarkdownCommon and MarkdownBasic, but thinking them out a bit more. For example, maybe one configured for trusted content for blog authors, another for comments and other untrusted content. Also, add string-based convenience methods to get rid of s := string(blackfriday.MarkdownCommon([]byte(input))) and the like. Maybe organize the convenience functions like the regexp package?
3. Drop support for XHTML, make everything HTML5.
4. Get rid of remaining sanitizing code and coordinate the interface with bluemonday so we have a good story for sanitized HTML.
5. Possibly add an io.Reader/io.Writer style interface. This interface could just be a thin wrapper that slurps everything into a buffer at least at first, but we could move to more of a stream-based model overall. This is tricky with footnotes, table of contents, etc., but since most rendered content doesn't use those features in practice, it could be a real improvement for many use cases. Maybe deprecate or remove the []byte interface if the new one works out.
6. Deprecate smartypants stuff, or at least take it out of the defaults. It's pretty clunky to begin with, and personally I have mostly stopped using it (I use digraphs in vim to type in the actual characters I want, or use HTML character entities instead). Maybe it should be a separate pass or even a different library? It's already incompatible with other smartpants implementations. I used LaTeX style dash rules, and re-wrote all the smart quote stuff to try to make better guesses (it's still pretty crap). I think people just need to learn to use their editors better, and browsers need to make it easy to use fancy quotes and dashes, etc.
7. (out of scope) I think moving to a PEG grammar would be awesome. The Go version of pegmarkdown was developed about the same time as blackfriday (I didn't know about it when I started this library or I probably would not have ported upskirt in the first place). It was slower and didn't have as many extensions (if I'm remembering right), so it wasn't as widely adopted, but I kind of think this was a mistake. I think some good optimization efforts could probably narrow the speed gap, and I think it is a more sane approach overall. I'm sad that the Common Markdown effort didn't go that route. Maybe there is a problem with it that I don't know about.

I think we are probably due to modernize blackfriday a bit, especially where the external interface is concerned. These are just a few of my ideas off the top of my head, but I'd love to hear from others. I think it makes sense to work on another branch where we can break the existing interface, then tag a version 2.0 when we are confident that the interface will be steady for a long time. Kind of a "Go 1" approach, except we already used v1.0.

[russross]

Maybe this would be a good opportunity to create a version 2.0 of blackfriday. A few things I'd like:

1. Write a new interface for specifying options. Get rid of the all-caps constants at a minimum, but explore other open-ended ways of configuring the parser and renderer. Generally try to excise the remaining C idioms and make it more Go friendly.
2. Create a new set of convenience methods--like MarkdownCommon and MarkdownBasic, but thinking them out a bit more. For example, maybe one configured for trusted content for blog authors, another for comments and other untrusted content. Also, add string-based convenience methods to get rid of s := string(blackfriday.MarkdownCommon([]byte(input))) and the like. Maybe organize the convenience functions like the regexp package?
3. Drop support for XHTML, make everything HTML5.
4. Get rid of remaining sanitizing code and coordinate the interface with bluemonday so we have a good story for sanitized HTML.
5. Possibly add an io.Reader/io.Writer style interface. This interface could just be a thin wrapper that slurps everything into a buffer at least at first, but we could move to more of a stream-based model overall. This is tricky with footnotes, table of contents, etc., but since most rendered content doesn't use those features in practice, it could be a real improvement for many use cases. Maybe deprecate or remove the []byte interface if the new one works out.
6. Deprecate smartypants stuff, or at least take it out of the defaults. It's pretty clunky to begin with, and personally I have mostly stopped using it (I use digraphs in vim to type in the actual characters I want, or use HTML character entities instead). Maybe it should be a separate pass or even a different library? It's already incompatible with other smartpants implementations. I used LaTeX style dash rules, and re-wrote all the smart quote stuff to try to make better guesses (it's still pretty crap). I think people just need to learn to use their editors better, and browsers need to make it easy to use fancy quotes and dashes, etc.
7. (out of scope) I think moving to a PEG grammar would be awesome. The Go version of pegmarkdown was developed about the same time as blackfriday (I didn't know about it when I started this library or I probably would not have ported upskirt in the first place). It was slower and didn't have as many extensions (if I'm remembering right), so it wasn't as widely adopted, but I kind of think this was a mistake. I think some good optimization efforts could probably narrow the speed gap, and I think it is a more sane approach overall. I'm sad that the Common Markdown effort didn't go that route. Maybe there is a problem with it that I don't know about.

I think we are probably due to modernize blackfriday a bit, especially where the external interface is concerned. These are just a few of my ideas off the top of my head, but I'd love to hear from others. I think it makes sense to work on another branch where we can break the existing interface, then tag a version 2.0 when we are confident that the interface will be steady for a long time. Kind of a "Go 1" approach, except we already used v1.0.

[dmitshur]

Item 4 is done now, thanks to #117.

[dmitshur]

Item 4 is done now, thanks to #117.

[rtfb]

I'm planning to fiddle with item 1 next.

[rtfb]

I'm planning to fiddle with item 1 next.

[kevinburke]

I'm curious how you plan to deploy 2.0 without breaking compatibility/behavior for existing library users (this isn't really specific to your library, but I'm curious for more data about how library users address it).

[kevinburke]

I'm curious how you plan to deploy 2.0 without breaking compatibility/behavior for existing library users (this isn't really specific to your library, but I'm curious for more data about how library users address it).

[rtfb]

I'm thinking about that too. On the one hand, we're taking opportunity to break everything with 2.0, so no need to preserve compatibility. On the other hand, are we going to keep all that stuff on another branch all this time?

[rtfb]

I'm thinking about that too. On the one hand, we're taking opportunity to break everything with 2.0, so no need to preserve compatibility. On the other hand, are we going to keep all that stuff on another branch all this time?

[kevinburke]

Fork the project to russross/blackfriday-2? Copy libs to a subdirectory?

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Tue, Sep 23, 2014 at 11:29 AM, Vytautas Šaltenis <
notifications@github.com> wrote:

I'm thinking about that too. On the one hand, we're taking opportunity to
break everything with 2.0, so no need to preserve compatibility. On the
other hand, are we going to keep all that stuff on another branch all this
time?

—
Reply to this email directly or view it on GitHub
#114 (comment)
.

[kevinburke]

Fork the project to russross/blackfriday-2? Copy libs to a subdirectory?

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Tue, Sep 23, 2014 at 11:29 AM, Vytautas Šaltenis <
notifications@github.com> wrote:

I'm thinking about that too. On the one hand, we're taking opportunity to
break everything with 2.0, so no need to preserve compatibility. On the
other hand, are we going to keep all that stuff on another branch all this
time?

—
Reply to this email directly or view it on GitHub
#114 (comment)
.

[tianon]

"gopkg.in" is an option worth considering for users who don't want to
follow to v2

[tianon]

"gopkg.in" is an option worth considering for users who don't want to
follow to v2

[mholt]

If I may chime in, I'd vote for just pushing v2.0 into your master when it's ready ("stable enough") - most gophers who want to control their dependencies do so already with their own tooling (mostly vendoring from what I've seen).

[mholt]

If I may chime in, I'd vote for just pushing v2.0 into your master when it's ready ("stable enough") - most gophers who want to control their dependencies do so already with their own tooling (mostly vendoring from what I've seen).

[s-oram]

@mholt Do you have a data source to back up your claim of most Gophers using vendoring?

[s-oram]

@mholt Do you have a data source to back up your claim of most Gophers using vendoring?

[yohcop]

@s-oram There is this design document at least:
https://docs.google.com/document/d/1Bz5-UB7g2uPBdOx-rw5t9MxJwkfpx90cqG9AFL0JAYo/view

[yohcop]

@s-oram There is this design document at least:
https://docs.google.com/document/d/1Bz5-UB7g2uPBdOx-rw5t9MxJwkfpx90cqG9AFL0JAYo/view

[rtfb]

Fun trivia: Blackfriday currently passes 224 of 617 CommonMark conformance tests out of the box. At a quick glance over the failing ones it seems that there are no major discrepancies, only a ton of minor cosmetic incompatibilities.

[rtfb]

Fun trivia: Blackfriday currently passes 224 of 617 CommonMark conformance tests out of the box. At a quick glance over the failing ones it seems that there are no major discrepancies, only a ton of minor cosmetic incompatibilities.

[weingart]

Is v2 still on the books?

[weingart]

Is v2 still on the books?

[rtfb]

Not only is it on the books, it is largely done. Import it via gopkg.in/russross/blackfriday.v2 and check out #218 for details. However, CommonMark thing was postponed until after 2.0 is out.

[rtfb]

Not only is it on the books, it is largely done. Import it via gopkg.in/russross/blackfriday.v2 and check out #218 for details. However, CommonMark thing was postponed until after 2.0 is out.

[rtfb]

I have pushed out v2-commonmark-testsuite branch. It contains the test suite and a little program to exercise it (it is external to the main test suite since hundreds of tests are expected to fail). I will be pushing some initial fixes there as well.

[rtfb]

I have pushed out v2-commonmark-testsuite branch. It contains the test suite and a little program to exercise it (it is external to the main test suite since hundreds of tests are expected to fail). I will be pushing some initial fixes there as well.