Add Markdown grammar tests

[svetlyak40wt]

These tests will help to improve parser. For example, nested lists are't working properly, but I'm afraid to broke something while fixing them. Thus tests should be added before any real changes to the grammar.

[svetlyak40wt]

@3b please, review this pull-request?

[3b]

Thanks, tests will be nice to have. I'll try to take a closer look at them soon.

Are the tests based on any particular markdown spec, or just trying to capture current behavior of 3bmd?

[svetlyak40wt]

These tests just capture current behaviour.

As a next step, I'd like to rewrite the part responsible for indentation handling and fix how blocks nested into the list items. For example, right now list items don't allow to embed code blocks or other lists.

[svetlyak40wt]

As a long term target, I'd like to make this library conform to CommonMark specifications. But this might break backward compatibility for somebody. What do you think about this?

[svetlyak40wt]

By the way, talking about tests. They will be much more useful when checked automatically for pull requests. If you'd like, I can set up a workflow for GitHub Actions that will trigger test checks on pull requests.

I even have a special tool for the lisp projects, to write Lisp code instead of YAML: https://40ants.com/ci/

[equill]

As a user of this library, I'm in favour of having it conform as completely as possible to the spec, and at this point it would cause minimal pain.

I'm curious about what breaking changes would be involved, though. It could be worth creating a separate issue for this, and having this discussion there.

[mdbergmann]

I would also be curious about this as my blog web page depends on it.

[svetlyak40wt]

I would also be curious about this as my blog web page depends on it.

Then please, search TODO in the diff of this pull request. I've market places which definitely should be fixed and will work the other way after the fix.

[mdbergmann]

I'm not sure about the visual result. I'd need to clone when done in your fork and have a look I guess.

Which spec was the current implementation based on?

[3b]

By the way, talking about tests. They will be much more useful when checked automatically for pull requests. If you'd like, I can set up a workflow for GitHub Actions that will trigger test checks on pull requests.

Yeah, will definitely have it running automatically once it is merged. You can add it to the PR if you want, though I'll probably end up rewriting it at some point to match how i want CI things to work. The yaml generator looks interesting though, will have to look into that to see if I can use that to abstract things better across the various things i want to test.

As a long term target, I'd like to make this library conform to CommonMark specifications. But this might break backward compatibility for somebody. What do you think about this?

As a user, moving to CommonMark sounds like a good idea, though my use is small enough that dealing with any breakage is not a problem. As a maintainer, I also agree with the concern about breaking compatibility. Not sure I have any strong opinions either way though.

Maybe deprecate 3bmd and move active development to a CommonMark fork, if there seems to be enough interest (in the form of people willing to help with that, since I don't have much energy available for it in the near future)? Would also give an opportunity to try to clean up the AST and/or try to make it compatible with other text markup parsers and html/pdf/whatever generators.

Another option would be to try to support both with variants of the same parser, though I suspect that would probably be error prone and more work than it is worth.

Or in between, just duplicate the parser code within 3bmd and have a separate system for the CommonMark parser, without any extra refactoring/cleanup.

Which spec was the current implementation based on?

Current implementation was written before CommonMark existed if i remember correctly, and was based on the original markdown description at https://daringfireball.net/projects/markdown/syntax along with looking at what the various popular implementations at the time did for extensions.

[svetlyak40wt]

I've pushed the commit adding GitHub workflow: 11bddf7

[svetlyak40wt]

Probably, I'll extract parser code into a separate ASDF system commonmark-parser and will work on refactoring there until it will conform most of the CommonMark tests.

[100phlecs]

an opportunity to try and clean up the AST and/or try to make it compatible with other text markup parsers and html/pdf/whatever generators.

Maybe CommonDoc could be the foundation toward this effort.
Incidentally I've been trying to find an org2html for org-mode files within common lisp.

I was planning to use this markdown library as a reference and make my own transformer, but if there is an effort to consolidate, then CommonDoc concept seems solid. It would be interesting to be able to translate all different types of documents, instead of having individual transformers

[svetlyak40wt]

@100phlecs this is a good idea. Actually, I'm already using 3bmd's grammar rules to convert Markdown to CommonDoc.

https://40ants.com/commondoc-markdown/

You can use this library, because I'm planning to make it to evolve it and to support CommonMark as well.

[svetlyak40wt]

@100phlecs but if you really prefer to use org-mode files, I'd recommend to look at the code I've made to publish my blog "Lisp Package of the Day". It renders org-mode to HTML and RSS.

Here you'll find the code written in literate style:

https://github.com/40ants/lisp-project-of-the-day/blob/master/readme.org#here-is-how-a-website-is-rendered