Outputting Markdown

[twardoch]

Dear @waylan ,

my apologies for guerilla-restating my question. It’s likely that I have not expressed myself clearly which led you to closing and locking #545

I asked two questions, but perhaps I didn’t explain them sufficiently clearly. Please allow me to rephrase them for clarity. If you still consider this outside of your interests, and close this issue, I won’t raise it any further. I’m not interested in trolling.

I am interested in Markdown (not CommonMark or any other younger siblings). I’m not advocating to change the parsing behavior of PyMarkdown’s input format at all.

On the contrary: I think Markdown (as it is, fully disregarding its “siblings” etc.) is so useful that I’d like to keep it as a writing, editing and archiving format for as long and possible.

Which is why I wonder if you think it’d be reasonable to allow Markdown as an output format of PyMarkdown, in addition to the HTML variants.

John Gruber writes:

Markdown’s syntax is intended for one purpose: to be used as a format for writing for the web.

Markdown is not a replacement for HTML (...) The idea for Markdown is to make it easy to read, write, and edit prose. HTML is a publishing format; Markdown is a writing format. Thus, Markdown’s formatting syntax only addresses issues that can be conveyed in plain text.

John made an excellent point there, and to me, the great value of Markdown is the ease of read, write and edit. Yes, Markdown does allow portions of HTML, but to me, Markdown is not really about HTML. It’s about authoring. The fact that in the end, people may want to convert it into HTML, is, in my view, secondary.

Output: Preprocessed Markdown

You write:

Preprocessors munge the source text before it is passed into the Markdown core. This is an excellent place to clean up bad syntax (...)

I discovered today that it’s trivial to use PyMarkdown to convert, let’s call it, my “markdownish input” into “proper Markdown” — by just using the preprocessors like this:

def getPreprocessedMarkdown(md_text, md_opts): 
  md = markdown.Markdown(**md_opts)
  md.convert(md_text)
  return u"\n".join(md.lines)

This, in a way, simulates the output_format = "markdown" behavior I was asking about. It just runs the preprocessors and gives me a “better” Markdown back — specifically because I want to use Markdown as a writing format and keep its clean form even if my preprocessing extensions allow me to bend the spec in the input.

Output: Pure Markdown

The PyMarkdown extensions system is truly fantastic, and it’s very productive in a controlled environment, but when used extensively, it can actually take the documents further away from John Gruber’s original syntax. Many PyMarkdown extensions add more “plain-text-style” syntax constructs, but this comes at a cost of making the Markdown documents less portable. And I’m looking for ways to make them more portable, or, closer to the original Markdown.

So I wonder if you’d think it’d be possible to have PyMarkdown work so that it takes as input a “markdownish document using custom extensions” document, and outputs a “pure Markdown” (“pure” by John Gruber’s original implementation standards, or as close to it as possible).

In other words, I’d like to turn “unportable Markdown-like syntax” into “portable Markdown” which then would not require extensions to compile, and could perhaps at a later stage compile even in the original markdown.pl (or at least with “extension-less PyMarkdown”).

In other words, all constructs that are provided by custom extensions would then be replaced by HTML, but the “core” Markdown constructs would be kept intact.

In my opinion, such behavior could bring the functionality of PyMarkdown closer to the spirit of John Gruber’s original Markdown spec.

I’m not informed enough if this is possible to do with PyMarkdown with little work, or would require significant changes. If the latter is the case, I don’t think I’d advocate doing it — because perhaps the use-case I’m thinking of is too limited.

Anyhow — I just wanted to ask my questions in a more clear way. If you still prefer not to discuss this any further or think this is indeed a duplicate of #338 , please feel free to close and lock this issue, and I’ll be no longer asking along these lines.

Many thanks,
Adam

[facelessuser]

@twardoch, just knowing how extensions work, what you are asking is very difficult. There is a degree of indirect interaction of extensions. What extension consumes what first matters.

I would say you could create a plugin that disables canonical output and only processes non-canonical Markdown, but that would not give you what you want for the reasons listed above.

You could create alternative extensions of canonical Markdown syntax that just finds and preserves canonical syntax, but some of the other "markdownish" plugins maybe rely on those things getting processed so they can act upon them.

If you want Markdown that is portable, I'd stick to canonical format. If you want custom Markdown, you have to stick to tools that understand that making it less portable.

I'm curious why you feel a need for a multi-stage Markdown compiler process? Is there some feature you are trying to take advantage of in different compilers?

[waylan]

So you want to output different formats? Sorry that was a lot of words for a rather simple request. Just making sure I didn't miss the point.

As it turns out, this is possible in theory, although not particularly practical. You could subclass the markdown.Markdown class and add an additional serializer which would serialize the ElementTree object to something other than HTML. The problem is that the postprocessors would no longer work as they act on the serialized HTML text. Sure you could recreate them to work on your new format, but then you also need to take the preprocessors into account. For example, the raw HTML preprocessor simply stores raw HTML for later retrieval by a postprocessor, which reinserts them into the document. You would need to parse that raw HTML and convert into your new format in your custom postprocessor.

In the end, it would be easier to take the entire HTML output and use some other tool to convert that into whatever format you desire (what the above-mentioned postprocessor would be doing). Python-Markdown is a Markdown-to-HTML tool and nothing more. I'm not interested in changing that. If you want to do other things, then chain multiple tools together--take the output of one and pass it as input to the next. At least that's the approach I take. That way each tool can specialize in its specific area.

[twardoch]

Thank you @waylan and @facelessuser for your responses.

Yes, one of my loose goals is to have a "complex Markdown to simple Markdown" or "private Markdown to public Markdown" route, i.e. have an output that is Markdown with HTML insertions for constructs that cannot be expressed in core (extension-less) Markdown, but to keep “core Markdown” fragments as Markdown.

One use case would be when you use the Github Pages system or Github files such as README.md, which are expected to be in Github-Flavored Markdown. In such case, core Markdown constructs and a few others are supported, but other constructs need to be provided as HTML, because those Markdown files will be compiled into HTML using a Ruby engine. Yet in my source files, I might want to have a more private flavor of Markdown with extended syntax elements.

I'll play a bit with PyMarkdown to see what it can do if I start removing things from the OrderedDicts of processors and patterns. Perhaps it’ll get me somewhere. If not, I always have the choice to compile my Markdown into HTML using PyMarkdown and then use something like html2text or pandoc to turn the HTML back into Markdown.

Thanks!

[waylan]

@twardoch your use case is exactly the sort of thing Pandoc was made for. I understand that you might want to use Python-Markdown (it is not "PyMarkdown" but "Python-Markdown") for some of its extensions, etc. But yeah, I would then use Pandoc for everything else.

Of course the problem with this is that there is a potential for data loss. You may be interested to know that Calibre (an e-book conversion utility among other things) previously used Python-Markdown under the hood to convert between different e-book formats. Markdown was the common format. It converted from Format A to Markdown to Format B (or vise versa). However, there was a lot of data loss. Anything that couldn't be expressed in vanilla Markdown (plus a couple extensions they used) didn't make it through, even if the formats at both end supported the same feature. At some point they switched the intermediary format to HTML which is a lot more feature rich (and probably the right move IMO). Even so, they are still piping content from one piece of the puzzle to the next as they effectively diagram in their docs. There is no "Format A to Format B" conversion. Its "Format A to Intermediary Format" and after some additional processing, "Intermediary Format to Format B". The docs explain it all in a lot more detail. And the entire project is opensource if you'd like to see how they do it. At the very least there are some good concepts to imitate from a very mature and well maintained code base.

[twardoch]

@waylan Thanks, this is valuable info. Data loss is indeed one concern I have, which I why I would prefer to avoid pandoc, since it uses a somewhat simplified AST model. Python-Markdown is my preferred environment because of its extension model, because I’m most comfortable with Python, and because I feel that I have more control over the entire process. Once again, thanks for clarifying your own intentions, and sharing some useful extra details.