Request for Feedback: v2

[yuin]

I’m currently working on a prototype of v2. I’d appreciate any feedback or suggestions you have regarding v2 in this discussion.

You can try v2 from here.
As it is currently in beta, you'll need to run go get with an explicit version, such as github.com/yuin/goldmark/v2@v2.0.0-beta.1.

The motivation behind v2 is as follows (reproduced from the README.md):

v2 Motivation

goldmark was originally created with a focus on my personal goals.

• Extensible. You can easily add your own syntax to Markdown.
• Performance. Prioritize performance over semantically perfect AST.
• Focus on the purpose of converting to HTML.

Unexpectedly, goldmark has been used by many people.

goldmark has become a major Markdown parser in Go ecosystem.

In such a situation, there have been many requests regarding use cases that were not emphasized at the time of initial creation.

• Semantic analysis of Markdown documents using AST
• Use cases that use more detailed position information, such as LSP servers

In particular, as Markdown documents have come to be used as Lingua franca for AI, there is an increasing need to analyze Markdown documents semantically.For the same reason, there are also increasing use cases for generating Markdown documents rather than parsing them. The use of CLI in AI agents is increasing, also a growing need to convert to formats other than HTML.

Breaking changes to an extensible library like goldmark have a huge impact, as third-party extensions will no longer work. Therefore, I have avoided making breaking changes for a long time.

It has been more than 7 years since goldmark was created, and technical debt has been accumulating. In the meantime, the Go language specification has changed significantly, including the introduction of generics. I believe that the changes in use cases, represented by AI, are a good opportunity to fundamentally review the design of goldmark, and I have decided to make breaking changes.

v2 and v1 differences

• v2 focuses on building a more semantic AST. For that reason, it is slightly less performant than v1.
• v2 uses generics.
• v2 clearly separates the parser and renderer. This makes it easier to implement rendering to formats other than HTML.
• v2 allows you to programmatically build an AST. And you can render the constructed AST to another format.
• v2 has all nodes hold the start position. In the future, third-party extensions that support v2 are also expected to hold the start position.
• The core parsing algorithm is the same as v1. Third-party extensions must support v2, but the most complex parsing part can be used almost as it is.

[bep]

For Hugo, I have started to test this in gohugoio/hugo#15125 -- I will come back here with some more to-the-point feedback when I understand this better.

[ossenthusiast]

Could you write a migration guide for users to switch to v2?

[yuin]

Certainly. In addition, for extension developers, I am testing a simple LLM-based skill for migration locally. I applied it to goldmark-emoji and goldmark-meta to perform a migration, and the conversion was almost entirely correct.

[bep]

I have upgraded Hugo and some other Goldmark extensions to v2, and it was mostly smooth sailing.

Breaking: Attribute values are now textm.MultilineValue. In v1 it was any and could contain string, int, []int etc. I think this is a mistake, and I suspect this will become clearer once people start rendering Markdown to other output formats. But even if we get #563 fixed, parser.ParseAttributes trims any quotes, so there's no way to guess the intended type (e.g. number=42 vs string="42", and also slice=[32,43] vs string="[32,43]").

Some general comments/wishes:

• There's no way to pass in a parser.Context to Parse anymore. I found workarounds for my uses of this, but I do think this is a common and reasonably requirement. The same may be said about Render, but here we can add any context data (e.g. a context.Context) to the io.Writer.
• The new Pre/Post render hooks would have been a lot more useful it it ran around all of the node renders (I have a cool use case in mind) and not just the top level.

[yuin]

@bep Thank you for your feedback.

Breaking: Attribute values are now textm.MultilineValue

This is an intended change as mentioned in README#1, README#2

Reasons:

• Very few users or extensions relied on attribute values being anything other than strings.
• Most implementations other than goldmark use attributes that are string-typed.
• Since goldmark was released, there has been virtually no discussion about allowing attribute values to be anything other than strings.
• Given the origins of Markdown and CommonMark, I thought it would be preferable for attribute values to remain compatible with HTML attribute values. In that respect, the implementation is not yet complete. I still plan to add support for parsing values such as [1,2] as strings and for decoding HTML entities.

Even though the README has noted that the attribute syntax might change in the future, the change will still affect many users. For that reason, I think one possible approach would be to provide a way—such as through a build tag—to opt into the v1-compatible parsing behavior.

There's no way to pass in a parser.Context to Parse anymore

This was something I debated as well. Since IDGenerator is now used instead of IDs, goldmark itself no longer needs the ability to pass a context. However, if being able to pass a context to the parser and renderer is useful for Hugo, I'd be happy to add support for it.

The new Pre/Post render hooks would have been a lot more useful it it ran around all of the node renders (I have a cool use case in mind) and not just the top level.

Interesting. Checking for the presence of a hook for every node would likely have some performance impact, but it's certainly something I can experiment with. From that perspective, the current hook could also be considered a hook for the Document node, which would make the overall design more consistent.

[bep]

Interesting. Checking for the presence of a hook for every node would likely have some performance impact,

As this pre/post hook feature is new in v2, I would have just added 1 pre/post-hook for every node (including the top level document node). If you add a no-op hook pair if not set, then you don't have to check for the presence for every node.

Very few users or extensions relied on attribute values being anything other than strings.

I'm pretty sure you're wrong about the "very few users" argument.

Since goldmark was released, there has been virtually no discussion about allowing attribute values to be anything other than strings.

Goldmark has supported non-string attributes since attribute support was added to Goldmark, why would we need to discuss it?

Given the origins of Markdown and CommonMark, I thought it would be preferable for attribute values to remain compatible with HTML attribute values

This thread in the CommonMark forum discussing attribute syntax in Markdown was started in 2014. Today, 12 years later, they're still going back on forth on how it should look. Given that context, we might as well be talking about papyrus documents. The JSON¹ like Markdown syntax that Goldmark supports is the best and most standard "node configuration" available, and, especially considering non-HTML output formats, this is not just about styling. But this is your project, so up to you.

I still plan to add support for parsing values such as [1,2] as strings and for decoding HTML entities.

If the above also includes foo=32 where 32 will be a string value and you added e.g. textm.MultilineValue.WasQuoted(), I think we could work around it.

Footnotes

1. Imagine removing all non-String types from the JSON spec. ↩

[yuin]

@bep

I'm pretty sure you're wrong about the "very few users" argument.

Using the term "users" was indeed a poor choice; it was an incorrect expression, partly due to my limited English proficiency.
Perhaps "product" would have been the more accurate term. I suppose the point is that while the number of products involved might be small, the number of users is large (which is, of course, because Hugo has a large user base :) ).
Therefore, I believe that even when introducing breaking changes, it would be best to provide a way to maintain compatibility—such as using build tags.

Goldmark has supported non-string attributes since attribute support was added to Goldmark, why would we need to discuss it?

I am discussing this in the context of CommonMark. Since goldmark is a CommonMark parser, the aim is to align it as closely as possible with generally accepted practices.

However, as you mentioned

This thread in the CommonMark forum discussing attribute syntax in Markdown was started in 2014. Today, 12 years later, they're still going back on forth on how it should look.

It is certain that discussions on this matter have stalled. It is also a fact that, even 7 years after goldmark's release, there has been hardly any increase in CommonMark implementations that support JSON-style attribute values ​​like goldmark does. In short, goldmark-style attribute values ​​represent a syntax with low portability—even if that syntax happens to be excellent.

But this is your project, so up to you.

Thanks :)

If the above also includes foo=32 where 32 will be a string value and you added e.g. textm.MultilineValue.WasQuoted(), I think we could work around it.

For now, I do not intend to define such a function. In HTML and DOM, both <div data-width=10> and <div data-width="10"> are retrieved as strings; we explicitly convert them when a numeric value is required. Would treating attribute values ​​in Markdown the same way be problematic?