Migrating Syntax to ST4?

[tajmone]

@polyglot-jones, I've been considering editing the syntax to enable ST4 feature, because the new match patterns branching features would allow us to better handle some of the more complex AsciiDoc constructs — we can test for a given series of expectations, and if we find a failing condition revert to the previous branching point. I believe that would allow us to overcome many of the current limitations, and fix some elements that currently break-up the syntax in some edge cases.

But then, by doing so (i.e. by setting version: 2) the package won't be backward compatible with ST3.

Which brings up the issue of the repository being named ST3-Asciidoctor, and whether it should be renamed to ST4-Asciidoctor, or just left as it is.

In theory, renaming the repository from GitHub's WebUI should allow URL redirection, without breaking any previous installations. This should work also with Git operations.

What's your view on this? Did you already migrate to ST4?

The problem is that I have no idea how many people are currently using this package, and whether migrating it to ST4 would leave them with a broken package (i.e. if they are still using ST3). Since this is not a registered package, I can't exploit package update messages to notify end users of upcoming changes when the package is updated — I actually believe the package currently has to be manually updated via Git, and that PackageControl doesn't handle any updates for it.

[polyglot-jones]

No, I did not migrate to using the new branch/fail features, yet. I say we wait a bit and let the dust settle, but eventually assume that anyone savvy enough to be using this package is also inclined to take advantage of ST4.

As far as renaming from ST3 to ST4 goes, I thought I read that they're actually doing away licensing based on major releases (and therefore doing away with major-release numbering). Going forward, they want to downplay the "ST4" and just go with "ST". In fact, I'm not sure there's any reason to mention ST at all, since it's implied.
I'd also like to drop the "Asciidoctor" and just go with AsciiDoc, since there is nothing that is Doctor-specific here.

Bottom line: I suggest we take a page from the MarkdownEditing playbook, and go with "AsciidocEditing". They have 537K installs to date, so they must be doing something right.

[tajmone]

No, I did not migrate to using the new branch/fail features, yet. I say we wait a bit and let the dust settle, but eventually assume that anyone savvy enough to be using this package is also inclined to take advantage of ST4.

The we'll have to lower our goals considerably in terms of how far this syntax might go. Before ST4, I believe that the only way to produce a good AsciiDoc syntax would be via LSP (which I still do), but then when I say the new ST4 feature of branching syntaxes I kind of regained some hope that we would be able to see a decent AsciiDoc package for ST.

Going forward, they want to downplay the "ST4" and just go with "ST".

Maybe so, but right now everyone is calling it ST4.

In fact, I'm not sure there's any reason to mention ST at all, since it's implied.

I agree that if were to rename the repository (which I would do, e.g. if/when I'd be submitting it to PackageControl) I'd probably just call it ST-Asciidoc, since PackageControl has dedicated tags regarding which ST versions are supported, so there would be no risk of a ST3 user installing it by mistake. But right, being just a repository, it's another matter, and we have multiple such packages in the wild, the original being for ST2, and various forks for ST3, so there's a confusion issue right now.

I'd also like to drop the "Asciidoctor" and just go with AsciiDoc, since there is nothing that is Doctor-specific here.

The problem is that the package covers some Asciidoctor specific features, which are not present in AsciiDoc Python, and there's also an AsciiDoc package designed around AsciiDoc Python. I do agree that the syntax (i.e. in the editor) should be indicated as just AsciiDoc (for there's only the AsciiDoc syntax, no Asciidoctor syntax), but the Asciidoctor term is a reminder of extra features being covered by the package.

But then, again, I recently took on a book project that was created for AsciiDoc Python, and wanted to work on it with Asciidoctor, and I soon discovered that there were indeed lot's of differences in terms of how tolerant each syntax was to minor formatting mistakes (i.e. parsing tolerance), as well as some apparently different constructs, so I had to do quite some conversion work to make the original book work with Asciidoctor Ruby. So I wouldn't really vouch that there's only one AsciiDoc syntax, since implementations do have significant differences.

Bottom line: I suggest we take a page from the MarkdownEditing playbook, and go with "AsciidocEditing".

That would be a good choice, either that or AsciidoctorEditing. The fact is that ST is a Python-first tool, so I fear that if we use just AsciiDoc most users will assume it's the Python implementation (Python and Ruby being two competitor poles at the extremes of a continuum of similar languages).

They have 537K installs to date, so they must be doing something right.

Sure, they do: they cover all the major Markdown flavors in a single package, which means they are not bound to specify "GFM" or "Common-Mark", etc.

And, BTW, I think the reason this is such a good package is because its maintainers had the gut to disregard all the absurd scoping rules and guidelines from ST official documentation, and make their own choices. If they had followed the (rather poor) ST docs, they probably would have never produced a working package.

With AsciiDoc, we face very similar problems — in fact, the original package didn't abide by the official scoping rules either, although I tend to think that the former package made better arbitrary scope naming choices.

[tajmone]

ST4's pop: <n>

Another very useful new syntax features introduced by ST4 is the pop: <n> match pattern:

pop — Pops contexts off the stack. The value true will pop a single context.

An integer greater than zero will pop the corresponding number of contexts. 4050

The pop key can be combined with push, set, embed and branch. When combined, the specified number of contexts will be popped off of the stack before the other action is performed. For push, embed and branch actions, the pop treats the match as if it were a lookahead, which means the match will not receive the meta_scope of the contexts that are popped. 4075

whereas ST3 only allows popping out a single context (pop: true), which requires adopting some rather ugly hacks to bail-out of entangled contexts.

It's precisely because of the new Syntax features that I decided to upgrade to ST4 straight away, because they made me hope that it would be possible to cover complex syntaxes like AsciiDoc, without having to resort to LPS (surely, without LSP it won't be possible to fully cover AsciiDoc, but at least it should be possible to cover it fairly decently).

@polyglot-jones, I don't really visit the ST forum very much lately (for various reasons), so I was wondering if you have a rough estimate on how many ST3 users are still using ST3 vs those who migrated to ST4.

Being able to use ST4-specific Syntax features could really speed up fixing this package, but it also would make it mostly backward incompatible with ST3 — it would still work, but ST4 specific constructs and settings would be ignored, possibly leading to syntax breakage in some contexts (e.g. a pop: 2 would probably be treated as a pop: true, or result in an error, and I have no idea how new patterns like branch would be handled).

[polyglot-jones]

I was wondering if you have a rough estimate on how many ST3 users are still using ST3 vs those who migrated to ST4.

I do not, but you make a compelling case that the syntax advancements in 4 are exactly what we need. On top of that, (a) the only real complaints I see are when an unsupported plugin stops working in 4, and (b) that's countered by the fact that you can run 3 and 4 side by side if necessary. So, yeah, let's officially make ST4 required.