Provide packages with a way to handle raw content

[alerque]

The way the parser works, we currently have to tell the parser ahead of time what tags contain raw content that needs to be passed through to an external handler rather than recursively parsed for more SILE commands. This works great for our builtin \script{} command. It can also be amended quite easily as I've been assuming we would de for math (#220), Lilypond (#435), TikZ (#437), SVG (#1091), etc. if and when any of those things were bundled with SILE.

This does not work out very well for things like @ctrlcctrlv's texmode package (see discussion in #947) that we do not plan to bundle. In fact it means 3rd party packages are at a distinct disadvantage to bundled ones in what they can and can't do.

There is a workaround, but in requires invoking SILE with special arguments:

$ sile -e 'SILE.inputs.TeXlike.passthroughCommands.foo = true' -d foo test.sil

We should either think of a way for packages to inject their requested handling early (I can't think of one other than using two parsing passes) or make a more convenient interface for this and document it for use by 3rd party packages.

[Omikhleia]

(See #675 (comment) for the original context).

Instead of cramming more and more special commands into SILE.inputs.TeXlike.passthroughCommands, maybe a dumb idear, but why don't we modify the parser to accept (besides math and script, admittedly) only one single 'inline-escaping' command, say \inline[format=name, other options]{pass-through stuff}. We then have the pass-through content and the extra options ready to be provided to some package.

Note that since we are at parser level, we could even decide for something that doesn't look as a real SILE command, so it could as well be @inline(name)[other options] or whatever (= think to the 3-backslash escape in Markdown, or the <![CDATA... esacpe in XML) and not necessarily use braces, but perhaps custom begin-end markers (= think to shell script escaping e.g. cat << MEH ... MEH)...

But for the sake of simplicity here, let's admit we may prefer using a SILE-familiar syntax...

Eg.

\svg[src=somefile.svg, width=dimension}

vs.

\inline[format=image/svg+xml, width=dimension]{<svg>....</svg>}

Or even:

\inline[format=base64:image/svg+xml, width=dimension]{PHN2Zz4uLi48L3N2Zz4=}

Another possible use:

\inline[format=text/x-lua]{some Lua code}

(And let's thus have a real verbatim thing, with possibly a package that even does syntax highlighting for the said language)

Packages could "register" a callback function for supported types. Say, from the svg package, SILE.registerInlineHandler("image/svg+xml", function() ... end)

Interestingly, a base64 package in that scheme would not need any user servicing commands, but would register its own function for decoding the content and invoking the next inline handler.

In brief:

• We just extend the parser with one special all-purpose inlining command. Impact on the parser is quite minimal and well separated from the rest.
• SILE core just have to provide a fairly simple registration and mapping API.
• 3rd party packages can thus register their callbacks when loaded, and get invoked back when an inline content needs them to act upon raw content.

Am I missing something?

[alerque]

An interesting idea. Off the top of my head I don't think it would play nice with the XML or other input methods, it would only work with SILE syntax where we define the grammar. As such the ASTs for different inputs would diverge. I'll have to sleep on that though.

Another idea might be just a keyword/macro trigger built into the language that adds/removes "passthrough" mode to specific commands via a callback in the LGEG grammar at runtime:

@passthrough{ftl}

A similar command could be used to disable passthrough mode. This would obviously need to be part of the document before any such tags were encountered.

[alerque]

Links to discussion in PR #675 that veered off into what this issue is about:

• Leverage Fluent and Unicode CLDR for localization #675 (comment)

  We might actually need more flexible ways for extending the parser, to be able to have package declare some hooks - true verbatim, inline scripts with syntax highlighting, inline SVG or any other text-based graphics (e.g. PlantUML and charts), etc. come to mind.

• Leverage Fluent and Unicode CLDR for localization #675 (comment)

  True, but as of yet I haven't figured out how to do it with a one-pass LPEG parser. By the time we get to the package load commands in the document the parsing grammar is already defined and it is too late for a callback to change it. I haven't wanted to go down the multi-pass rabbit hole yet. [...] PlanetUML (I can't find the issue, maybe this was in my head) and SVG (Allow inline SVG content #1091) have come up before, as have MathJax (Support Math — somehow (general discussion of approaches and status) #220), Lilypond (Add Lilypond support #435), ImageMagick (Add image package that filters raster resources through ImageMagick #436), and TikZ (Add TikZ support package #437). The only alternatives I found to that problem to allow packages to require package loading on the command line before loading the document or do to multi-pass parsing, both of which have huge downsides. See Provide packages with a way to handle raw content #1092.

• Leverage Fluent and Unicode CLDR for localization #675 (comment)

• Leverage Fluent and Unicode CLDR for localization #675 (comment)

• Leverage Fluent and Unicode CLDR for localization #675 (comment)

[alerque]

Having slept (poorly) on this, one downside I can see to my own previous suggestion about the parser finding a keyword to enable raw modes is that it would be limited to the primary document or segment being parsed. This would mean that a package could not enable a new tag to be passthrough for the main document. It could probably enable it for itself and any future strings that get parsed, but by the time the package include happens the primary document will already be parsed.

[alerque]

Clarifying usage for the -I argument and if my proposal in #1373 will make it just a little bit easier to get a path to allow people to add their own RAW commands.

Another note is that if the TOC iteration issues is fixed properly (cf. issue #230 and PR #593) then we would have a way to restart processing after one pass. This would possibly enable seeing a library load that added a raw tag, then re-parse the input with the new function(s) available. This might run into some issues with syntax validation on the initial pass, but a failed pass could also be a good time to restart if we had new parser rules registered. Somewhat inefficient maybe but ergonomic.

[alerque]

I expect the next release to have not one but 2 ways to handle this, one with the \raw command in #1347 and one using the new --requires CLI flag that will load a module early enough that it can modify the parser before it runs and hence allow modules to define their own raw commands.

[alerque]

Re-opening because the PR that closed this is the less direct of the two solutions slated for v0.14.0, I'll let the more direct answer be the final closer here.