Custom rule matcher / nested grammars

[filipjezek]

My language supports nested blocks of other languages - something like JS in a HTML script tag, or a templating language like PHP. What I'm looking for is a way to switch to a different grammar during the parsing.

nestedBlockStart {
  const { result, rest } = nestedGrammar.parse(remainingText);
  setRemainingText(rest);
  return result;
}

It is not possible to just find the nested block start and end and then parse the contents later, as the nested block may include false block ends (e.g. as strings etc).

...outer text
nested {
  "}}}"
}

Something like custom rule matchers would likely solve this problem as well, but based on #312, those are not supported at all.

So my question is: is this currently possible? If not, is it viable to implement support for either custom matchers or manipulating the remaining input? I am very much open to doing that myself.

For the context, I have already achieved this with jison, but PEG grammars are a better fit for the language. I have already spent several hours rewriting the parser into peggy before realizing I don't know how to deal with the nesting.

[filipjezek]

Turns out the input variable available to semantic actions is a direct reference to the parse input, not just a local parameter of the semantic action. I was thus able to make the following work:

Expression = Langswitch

Langswitch = blockStart blockEnd

blockStart = "nested {" {
    input = 'nested {}';
    return 'foo';
}

blockEnd = "}"

// accepts:
// nested {}
// nested { anything }

Seems quite fragile though.

[hildjj]

If you look at cbor-edn you'll find an example of calling a rule by name, based on some context. As the comment says Peggy internals. Don't do this yourself.

However, there might be an idea in there for you that we could generalize to make documented and supported.

[filipjezek]

While interesting, I don't see a way to modify your approach to solve my problem. The main difference is that in my case, the exact length of the nested expression is unknown until it is actually parsed. I can run the nested parser and then modify the input variable and it works (code) as long as I turn on caching. Without caching, this all breaks, as the parser may match the rule multiple times before deciding to keep it (and each of the matches would try to modify the input). For this reason, I don't really think that this is something that can be generalized and explicitly supported.

The only way I see to make it work and have it be reasonably 'clean' is to use some kind of custom matchers, like so:

Expression = Langswitch

Langswitch = blockStart nestedText:{{
    const { result, rest } = nestedGrammar.parse(input.slice(offset()));
    return input.slice(offset(), -rest.length); // the matched text
    // I could either run the nested parser again in the semantic action
    // or I could store the result somewhere and just retrieve it then
}} blockEnd

blockStart = "nested {"

blockEnd = "}"

Anyway, I'm fine with what I have right now, so I'm okay with closing the discussion as resolved.