Implement a 007 parser in 007

[masak]

Something tells me that Sooner Or Later™ we're going to need something out of parsing that Perl 6 grammars won't easily provide for us (gasp!). At that point, we'll wish the parser was bootstrapped and written in 007.

So let's write a 007 parser in 007 before that happens.

We may never actually swap it in for the real thing. Depends what kind of bootstrapping crisis we actually encounter. But it'd be an interesting exercise in any case, and a way to find where 007 as it stands is lacking right now.

The base assumption is that a recursive-descent parser can be hand-coded with enough if statements, subroutines, index/substr/charat, and pure bloody-mindedness.

But we should also look out for opportunities: can we simplify the parser using macros somehow? What if we had the case expression syntax of #34? What if we thought in terms of parser combinators?

There are currently 189 tests in the test suite. Of these, 34 are parses-to and 27 are parse-error. So, 61 tests could form the basis for this 007 parser in 007. Probably quite a good starting point.

[masak]

Before #25 lands, we can still write the parser. But I think it'll be hard to get the kind of hooking/extensibility that would make a 007-in-007 parser strictly more flexible than a Perl 6 parser without objects.

(What kind of extensibility am I thinking of? The kind where we say "these things take type annotations", and now we'll want to swap in a new grammar that derives _007::Parser::Syntax. I guess we could EVAL the new parser and build all its rules from strings or something... but it feels like a bad fit.)

[masak]

To give a further indication of what it is we need from the parser: usually a parser accepts characters, or tokens — some kind of textual-ish input. We need a parser that sometimes takes it on faith that it will eventually get a certain Qtree component at a certain position.

my x = 5;
{{{stmt: Q::Statement}}}
say(x);

Call it "abstract parsing", in analogy with abstract interpretation. We don't know what goes in the middle statement in that code, but we do know that the code has three statements. We know that whatever gets passed to that slot will be type-checked, and so whatever ends up in that slot will be a statement. Finally, we know what state to put the parser in after that slot so that it's prepared for a new statement (as opposed to, say, an infix operator).

[masak]

Currently blocking, sort of, on #174.

[masak]

Nah, I disagree with my former self. Regexes are nice-to-have when parsing, but not an absolute necessity.

[masak]

One lesson from implementing the parser in Perl 6 is that it's actually fairly closely bound to the 007 runtime. Any declaration of variables, subs, classes etc. will go through the runtime, so will type lookups and macro expansions.

Issue #51 talks about implementing a runtime in 007, and that would of course be swell, but there's another option that doesn't depend on a 007 runtime existing. Namely, make this work and faux-import the Perl 6 runtime, wrapped into a 007 façade object.

import runtime;