Have a mechanism to allow arbitrary (undeclared) identifiers in expressions

[masak]

In the action method term:identifier, there's this wonderful piece of code:

my $name = $<identifier>.Str;
if !$*runtime.declared($name) {
    my $frame = $*runtime.current-frame;
    $*parser.postpone: sub checking-postdeclared {
        my $value = $*runtime.maybe-get-var($name, $frame);
        die X::Macro::Postdeclared.new(:$name)
            if $value ~~ Val::Macro;
        die X::Undeclared.new(:symbol($name))
            unless $value ~~ Val::Sub;
    };
}

Actually, for the purposes of this issue we can ignore the postponement logic and the check for postdeclared macros; the code then becomes much simpler:

if !$*runtime.declared($name) {
    die X::Undeclared.new(:symbol($name));
}

In other words, undeclared variables cause the compiler to abort with X::Undeclared.

What if we could selectively turn off this check? Or, even better, what if we could selectively let some identifiers through? I think that would make a big difference for DSLs, to enable stuff like this:

myfor [1, 2, 3] { say(it); }    # 'it' supplied by the `myfor` macro

say(each([1, 2, 3]);          # 'each' supplied by the `each` pragma, see #158 

The details are not clear to me, but it just occurred to me that we can do this. And we probably should.

It's funny, I've used "Perl 6 has a very particular intertwining between parser and static checks" as an argument many times, but I've never considered doing the above, which seems like it'd solve quite a few problems.

[vendethiel]

I think it goes well with Racket's define-syntax-parameter, I remember talking about it a while ago.

[masak]

I think it goes well with Racket's define-syntax-parameter, I remember talking about it a while ago.

It's a humbling feeling to read through a manual page and recognize that it's well-written, but still not understand much of it. I think this is due to me not having enough Racket context, though. Here, let me just throw "study Racket" on top of my TODO pile, right after "finish the 'Let Over Lambda' book"...

[masak]

Another raw/undigested comment about identifiers: in quite a few built-in and user-defined syntactic constructs, the presence of an identifier is a declaration. We might want to have a nice way to specify that too. In that case, there would be three modes:

• The identifier is a use/lookup (default)
• The identifier is just an unhandled symbol (as detailed above)
• The identifier declares itself in some block

Though the devil is in the "some block" details, I fear. Often enough the block in question is not the immediately surrounding one, but perhaps the next one down. (Examples: sub/macro parameter lists, for/if arrows, catch clause.)

[vendethiel]

It's a humbling feeling to read through a manual page and recognize that it's well-written, but still not understand much of it.

Sorry, it's a bit hard to digest without having read some racket macros thing (like the Fear of Macros tutorial).

It's basically dynamic variables (Perl 6's * twigil), but for macros. The example used is the "anaphoric if":

(aif condition (+ 3 it)) ;; "it" is bound to the condition

The mechanism is used this way:

(define-syntax-parameter it ;; define "it" is a special syntactical construct, which some macro will define
  (lambda (stx) (raise-syntax-error #f "use of it out of anaphoric macro" stx))) ;; default value, if you try to use a bare "it" outside of an anaphoric macro

-- then you define the macro:
(define-syntax aif ;; anaphoric if
    (syntax-rules () ;; this () is the list of identifiers to match literally. none here
      [(aif test then) ;; the macro syntax
       (let ([t test]) ;; the result: first evaluate test
         (syntax-parameterize ([it (syntax-id-rules () [_ t])]) ;; then bind "it" to t's value
           (if t then else)))])) ;; and generate an "if" node

[vendethiel]

let's make this Perl 6, to try to be clearer ...

my $?*it = Failure.new("not in an anaphoric construct");
macro aif($condition, $body) {
  quasi {
    my $?*it = {{{$condition}}}; # dynamically scoped for the rest of the quasi block.
    if $?*it { {{{$body}}} }
  }
}

aif 3 + 3, {
  say $?*it; # the identifier already exists at this point, even without us knowing `aif` is a macro! it doesn't need to be specially introduced! it's just dynamically redefined!
}

(there are chances you could even do it with just my $*it;)

[masak]

Yes, that was much clearer. I haven't tried, but I expect I'll be able to go back and read that manual page without a problem now. Thank you for your efforts at getting the point across; it's much appreciated.

Yes, I expect you would be able to use dynamic variables with macros like that in Perl 6. Or at least, I got it to work in Rakudo just now:

$ perl6 -e 'use experimental :macros; my $*d = "nope"; macro moo($ast) { quasi { my $*d = 42; {{{$ast}}} } }; moo say $*d'
Weird node visited: QAST::BVal
42

(No, I have no idea what that "Weird node" warning is about.)

[masak]

Though needless to say, using dynamic variables kind of misses the point of anaphoric macros... 😄

[vendethiel]

Though needless to say, using dynamic variables kind of misses the point of anaphoric macros... 😄

oh, for sure. It's dynamic at compile-time, but very lexical at runtime :).

[masak]

No, it ends up dynamic at run-time too:

$ perl6 -e 'use experimental :macros; my $*d = "nope"; sub bar { say $*d }; macro moo($ast) { quasi { my $*d = 42; {{{$ast}}} } }; moo bar()'
Weird node visited: QAST::BVal
42

What I meant was that variables introduced anaphorically are meant to look like ordinary lexical variables, which dynamic variables don't do.

[vendethiel]

I was talking about my $?*, not dynamic variables as they exist.

[masak]

Even so, having to mark a variable with sigils just so that it'll work inside of the macro invocation defeats the whole point of anaphoric macros (and of this issue), does it not?

[vendethiel]

I don't understand what you mean here? This feature "dynamically scoped at compile-time only" is very specific, isn't it? It's a workaround to "introducing arbitrary identifiers": you have a fixed set of identifiers to work with.

[masak]

Yes, we seem to be speaking at cross-purposes a little for the past three messages or so. But I think it can be worth sorting out. I think what's missing is a clear statement of our goals.

My goal is for a symbol such as it to be usable as part of invoking a macro, even though no it has (necessarily) been declared in the calling scope.

Your goal was to explain define-syntax-parameter to me. It clicked for me when I saw the actual Racket code that used it. (It's from the middle of that documentation page, but my eyes must have already glazed over at that point.) You also translated the aif example to Perl 6, along the way introducing a new twigil ?* for "dynamic at compile time". Now, your goal may still be at this point to explain define-syntax-parameter clearly, which is fine. But if the goal is to transparently introduce a normal-looking variable without having an explicit declaration on the calling side, then $?*it fails on two counts: (a) it's not normal-looking, and (b) it does require an explicit declaration on the calling side.

Did I get that right?

[vendethiel]

But if the goal is to transparently introduce a normal-looking variable without having an explicit declaration on the calling side, then $?*it fails on two counts: (a) it's not normal-looking, and (b) it does require an explicit declaration on the calling side.

Oh, it doesn't look normal for sure. That can deemed fine. However your (b) is wrong: the very idea is that the variable is only declared in the module that declared aif. You just export that variable and any code using aif can use that $it (or $?*it).

[masak]

I was thinking about this one today. Here's a possible mechanism:

import parser;

# meanwhile, later, in an `is parsed`

/ { parser.declare("_") } <block> / # the `_` variable will be available in the `<block>`

I was also thinking a parser.newBlock() could declare a block until current block's end. Something like #13 could use that.

[masak]

Another raw/undigested comment about identifiers: in quite a few built-in and user-defined syntactic constructs, the presence of an identifier is a declaration. We might want to have a nice way to specify that too. In that case, there would be three modes: [...]

I just ran head-first into this distinction when trying to implement #410 in a branch. We need to distinguish now, at the grammar level. At least between things that do lookup (identifier terms) and things that don't.

[masak]

Another use case for this feature would be automatically declaring self inside of methods (and field initializers).