Unclear sharing of `$/` causing race conditions

[lizmat]

The following code produces consistent results:

grammar G { token TOP { \w+ } }

sub summarize() {
    ("aaa" .. "zzz").map( {
        my $result = G.parse($_);
        $result.Str.comb.Set.keys.ords.sum  # something to keep CPU busy
    } ).sum
}

say summarize;  # 6615297

However, if we put a .race or a .hyper in the mix (as in ("aaa" .. "zzz").race.map( {), the results become random with only occasionally producing the correct result.

This can be fixed by adding a my $/ inside the code block of the .map. Or by changing the block to a proper sub (("aaa" .. "zzz").map( sub ($_) {).

So clearly this problem is caused by the block in the map not having its own lexical $/, so that G.parse is setting the $/ that is implicitly defined outside the block, inside sub summarize.

In 6.e currently Grammar.parse (and friends) will not set $/, and then the above code will work as it should. However, that is a breaking change that we may not want to keep. On the other hand we may want to keep this change to reduce "action at a distance" more generally.

[2colours]

Massive +1 for eliminating this "action at a distance" in the future (whether it's as soon as 6.e or later on).

For setting (and only that - not reading!) $/, I realize now that I didn't express myself clearly on IRC. I will try to use "it" less often...

I don't feel strongly about setting $/ or not. People more like AlexDaniel would be very much in favor of eliminating $/, I think. I don't mind eliminating $/ but I don't mind keeping it either, as long as only the end user reads the value of it.

I could even imagine a pragma for these implicit "magic variables" that are set in operations, something like use result-vars, if we think it should be opt-in.

[lizmat]

In an unexpected turn of events:

grammar G { token TOP { (\w+) } }  # add () to set $0

sub summarize() {
    ("aaa" .. "zzz").race.map( {
        G.parse($_);
        $0.Str.comb.Set.keys.ords.sum  # something to keep CPU busy
    } ).sum
}

say summarize;  # 6615297

It appears that explicitely mentioning $/, or implicitely with $0 in the block scope, will codegen a lexical $/ in that scope.

So it looks more like this is a bug / deficiency in not noticing that $/ is going to be accessed, that is preventing $/ from getting its own lexical copy.

Sadly my testing was bogus, probably because I used a sub inside the map instead of a just a block.

[2colours]

@lizmat does this effectively mean that all matchings should create their own $/? In which case the name of that variable could really be anything?

[lizmat]

Further golf of the code:

say ("aaa" .. "zzz").race.map({
    / \w+ /;
    $/.Str.comb.Set.keys.ords.sum
} ).sum

So note that this problem is actually more general, and not limited to using grammars.

[2colours]

Is the problem because of the user-space access of $/ or would it also happen with $match = .match(/ \w+ /); $match.Str.... as well? I would say that makes a big difference because the user-space access is "clear sharing".

[jubilatious1]

Don't see any reason a user should set $/ ever. How to prohibit?

[lizmat]

A similar situation may actually make sense:

$_ = "foo";
if 42 {
    m/ \w+ /;
}
else {
    m/ \d+ /;
}
say $/;  # ｢foo｣

However, there are a few strange things happening here. If we remove the m from m/ \w+ /, the outer $/ is Nil in the legacy grammar, but remains ｢foo｣ in the new Raku grammar.

[patrickbkr]

Sharpening my understanding. Is the following a correct description of the situation?
Subs and methods always bring their own lexical $/, blocks do not. That $/ is set by regex calls and Grammar.parse and friends. rakudo/rakudo@8542ad2116 changed it so Grammar.parse and friends don't set $/ while Regex calls still do. Currently noone suggests to get rid of regexes setting $/.
Also there is an unexplained behavior bug in the legacy grammar where $/ seems to not be set in a corner case.

[patrickbkr]

A similar situation may actually make sense:

$_ = "foo";
if 42 {
    m/ \w+ /;
}
else {
    m/ \d+ /;
}
say $/;  # ｢foo｣

However, there are a few strange things happening here. If we remove the m from m/ \w+ /, the outer $/ is Nil in the legacy grammar, but remains ｢foo｣ in the new Raku grammar.

$_ = "foo";
{
    / \w+ /;
    say $/;  # ｢foo｣
}
say $/;  # ｢foo｣

$_ = "foo";
{
    / \w+ /;
}
say $/;  # Nil

I'd firmly classify this as a bug. I would have expected for $/ to be ｢foo｣ in all of the above cases.

[lizmat]

@patrickbkr Yes, that is a bug. Fortunately, this is already fixed in RakuAST, so I won't be spending time on trying to fix this in the legacy grammar.

[lizmat]

After discussing this with @jnthn, a lot has become clearer to me.

Basically, the functionality of $/ is based on nqp::getlexcaller which states: "Looks up the lexical with the specified name, starting at the calling frame. It checks all outer frames of the caller chain."

I didn't really grok this, and will probably adapt the text to make it clearer in what it does.

The pseudo code for matching is basically:

method Str::match(...) {
    nqp::getlexcaller('$/') = match-result;
}

What nqp::getlexcaller does, is walk the caller stack to see if a lexical $/ is known in that scope. If there isn't, it will walk the OUTER scope of that scope to find a lexically defined $/. Now in practice, this means it will always just find the $/ in one of the OUTER scopes of the caller scope, because there is bound to be a subroutine / method / mainline with a $/ in the OUTER scope of the caller's scope.

This, combined with the fact that blocks do not declare a lexical $/, creates the situation that allows the above examples to work.

Now, this is a very elegant design that works like a charm in a single-threaded world. But in a multi-threaded world, this is a nasty WAT. For example:

start { "foo" ~~ / \w+ / }
sleep 1;  # give thread startup a little time
say $/;  # Nil

Huh? Why doesn't that say ｢foo｣?

Promise.start: { "foo" ~~ / \w+ / }
sleep 1;  # give thread startup a little time
say $/;  # ｢foo｣

Isn't that supposed to be the same?

Yes, it is, but the grammar handling of start { } stealthily adds a my $/ to the block.

start { say MY::.keys }  # ($/ $_ $!)
sleep 1;  # give thread startup a little time

versus:

Promise.start: { say MY::.keys }  # ($_)
sleep 1;  # give thread startup a little time

If there is a keyword in the Raku grammar that allows for special compile time tweaking of the opcode, it can be done. However, in the case of Promise.start, it's just a method called on a class with a block as an argument. And applying heuristics on that, would be very fragile, and possibly even harder to explain and document.

Conclusion: I think we need to document this behaviour of $/ more thoroughly, and add it as a possible trap.

Going back to the golfed code, adding a my $/:

say ("aaa" .. "zzz").race.map({
    my $/ = / \w+ /;  # define lexical $/ to prevent race condition
    $/.Str.comb.Set.keys.ords.sum
} ).sum

is a simple workaround that I believe can be easily explained and documented.

[2colours]

I think this issue is getting fragmented.

The original issue that I responded to was about $/ acting as a hidden, internal parameter in the parsing process. Note that the code had no appearances of $/, yet it was affected by the scoping of that variable. That's an implementation topic.

The more recent comments were about how $/ works when somebody actually and deliberately uses it. That's more of an interface-level topic.

Let's decide which issue is the one discussed here. Another issue can be opened for the other phenomenon.

(Honestly, since the original code snippet doesn't have $/ as a part of any interface, I tend to think that the original topic should be treated as a mere bug, even if it has implications for existing code for some reason. Perhaps it shouldn't.)

[lizmat]

The title stated the subject. The G.parse was just a part of the problem, which is now fixed in 6.e by not letting Grammar.parse set $/ anymore.

[jubilatious1]

Do we need a @/ Array for Grammars?