Regex bug on custom <ws>

[tisonkun]

grammar { token ws { \s+ }; token zs { \s+ }; token TOP {['if' | 'if' <.zs> 'else'] $} }.parse("if else").say # OUTPUT: «｢if else｣␤»

grammar { token ws { \s+ }; token zs { \s+ }; token TOP {['if' | 'if' <.ws> 'else'] $} }.parse("if else").say # OUTPUT: «Nil␤»

zs was used to confirm ws was treated as non-declarative, even after you've redefined it in a declarative way.

[ugexe]

Same issue as Raku/nqp#367 when naming a match after an internal cursor method (that also affects the cursors state).

[tisonkun]

@ugexe Raku/nqp#367 is closed, but the issue still exists. How about it?

[ugexe]

I’ve closed all my perl6 related PRs and issues.

[tisonkun]

So it remains? I wonder if I can close this issue.

[zoffixznet]

I wonder if I can close this issue.

The bug still exists. Don't close.

[moritz]

@ugexe may I ask why?

[ugexe]

@moritz you can read more here.
However this is not nuclear option blackmail, contrary to what messages someone may be portraying this as (and which i hope the fact I’ve not pulled modules from the ecosystem or similar would suggest). Simply put I had been drained of my energy through negativity in my public dealings in the perm 6 community. Now I’m also being provoked to defend myself for closing my own PRs/issues I lack the wherewithal to support (not per this inquiry). Cutting those out sends a message while simultaneously distancing myself from the source draining my energy in the first place.

And here is the thing: I did not make any demands. I did say I would come back when I witnessed a change in attitude, but some have construed this as some type of ultimatum or demand. Unfortunately I think the fact that I did not give a concrete way to resolve this has brought me further animosity, yet i cannot give such an answer. And the answer may require time and self reflection, which is not a satisfactory to someone looking for that concrete right-now resolution path.

I did not like what I was experiencing. I cut ties and backed out until the problem resolved itself, if ever. And now I’m experiencing additional problems for bringing this up period in my own way instead of what someone else thinks they might have done. I can use all this energy towards something productive, and maybe that includes perl 6 again sooner than later... but right now it’s a net negative for me (and strangely only getting worse).

[moritz]

Thank for the explanation @ugexe, I totally missed that other thread.
I'm sorry that working on Perl 6 caused emotional stress to you. I've been in a similar position at times, and have drawn back for a while. My selfish part hopes that you come to contributing more to Perl 6 eventually.

[JJ]

Ping?

[tisonkun]

No idea what's going on. Feel free to open another issue if it still exists

[AlexDaniel]

It's still there.

[tisonkun]

As #1222 (comment), close this one. Please file another to continue the discussion if any.

[lizmat]

Reopening because this issue., although related, still exists while the problem of Raku/nqp#367 was fixed.

[lizmat]

Turns out this is not a shadowing issue at all, but an LTM issue:

% raku -e 'grammar { token ws { \s+ }; token TOP {[if <.ws> else | if] $} }.parse("if else").say'
｢if else｣

The custom <ws> token blocks Longest Token Matching.

Changing the order of the alternatives does make it match. According to jnthn, this is because if there are multiple alternatives with the same length declarative prefix, it will try them in order.

Removing https://github.com/Raku/nqp/blob/main/src/QRegex/NFA.nqp#L376-L383 would allow a custom <ws>to become part of LTM, but this could potentially break code in the wild.