context-aware tokenization: Sometimes >> is better parsed as two copies of > (similar for ]] )

[dneto0]

Example for the >> case:

[[block]] struct B { a:  array<array<i32,2>>; } 
// That >> should two copies of GREATER_THAN, not one copy of SHIFT_RIGHT

Example for the ]] case:

fn foo() {
  var a : array<i32,10>;
  let v = a[a[0]]; // That ]] should be two copies of BRACKET_RIGHT instead of ATTR_RIGHT.
}

Tint parses these provisionally as the longer token, then backtracks to split them if needed.

I think the group has already discussed this but I can't find the tracking issue.

Related: #1063 validating the grammar is unambiguous

[kdashg]

Isn't one solution to not tokenize as e.g. ]] but instead ] ]?

[dneto0]

Isn't one solution to not tokenize as e.g. ]] but instead ] ]?

Yes. The unfortunate part of that is then under current rules you also have to accept those two tokens as separated by blankspace or a comment.

[dneto0]

C++14 solves this problem by saying tokenization is context-dependent.

2.2 Phases of translation, paragraph 1 step 3, says in part:

3. The source file is dcomposed into processing tokens (2.5) and sequences of white-space characters (including comments).

and then

The process of dividing a source file’s characters into preprocessing tokens is context-dependent. [ Example: see the handling of < within a #include preprocessing directive. — end example ]

Then step 7 says (emphasis added)

7. White-space characters separating tokens are no longer significant. Each preprocessing token is converted into a token. (2.7). The resulting tokens are syntactically and semantically analyzed and translated as a translation unit. [ Note: The process of analyzing and translating the tokens may occasionally result in one token being replaced by a sequence of other tokens (14.2). — end note ]

And 14.2 is "Names of template specializations" which handles the special >> cases. The text there is quite detailed.

[kdashg]

WGSL meeting minutes 2022-02-08

* MM: Can we…have a builtin instead?
* RM: I think it’s better to have C-style compat here, and that we should try to fix it, and only punt if we fail to do it reasonably well.

[kdashg]

Is this editorial?

[litherum]

Nit: Because we switched to using @foo, that takes care of the ]] ambiguity. So this issue is now just about >>.

Is this editorial?

Not necessarily.

I think we're picking between a few options:

1. Don't have a token in the language that represents >>. Instead, take all the places which would have accepted those tokens (e.g. let x = 4 >> 1;) and make them accept two individual > tokens. This has the side-effect that let x = 3 > > 4; (note the space character in between the two >s) would be valid.
2. Have a token in the language that represents >>, and naively tokenize it greedily. If the author wants to write array<array<i32,2>> they have to spell it array<array<i32,2> > (note the space character in between the two >s).
3. Have both tokens in the grammar, but add a syntax-aware rule about which one gets used where. This is what @dneto0 is describing above. This is technically a layering violation, but it lets the author write what is natural to them, and isn't particularly difficult to implement.
4. Replace the shift operator with a function. Make the author spell let x = 4 >> 1; as let x = rightShift(4, 1); instead.

[litherum]

Resolved: option 4. Replace << with a function, and replace >> with a (single) function.

Names for functions forthcoming.