[lex.phases] Replacement of universal-character-names in translation phase three

[Halalaluyafail3]

Full name of submitter (unless configured in github; will be published with the issue): Jay Ghiron

Reference (section label): [lex.phases]/1.3

Issue description:

Consider the following two macro definitions:

#define X π
#define X \u03C0

According to translation phase three, these macro definitions should be equivalent because the universal-character-name should be replaced by the character it represents (U+03C0). However, GCC and MSVC reject this and consider them to be different. Clang accepts them as the same, but does not actually treat them the same:

#include<iostream>
#define S(...)#__VA_OPT__(__VA_ARGS__)
int main(){
    #define X π
    std::cout<<S(S(X)\n);//"π"
    //if #undef is added here, all three compilers agree on these outputs
    #define X \u03C0
    std::cout<<S(S(X)\n);//"\u03C0"
}

Additionally, the current wording forbids the following:

#define Y \##u03C0
int Y;

Since there is no preprocessing token of the form \u03C0 this is invalid, even though the intent is to allow it.

Related: P2314 and P2621

[jensmaurer]

Why is it a CWG issue if compilers are buggy? Please post bug reports to the respective implementation.

Since there is no preprocessing token of the form \u03C0 this is invalid, even though the intent is to allow it.

Why do you believe the "intent" is to allow this?

[Halalaluyafail3]

Why is it a CWG issue if compilers are buggy? Please post bug reports to the respective implementation.

P2314 says "However, it turns out that all major implementations already implement what this paper specifies, i.e. no implementation provides an escaped UCN.", so I assumed the specification is wrong given that existing compilers do not implement it. Moreover, it creates quite a large discrepancy with C.

Since there is no preprocessing token of the form \u03C0 this is invalid, even though the intent is to allow it.

Why do you believe the "intent" is to allow this?

That is what P2621 says "We propose that spliced UCNs be supported because, in addition to 3/4 of surveyed compilers supporting it, it falls off naturally of the specification: splicing happens before any other form of tokenization and supporting it avoid special-casing this oddity.".

[jensmaurer]

The "splicing" that P2621 talks about is line splicing, which is in phase 2 and does work with the current specification.

However, ## (concatenation) in the current specification cannot produce UCNs, contrary to the rather broad stroke in P2621 in the note that it can. Given that the note is not supported by normative text, it's probably best to strike it, or invert its statement.

Is your larger example with __VA_ARGS__ inconsistent with the claim in P2314 that all major implementations already do the right thing with the example shown there?

[Halalaluyafail3]

The "splicing" that P2621 talks about is line splicing, which is in phase 2 and does work with the current specification.

Looks like I misunderstood that, but P2621 clearly assumes that what it did was sufficient to make concatenation forming universal character names allowed.

Is your larger example with __VA_ARGS__ inconsistent with the claim in P2314 that all major implementations already do the right thing with the example shown there?

#define S(...)#__VA_OPT__(__VA_ARGS__)
#define X \u03C0
S(S(X))

The current wording says that \u03C0 should be replaced by π here during tokenization, so S(X) should be "π" and S(S(X)) should be "\"π\"". However, existing implementations do not do this so S(X) becomes "\u03C0" and S(S(X)) becomes "\"\\u03C0\"" (in C it could be "\"\u03C0\"" as well). The differences between "π" and "\u03C0" are only observable when stringizing again which will reveal that their spelling is different. The right side of the table is correct when not considering stringizing the results. Though I do not think the left side of the table is correct, S(K\u00f6ppe) would become "K\u00f6ppe" not "K\\u00f6ppe" in C++20.

Here is another scenario where implementations do not implement the current wording:

#define H <\u03C0>//not parsing a header name, just three tokens
#include H
#include<π>//should include the same file twice, but implementations do not do that

[QuaternionsRock]

The "splicing" that P2621 talks about is line splicing, which is in phase 2 and does work with the current specification.

P2621 explicitly lists both line splicing and concatenation as motivations:

#define CONCAT(x, y) x ## y
int CONCAT(\, u0393) = 0; // UB: universal character name formed by macro expansion

Given this and the fact that clang, gcc, and even msvc all support UCNs formed through concatenation, I think that the proposal intended to allow both spliced UCNs and UCNs produced by ##.

However, ## (concatenation) in the current specification cannot produce UCNs, contrary to the rather broad stroke in P2621 in the note that it can. Given that the note is not supported by normative text, it's probably best to strike it, or invert its statement.

While it could be made clearer, I don't think that the note conflicts with the normative text. The preprocessor must immediately determine if the result of the ## operator forms a valid preprocessing-token, essentially repeating the process that initially occurs in Phase 3. One could argue that the "preprocessing token recognition" was intended to include universal-character-name replacement. I don't think that making this explicit in the definition of the ## operator would be a huge issue.

[jensmaurer]

Agreed. Phase 3 is the only place where we form preprocessing tokens (and check their validity), so assuming that ## feeds into phase 3 (where UCNs are formed and replaced, too) seems reasonable.

[Halalaluyafail3]

While it could be made clearer, I don't think that the note conflicts with the normative text. The preprocessor must immediately determine if the result of the ## operator forms a valid preprocessing-token, essentially repeating the process that initially occurs in Phase 3. One could argue that the "preprocessing token recognition" was intended to include universal-character-name replacement. I don't think that making this explicit in the definition of the ## operator would be a huge issue.

If it is considered to go through phase three again then can comments be formed in concatenation?

#define C(X,Y)X##Y
C(/,/)//OK?

Additionally, the wording that makes invalid concatenations invalid in [cpp.concat] "If the result is not a valid preprocessing token, the program is ill-formed." would not make sense as phase three never forms invalid preprocessing tokens, so presumably the following would also be OK:

int C(*,*)p;//same C as before, phase three would just split these into two * tokens

So I do not see how the current wording implies that it undergoes phase three again.

[jensmaurer]

Hm...

"phase three never forms invalid preprocessing tokens"

Phase 3 is [lex.pptoken], and e.g. the last sentence of p1 there talks about "ill-formed". However, it feels self-contradictory to talk about an "invalid preprocessing token": Either something is a preprocessing token, or it doesn't match the lexer grammar for preprocessing-token, and then it's simply not a preprocessing token (not even an invalid one).

[QuaternionsRock]

If it is considered to go through phase three again then can comments be formed in concatenation?

I can't think of any cases where the answer to this question would matter, can you? If concatenation can form comments, then the result (a partial comment) is not a preprocessing token, and the program is ill-formed. If concatenation cannot form comments, then the result (//) is not a valid preprocessing token, and the program is ill-formed.

Additionally, the wording that makes invalid concatenations invalid in [cpp.concat] "If the result is not a valid preprocessing token, the program is ill-formed." would not make sense as phase three never forms invalid preprocessing tokens

I don't think the meaning of that sentence actually changes if you remove the word "valid". The point is that the result must be exactly one preprocessing token. I think "valid" is just intended to clarify that the ## operator can't just stuff arbitrary sequences of characters into a single instance of the last preprocessing-token category and call it a day.

Phase 3 is [lex.pptoken], and e.g. the last sentence of p1 there talks about "ill-formed". However, it feels self-contradictory to talk about an "invalid preprocessing token": Either something is a preprocessing token, or it doesn't match the lexer grammar for preprocessing-token, and then it's simply not a preprocessing token (not even an invalid one).

It's not particularly obvious to me why that wording is there. Either the excluded characters could be added to the definition of the last preprocessing-token category in the grammar, or those cases could be handled along with the rest of the partial preprocessing tokens and partial comments are not so trivially accounted for.

[Halalaluyafail3]

I don't think the meaning of that sentence actually changes if you remove the word "valid". The point is that the result must be exactly one preprocessing token. I think "valid" is just intended to clarify that the ## operator can't just stuff arbitrary sequences of characters into a single instance of the last preprocessing-token category and call it a day.

The way I interpret it is "(not a valid) preprocessing token", rather than "not a (valid preprocessing token)".

[QuaternionsRock]

The way I interpret it is "(not a valid) preprocessing token", rather than "not a (valid preprocessing token)".

I suppose that is technically ambiguous, but I think most people would use "an invalid preprocessing token" to convey the former meaning. I also think we can apply a bit of common sense to arrive at the conclusion that concatenating two preprocessing tokens was never intended to be able to produce two preprocessing tokens. Is that even concatenation?

[Halalaluyafail3]

I suppose that is technically ambiguous, but I think most people would use "an invalid preprocessing token" to convey the former meaning. I also think we can apply a bit of common sense to arrive at the conclusion that concatenating two preprocessing tokens was never intended to be able to produce two preprocessing tokens. Is that even concatenation?

It would be unambiguous if it was interpreted as not undergoing phase three again, as "the preceding preprocessing token is concatenated with the following preprocessing token" does not seem to imply that. If the intent was currently to repeat phase three, I would expect the wording to more clearly forbid multiple preprocessing tokens attempting to be formed. When concatenating two * preprocessing tokens the result would not be a valid preprocessing token, as there is no token that has the representation **. So the current surrounding wording seems to align more with the idea that it does just a concatenation, rather than concatenating the spellings and doing phase three again.