Support `u8` type suffix for UTF8 string literals. #58991

AlekseyTs · 2022-01-21T05:06:25Z

Proposal - https://github.com/dotnet/csharplang/blob/main/proposals/utf8-string-literals.md
Relates to test plan #58848

Proposal - https://github.com/dotnet/csharplang/blob/main/proposals/utf8-string-literals.md

CyrusNajmabadi · 2022-01-21T05:22:22Z

Slightly worried about two lang features going in at the same time that affect strings literals :-)

In reply to: 1018191692

AlekseyTs · 2022-01-21T15:27:02Z

@cston, @RikkiGibson Please review.

In reply to: 1018610271

RikkiGibson

LGTM with minor comments and questions

RikkiGibson · 2022-01-21T19:18:17Z

src/Compilers/CSharp/Portable/Parser/Lexer_StringLiteral.cs

+
+                if (!inDirective && TextWindow.PeekChar() == 'u' && TextWindow.PeekChar(1) == '8')
+                {
+                    info.Kind = SyntaxKind.UTF8StringLiteralToken;


should the above assignment to info.Kind be in an else below this if? #Resolved

should the above assignment to info.Kind be in an else below this if?

That is an alternative way to get to the same result. If you prefer, I can refactor the code that way.

RikkiGibson · 2022-01-21T19:18:47Z

src/Compilers/CSharp/Portable/Parser/Lexer_StringLiteral.cs

+
+            if (TextWindow.PeekChar() == 'u' && TextWindow.PeekChar(1) == '8')
+            {
+                info.Kind = SyntaxKind.UTF8StringLiteralToken;


Should the above assignment to info.Kind be in an else below this if? #Resolved

That is an alternative way to get to the same result. If you prefer, I can refactor the code that way.

RikkiGibson · 2022-01-21T19:54:24Z

src/Compilers/CSharp/Test/Semantic/Semantics/Utf8StringsLiteralsTests.cs

+    static void Main()
+    {
+        var y = (C2)""dog""u8;
+        var z = (C3)""cat""u8;


I didn't understand why an error is produced here. I thought "cat"u8 has natural type ReadOnlySpan<byte> and then we can just use the user-defined conversion. Maybe we can sync offline? #Resolved

I didn't understand why an error is produced here.

You are correct. This doesn't follow the proposed specification. I guess, I forgot to add a PROTOTYPE comment about that in the Binder. I decided to go with byte[] as the natural type for now. That allows us to get away without special conversion rules for UTF8 string literals (do not confuse with string constant expressions, UTF8 string literals are not constants) because there are already implicit user-defined conversions from byte[] to Span<byte> and ReadOnlySpan<byte>, and there are no existing conversions from ReadOnlySpan<byte> to byte[] or Span<byte>. It feels like the language rules are simpler. Also, even when we are creating a ReadOnlySpan<byte>, we are creating a byte[] first. It feels like byte[] is a better choice for a natural type.

RikkiGibson · 2022-01-21T20:02:35Z

src/Compilers/CSharp/Test/Syntax/Parsing/UTF8StringLiteralsParsingTests.cs

+        [Fact]
+        public void Errors_01()
+        {
+            // The behavior is consistent with how type syffixes are handled on numeric literals, see Errors_07.


nit: "suffixes" #Resolved

RikkiGibson · 2022-01-21T20:02:56Z

src/Compilers/CSharp/Test/Syntax/Parsing/UTF8StringLiteralsParsingTests.cs

+        [Fact]
+        public void Errors_05()
+        {
+            // The behavior is consistent with how type syffixes are handled on numeric literals, see Errors_06.


nit: "suffixes" #Resolved

RikkiGibson · 2022-01-21T20:03:12Z

src/Compilers/CSharp/Test/Syntax/Parsing/UTF8StringLiteralsParsingTests.cs

+        [Fact]
+        public void Errors_08()
+        {
+            // The behavior is consistent with how type syffixes are handled on numeric literals, see Errors_07.


nit: "suffixes" #Resolved

RikkiGibson · 2022-01-21T20:03:23Z

src/Compilers/CSharp/Test/Syntax/Parsing/UTF8StringLiteralsParsingTests.cs

+        [Fact]
+        public void Errors_12()
+        {
+            // The behavior is consistent with how type syffixes are handled on numeric literals, see Errors_06.


nit: "suffixes" #Resolved

RikkiGibson · 2022-01-21T20:07:35Z

src/Compilers/CSharp/Test/Syntax/Parsing/UTF8StringLiteralsParsingTests.cs

+        [Fact]
+        public void Interpolation_01()
+        {
+            UsingExpression(@"$""hello""u8",


It might be good to modify the proposal to specify what happens when @ or $ prefix is used, and either:

the u8 suffix is used, or

the new utf8 conversion is used

Intuitively I thought that a constant-valued interpolated string would be fine to use with the u8 suffix, but that doesn't appear to be the case. I don't have a strong position on that, but it seems like whatever is decided should be included in the proposal.
#Resolved

I am not sure where confusion is coming from. I think the proposal is clear: "the language will provide the u8 suffix on string literals to force the type to be UTF8." The suffix is for literals, not for constant expressions. Interpolated strings are not literals, even when they have a constant value.

cston · 2022-01-22T00:23:10Z

src/Compilers/CSharp/Portable/Syntax/SyntaxEquivalence.cs

@@ -82,6 +82,7 @@ private static bool AreTokensEquivalent(GreenNode? before, GreenNode? after, Fun
                case SyntaxKind.NumericLiteralToken:
                case SyntaxKind.CharacterLiteralToken:
                case SyntaxKind.StringLiteralToken:
+                case SyntaxKind.UTF8StringLiteralToken:


SyntaxKind.UTF8StringLiteralToken

Are we testing this case? #Closed

Are we testing this case?

Not explicitly. I'll make note in test plan

cston · 2022-01-22T00:31:51Z

src/Compilers/CSharp/Test/Semantic/Semantics/Utf8StringsLiteralsTests.cs

+";
+            var comp = CreateCompilation(source, targetFramework: TargetFramework.NetCoreApp, options: TestOptions.DebugExe);
+            CompileAndVerify(comp, expectedOutput: @"byte[]"). // The behavior has changed
+                VerifyDiagnostics();


Please consider adding:

comp = CreateCompilation(source, targetFramework: TargetFramework.NetCoreApp, parseOptions: TestOptions.Regular10, options: TestOptions.DebugExe); CompileAndVerify(comp, expectedOutput: "string"); ``` #Closed

Please add an entry in "docs/compilers/CSharp/Compiler Breaking Changes - DotNet 7.md".

The scenario will become an error. We will not change semantics based on language version.

Please add an entry in "docs/compilers/CSharp/Compiler Breaking Changes - DotNet 7.md".

I'll add a PROTOTYPE comment about that for now. We might still change the design around this

cston · 2022-01-22T00:41:44Z

src/Compilers/CSharp/Test/Semantic/Semantics/Utf8StringsLiteralsTests.cs

+            CompileAndVerify(comp, expectedOutput: @"array").VerifyDiagnostics();
+        }
+
+        [ConditionalFact(typeof(CoreClrOnly))]


[ConditionalFact(typeof(CoreClrOnly))]

[Fact] #WontFix

HelpersSource refers to types that are not available on desktop. I am fairly comfortable to test this scenario only on CoreClr.

cston · 2022-01-22T00:42:00Z

src/Compilers/CSharp/Test/Semantic/Semantics/Utf8StringsLiteralsTests.cs

+").VerifyDiagnostics();
+        }
+
+        [ConditionalFact(typeof(CoreClrOnly))]


[ConditionalFact(typeof(CoreClrOnly))]

[Fact] #WontFix

HelpersSource refers to types that are not available on desktop. I am fairly comfortable to test this scenario only on CoreClr.

cston · 2022-01-22T00:43:32Z

src/Compilers/CSharp/Test/Semantic/Semantics/Utf8StringsLiteralsTests.cs

+                );
+        }
+
+        [ConditionalFact(typeof(CoreClrOnly))]


[ConditionalFact(typeof(CoreClrOnly))]

[Fact] #WontFix

HelpersSource refers to types that are not available on desktop. I am fairly comfortable to test this scenario only on CoreClr.

CyrusNajmabadi · 2022-01-22T00:59:27Z

src/Compilers/CSharp/Portable/Binder/Binder_Expressions.cs

@@ -673,6 +673,9 @@ private BoundExpression BindExpressionInternal(ExpressionSyntax node, BindingDia
                case SyntaxKind.NullLiteralExpression:
                    return BindLiteralConstant((LiteralExpressionSyntax)node, diagnostics);

+                case SyntaxKind.UTF8StringLiteralExpression:


fyi, in the raw-string-work we decided to not have a special expression type. it's just a string-literal that consumer can check the token kind on. up to you if you want to do the same. #Resolved

fyi, in the raw-string-work we decided to not have a special expression type. it's just a string-literal that consumer can check the token kind on. up to you if you want to do the same.

I will add a PROTOTYPE comment for now and will revisit this once raw-string-work gets into this branch.

CyrusNajmabadi · 2022-01-22T01:00:35Z

src/Compilers/CSharp/Portable/Parser/Lexer_StringLiteral.cs

@@ -77,7 +77,18 @@ private void ScanStringLiteral(ref TokenInfo info, bool inDirective)
            }
            else
            {
-                info.Kind = SyntaxKind.StringLiteralToken;
+                if (!inDirective && TextWindow.PeekChar() == 'u' && TextWindow.PeekChar(1) == '8')


nit: this may follow the spec, but we probably want to take back to ldm if we support U8 as well as u8 all our other suffixes are case insensitive. so we probably want the same here. #Resolved

this may follow the spec, but we probably want to take back to ldm if we support U8 as well as u8 all our other suffixes are case insensitive. so we probably want the same here.

Will add a PROTOTYPE comment and will bring this design question to LDM

CyrusNajmabadi · 2022-01-22T01:01:03Z

src/Compilers/CSharp/Portable/Parser/Lexer_StringLiteral.cs

@@ -77,7 +77,18 @@ private void ScanStringLiteral(ref TokenInfo info, bool inDirective)
            }
            else
            {
-                info.Kind = SyntaxKind.StringLiteralToken;
+                if (!inDirective && TextWindow.PeekChar() == 'u' && TextWindow.PeekChar(1) == '8')


note: for raw strings, i allowed them to be parsed in directives, but then give a particular message that it isn't allowed. up to you if you want to do the same. #Resolved

note: for raw strings, i allowed them to be parsed in directives, but then give a particular message that it isn't allowed. up to you if you want to do the same.

I do not believe it is worth the effort

Support u8 type suffix for UTF8 string literals.

27a0ea9

Proposal - https://github.com/dotnet/csharplang/blob/main/proposals/utf8-string-literals.md

AlekseyTs requested a review from a team as a code owner January 21, 2022 05:06

dotnet-issue-labeler bot added the Area-Compilers label Jan 21, 2022

Addd a test

a1336e0

RikkiGibson self-assigned this Jan 21, 2022

Add tests

33f43d8

RikkiGibson approved these changes Jan 21, 2022

View reviewed changes

PR feedback

ea026f4

cston reviewed Jan 22, 2022

View reviewed changes

CyrusNajmabadi reviewed Jan 22, 2022

View reviewed changes

jcouv added the New Feature - Utf8StringLiterals label Jan 24, 2022

PR feedback

26fa7eb

cston approved these changes Jan 25, 2022

View reviewed changes

AlekseyTs enabled auto-merge (squash) January 25, 2022 14:38

AlekseyTs merged commit d681e4e into dotnet:features/Utf8StringLiterals Jan 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support `u8` type suffix for UTF8 string literals. #58991

Support `u8` type suffix for UTF8 string literals. #58991

AlekseyTs commented Jan 21, 2022 •

edited by jcouv

CyrusNajmabadi commented Jan 21, 2022 •

edited by AlekseyTs

AlekseyTs commented Jan 21, 2022 •

edited

RikkiGibson left a comment

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

AlekseyTs Jan 21, 2022

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

AlekseyTs Jan 21, 2022

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

AlekseyTs Jan 21, 2022

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

AlekseyTs Jan 21, 2022

cston Jan 22, 2022 •

edited

AlekseyTs Jan 22, 2022

cston Jan 22, 2022 •

edited

cston Jan 22, 2022

AlekseyTs Jan 22, 2022

AlekseyTs Jan 22, 2022

cston Jan 22, 2022 •

edited by AlekseyTs

AlekseyTs Jan 22, 2022

cston Jan 22, 2022 •

edited by AlekseyTs

AlekseyTs Jan 22, 2022

cston Jan 22, 2022 •

edited by AlekseyTs

AlekseyTs Jan 22, 2022

CyrusNajmabadi Jan 22, 2022 •

edited by AlekseyTs

AlekseyTs Jan 24, 2022

CyrusNajmabadi Jan 22, 2022 •

edited by AlekseyTs

AlekseyTs Jan 24, 2022

CyrusNajmabadi Jan 22, 2022 •

edited by AlekseyTs

AlekseyTs Jan 24, 2022

Support u8 type suffix for UTF8 string literals. #58991

Support u8 type suffix for UTF8 string literals. #58991

Conversation

AlekseyTs commented Jan 21, 2022 • edited by jcouv

CyrusNajmabadi commented Jan 21, 2022 • edited by AlekseyTs

AlekseyTs commented Jan 21, 2022 • edited

RikkiGibson left a comment

Choose a reason for hiding this comment

RikkiGibson Jan 21, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RikkiGibson Jan 21, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RikkiGibson Jan 21, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RikkiGibson Jan 21, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

RikkiGibson Jan 21, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

RikkiGibson Jan 21, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

RikkiGibson Jan 21, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

RikkiGibson Jan 21, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cston Jan 22, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cston Jan 22, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cston Jan 22, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cston Jan 22, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cston Jan 22, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CyrusNajmabadi Jan 22, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CyrusNajmabadi Jan 22, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CyrusNajmabadi Jan 22, 2022 • edited by AlekseyTs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Support `u8` type suffix for UTF8 string literals. #58991

Support `u8` type suffix for UTF8 string literals. #58991

AlekseyTs commented Jan 21, 2022 •

edited by jcouv

CyrusNajmabadi commented Jan 21, 2022 •

edited by AlekseyTs

AlekseyTs commented Jan 21, 2022 •

edited

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

RikkiGibson Jan 21, 2022 •

edited by AlekseyTs

cston Jan 22, 2022 •

edited

cston Jan 22, 2022 •

edited

cston Jan 22, 2022 •

edited by AlekseyTs

cston Jan 22, 2022 •

edited by AlekseyTs

cston Jan 22, 2022 •

edited by AlekseyTs

CyrusNajmabadi Jan 22, 2022 •

edited by AlekseyTs

CyrusNajmabadi Jan 22, 2022 •

edited by AlekseyTs

CyrusNajmabadi Jan 22, 2022 •

edited by AlekseyTs