New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for extended Unicode escape sequences in strings and templates #2169

Merged
merged 16 commits into from Mar 3, 2015

Conversation

Projects
None yet
4 participants
@DanielRosenwasser
Member

DanielRosenwasser commented Feb 27, 2015

Fixes #2047.

ES6 extended escapes take the form of \u{XXXXX...} where XXXXX... is any number of hex digits. It is a syntax error if the value of these digits is greater than 0x10FFFF. When interpreting the value of such a code point, it must be interpreted as two individual UTF16 codepoints through a well-defined encoding scheme.

Notable changes are:

  • We support extended ES6 escape sequences as is.
  • We support downlevel emit for ES6 escape sequences, using \uXXXX escapes for any non-ASCII characters in the original string.
  • Template strings always use \uXXXX escapes for any non-ASCII characters.

Things we still don't support:

  • Escapes in regular expressions.
@yuit

This comment has been minimized.

Should we warn the users that we don't support the feature

@yuit

This comment has been minimized.

Contributor

yuit commented on src/compiler/scanner.ts in ee71954 Feb 28, 2015

You could just leave out the maxCount

@@ -155,6 +155,8 @@ module ts {
Catch_clause_variable_name_must_be_an_identifier: { code: 1195, category: DiagnosticCategory.Error, key: "Catch clause variable name must be an identifier." },
Catch_clause_variable_cannot_have_a_type_annotation: { code: 1196, category: DiagnosticCategory.Error, key: "Catch clause variable cannot have a type annotation." },
Catch_clause_variable_cannot_have_an_initializer: { code: 1197, category: DiagnosticCategory.Error, key: "Catch clause variable cannot have an initializer." },
An_extended_Unicode_escape_value_must_be_between_0x0_and_0x10FFFF_inclusive: { code: 1198, category: DiagnosticCategory.Error, key: "An extended Unicode escape value must be between 0x0 and 0x10FFFF inclusive." },
expected: { code: 1199, category: DiagnosticCategory.Error, key: "'}' expected." },

This comment has been minimized.

@CyrusNajmabadi

CyrusNajmabadi Feb 28, 2015

Contributor

'Unterminated unicode escape'

"category": "Error",
"code": 1198
},
"'}' expected.": {

This comment has been minimized.

@CyrusNajmabadi

CyrusNajmabadi Feb 28, 2015

Contributor

We already have "'{0}' expected."" for this.

@yuit

This comment has been minimized.

Contributor

yuit commented on src/compiler/scanner.ts in ee71954 Feb 28, 2015

Can you comment that this function?


// fall through
case CharacterCodes.x:
var escapedValue = scanExactNumberOfHexDigits(ch === CharacterCodes.x ? 2 : 4);

This comment has been minimized.

@CyrusNajmabadi

CyrusNajmabadi Feb 28, 2015

Contributor

can you just extract this code, and then call it from the 'x' or 'u' cases?

@yuit

This comment has been minimized.

Contributor

yuit commented on src/compiler/scanner.ts in ee71954 Feb 28, 2015

I prefer the boolean one

return String.fromCharCode(codePoint);
}

var codeUnit1 = Math.floor((codePoint - 65536) / 1024) + 0xD800;

This comment has been minimized.

@CyrusNajmabadi

CyrusNajmabadi Feb 28, 2015

Contributor

Can you just refernece the relevant portion of theES6 spec?

Can you just do "|0" to floor instead?

This comment has been minimized.

@DanielRosenwasser

DanielRosenwasser Feb 28, 2015

Member

It's exactly the same as 10.1.1

@yuit

This comment has been minimized.

Contributor

yuit commented on 6ad1780 Feb 28, 2015

👍

@yuit

This comment has been minimized.

Contributor

yuit commented on src/compiler/diagnosticInformationMap.generated.ts in bbf9579 Feb 28, 2015

I like that you mention "inclusive"

@@ -1130,7 +1130,7 @@ module ts {
newEndN = Math.max(newEnd2, newEnd2 + (newEnd1 - oldEnd2));
}

return createTextChangeRange(createTextSpanFromBounds(oldStartN, oldEndN), /*newLength: */newEndN - oldStartN);
return createTextChangeRange(createTextSpanFromBounds(oldStartN, oldEndN), /*newLength: */ newEndN - oldStartN);

This comment has been minimized.

@CyrusNajmabadi

CyrusNajmabadi Feb 28, 2015

Contributor

remove space after :

@@ -1212,4 +1212,53 @@ module ts {
}
}
}

var backslashOrDoubleQuote = /[\"\\]/g;
var escapedCharsRegExp = /[\u0000-\u001f\t\v\f\b\r\n\u2028\u2029\u0085]/g;

This comment has been minimized.

@CyrusNajmabadi

CyrusNajmabadi Feb 28, 2015

Contributor

Document this.

@yuit

This comment has been minimized.

Contributor

yuit commented on bbf9579 Feb 28, 2015

👍

@yuit

This comment has been minimized.

Contributor

yuit commented on src/compiler/utilities.ts in a81bf8c Feb 28, 2015

small thing, I would prefer the function defined before everything else in the body function

This comment has been minimized.

Member

DanielRosenwasser replied Feb 28, 2015

I prefer it this way; it impedes the flow of logic, and is clearer to understand offhandedly.

@yuit

This comment has been minimized.

Contributor

yuit commented on src/compiler/utilities.ts in 9d89668 Feb 28, 2015

why we have to convert to upperCase?

This comment has been minimized.

Member

DanielRosenwasser replied Feb 28, 2015

It's easier to differentiate when you have uppercase escapes, since Cyrus and I made the offhand judgement that most text is in lowercase anyway.

@yuit

This comment has been minimized.

Contributor

yuit commented on src/compiler/emitter.ts in 4657c2d Feb 28, 2015

What is the TODO comment suppose to mean?

Harden against trees without parent pointers for emitting literals; f…
…ix lookahead in text for numeric literal indicators.
@CyrusNajmabadi

This comment has been minimized.

Contributor

CyrusNajmabadi commented on src/compiler/utilities.ts in b1837c8 Mar 2, 2015

newLength:

@CyrusNajmabadi

This comment has been minimized.

Contributor

CyrusNajmabadi commented on 6be13a9 Mar 2, 2015

👍

@CyrusNajmabadi

This comment has been minimized.

Contributor

CyrusNajmabadi commented on src/compiler/emitter.ts in 5ec68eb Mar 2, 2015

This is super confusing. Can you just break it into simple if/else blocks? And comment when each would be hit?

@CyrusNajmabadi

This comment has been minimized.

Contributor

CyrusNajmabadi commented on f9cc013 Mar 3, 2015

nice!

👍

DanielRosenwasser added a commit that referenced this pull request Mar 3, 2015

Merge pull request #2169 from Microsoft/withANameLikeUnicodeYoudThink…
…ThereWouldntBeSoManyWaysToDoIt

Add support for extended Unicode escape sequences in strings and templates

@DanielRosenwasser DanielRosenwasser merged commit 7212912 into master Mar 3, 2015

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@DanielRosenwasser DanielRosenwasser deleted the withANameLikeUnicodeYoudThinkThereWouldntBeSoManyWaysToDoIt branch Mar 3, 2015

@Microsoft Microsoft locked and limited conversation to collaborators Jun 18, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.