fix(ivy): i18n - ensure that colons in i18n metadata are not rendered #33820

petebacondarwin · 2019-11-14T11:40:38Z

The : char is used as a metadata marker in $localize messages.
If this char appears in the metadata it must be escaped, as \:.
Previously, although the : char was being escaped, the TS AST
being generated was not correct and so it was being output double
escaped, which meant that it appeared in the rendered message.

As of TS 3.6.2 the "raw" string can be specified when creating tagged
template AST nodes, so it is possible to correct this.

ocombe

Looks good, except for debug-test.sh and yarn.lock that shouldn't be part of the PR! :)

petebacondarwin · 2019-11-14T12:55:20Z

Thanks @ocombe - I removed the yarn.lock but I moved the debug-test.sh to its own commit since it is an important fix and might as well go in with this PR.

AndrewKushnir

LGTM 👍

josephperrott

LGTM

AndrewKushnir · 2019-11-15T01:44:43Z

New set of g3 presubmits:

AndrewKushnir · 2019-11-15T02:24:38Z

@petebacondarwin g3 presubmit indicated the following problem:

Error: Debug Failure. False expression: Expected argument 'text' to be the normalized (i.e. 'cooked') version of argument 'rawText'.
    at createTemplateLiteralLikeNode (javascript/node_modules/typescript/stable/lib/typescript.js:64087:22)
    at Object.createNoSubstitutionTemplateLiteral (javascript/node_modules/typescript/stable/lib/typescript.js:64111:20)
    at visitLocalizedString (packages/compiler-cli/src/ngtsc/translator/src/translator.ts:552:19)
...

Even though TS version requirement is satisfied. I sent you a message in Slack with additional info.

petebacondarwin · 2019-11-15T16:23:05Z

From looking at the assertion - TS is re-parsing the raw text as though it was in the original source.

For example if we are creating a simple no-substitution template literal with the text abc they put this back through their normal tokenizer (scanner) wrapped in backticks abc and check that they then get the original abc out.

It is likely that the failing messages contain unescaped backticks or ${ sequences.

For example

ab`c

In this case they would put it through the scanner as

`ab`c`

which would obviously return only ab as the cooked string.

I have added a commit that escapes these sequences.

AndrewKushnir · 2019-11-15T17:45:20Z

Thanks for the fix @petebacondarwin.

I've restarted g3 presubmits with the most up-to-date version of this PR:

IgorMinar · 2019-11-15T23:45:31Z

Still fails with "Debug Failure. False expression: Expected argument 'text' to be the normalized (i.e. 'cooked') version of argument 'rawText'."

AndrewKushnir · 2019-11-16T01:51:44Z

@petebacondarwin I performed additional investigation and here is the pattern that is causing failures in g3:

<div i18n="Some text \' [BACKUP_MESSAGE_ID: xxx]">Test</div>

petebacondarwin · 2019-11-16T08:24:06Z

Thanks @AndrewKushnir - is the \' supposed to be rendered as a ' or as \' in the browser?
I can "fix" this by double escaping any \s that are found in the description, but I am wondering what the actual intent is.

petebacondarwin · 2019-11-16T13:13:59Z

I have amended the last commit to also escape \ in the "raw" form.
@AndrewKushnir - please can you try the presubmit one more time?

The `:` char is used as a metadata marker in `$localize` messages. If this char appears in the metadata it must be escaped, as `\:`. Previously, although the `:` char was being escaped, the TS AST being generated was not correct and so it was being output double escaped, which meant that it appeared in the rendered message. As of TS 3.6.2 the "raw" string can be specified when creating tagged template AST nodes, so it is possible to correct this.

Since i18n messages are mapped to `$localize` tagged template strings, the "raw" version must be properly escaped. Otherwise TS will throw an error such as: ``` Error: Debug Failure. False expression: Expected argument 'text' to be the normalized (i.e. 'cooked') version of argument 'rawText'. ``` This commit ensures that we properly escape these raw strings before creating TS AST nodes from them.

AndrewKushnir · 2019-11-16T22:45:11Z

Thanks for the fix @petebacondarwin. I've started new presubmits:

AndrewKushnir · 2019-11-18T00:49:35Z

Hi @petebacondarwin, VE and Ivy presubmits look good with the most recent changes, so I've set the g3 status to "green" and removed the "blocked" label. Thanks for the fixes 👍

gkalpak · 2019-11-18T15:42:38Z

packages/compiler-cli/test/compliance/mock_compile.ts

 const STRING = /'(\\'|[^'])*'|"(\\"|[^"])*"/;
-const BACKTICK_STRING = /\\`(([\s\S]*?)(\$\{[^}]*?\})?)*?\\`/;
+const BACKTICK_STRING = /\\`(([\s\S]*?)(\$\{[^}]*?\})?)*?[^\\]\\`/;


OOC, what does the [^\\] do?

:-)

This is so that we can pick up escaped single quotes (i.e. \') that exist outside of single quoted strings.

I see. I guess you don't need the capturing groups (because they will be messed up anyway 😁), right?

Nit: BTW, \$\{[^}]*?\} doesn't need to be non-greedy: \$\{[^}]*\}

PR Close #33820

…#33820) The `:` char is used as a metadata marker in `$localize` messages. If this char appears in the metadata it must be escaped, as `\:`. Previously, although the `:` char was being escaped, the TS AST being generated was not correct and so it was being output double escaped, which meant that it appeared in the rendered message. As of TS 3.6.2 the "raw" string can be specified when creating tagged template AST nodes, so it is possible to correct this. PR Close #33820

…#33820) Since i18n messages are mapped to `$localize` tagged template strings, the "raw" version must be properly escaped. Otherwise TS will throw an error such as: ``` Error: Debug Failure. False expression: Expected argument 'text' to be the normalized (i.e. 'cooked') version of argument 'rawText'. ``` This commit ensures that we properly escape these raw strings before creating TS AST nodes from them. PR Close #33820

…#33820) The `:` char is used as a metadata marker in `$localize` messages. If this char appears in the metadata it must be escaped, as `\:`. Previously, although the `:` char was being escaped, the TS AST being generated was not correct and so it was being output double escaped, which meant that it appeared in the rendered message. As of TS 3.6.2 the "raw" string can be specified when creating tagged template AST nodes, so it is possible to correct this. PR Close #33820

…#33820) Since i18n messages are mapped to `$localize` tagged template strings, the "raw" version must be properly escaped. Otherwise TS will throw an error such as: ``` Error: Debug Failure. False expression: Expected argument 'text' to be the normalized (i.e. 'cooked') version of argument 'rawText'. ``` This commit ensures that we properly escape these raw strings before creating TS AST nodes from them. PR Close #33820

angular-automatic-lock-bot · 2019-12-20T16:04:47Z

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

petebacondarwin added type: bug/fix action: review The PR is still awaiting reviews from at least one requested reviewer area: i18n target: patch This PR is targeted for the next patch release hotlist: release-blocker labels Nov 14, 2019

petebacondarwin requested review from a team as code owners November 14, 2019 11:40

ngbot bot modified the milestone: needsTriage Nov 14, 2019

googlebot added the cla: yes label Nov 14, 2019

petebacondarwin force-pushed the i18n-escape-meta-blocks branch from 35672c5 to 33f9a71 Compare November 14, 2019 12:32

ocombe suggested changes Nov 14, 2019

View reviewed changes

petebacondarwin force-pushed the i18n-escape-meta-blocks branch from 33f9a71 to a5df62c Compare November 14, 2019 12:54

petebacondarwin modified the milestones: needsTriage, v9-blockers Nov 14, 2019

ocombe approved these changes Nov 14, 2019

View reviewed changes

AndrewKushnir approved these changes Nov 14, 2019

View reviewed changes

AndrewKushnir added action: presubmit The PR is in need of a google3 presubmit and removed action: review The PR is still awaiting reviews from at least one requested reviewer labels Nov 14, 2019

alxhub approved these changes Nov 14, 2019

View reviewed changes

josephperrott approved these changes Nov 14, 2019

View reviewed changes

petebacondarwin added the action: merge The PR is ready for merge by the caretaker label Nov 14, 2019

AndrewKushnir added the state: blocked label Nov 15, 2019

petebacondarwin removed the hotlist: release-blocker label Nov 15, 2019

petebacondarwin mentioned this pull request Nov 15, 2019

ngcc: handle localized messages in ES5 #33857

Closed

petebacondarwin force-pushed the i18n-escape-meta-blocks branch from 119ea34 to 6ce72cc Compare November 15, 2019 16:20

petebacondarwin force-pushed the i18n-escape-meta-blocks branch from 6ce72cc to d4f2577 Compare November 15, 2019 16:25

petebacondarwin force-pushed the i18n-escape-meta-blocks branch from 8080c75 to 4c52d4d Compare November 16, 2019 13:13

petebacondarwin added 3 commits November 16, 2019 20:56

build: make ivy-i18n/debug-test.sh executable

9f4c3fa

petebacondarwin force-pushed the i18n-escape-meta-blocks branch from 4c52d4d to 5e3d9a6 Compare November 16, 2019 20:56

AndrewKushnir removed action: presubmit The PR is in need of a google3 presubmit state: blocked labels Nov 18, 2019

petebacondarwin added action: merge The PR is ready for merge by the caretaker and removed action: merge The PR is ready for merge by the caretaker labels Nov 18, 2019

gkalpak reviewed Nov 18, 2019

View reviewed changes

alxhub pushed a commit that referenced this pull request Nov 19, 2019

build: make ivy-i18n/debug-test.sh executable (#33820)

b7c1a5e

PR Close #33820

alxhub closed this in 74e6d37 Nov 19, 2019

petebacondarwin deleted the i18n-escape-meta-blocks branch November 19, 2019 10:35

angular-automatic-lock-bot bot locked and limited conversation to collaborators Dec 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ivy): i18n - ensure that colons in i18n metadata are not rendered #33820

fix(ivy): i18n - ensure that colons in i18n metadata are not rendered #33820

petebacondarwin commented Nov 14, 2019

ocombe left a comment

petebacondarwin commented Nov 14, 2019

AndrewKushnir left a comment

josephperrott left a comment

AndrewKushnir commented Nov 15, 2019

AndrewKushnir commented Nov 15, 2019

petebacondarwin commented Nov 15, 2019

AndrewKushnir commented Nov 15, 2019

IgorMinar commented Nov 15, 2019

AndrewKushnir commented Nov 16, 2019

petebacondarwin commented Nov 16, 2019

petebacondarwin commented Nov 16, 2019

AndrewKushnir commented Nov 16, 2019

AndrewKushnir commented Nov 18, 2019

gkalpak Nov 18, 2019

petebacondarwin Nov 18, 2019

gkalpak Nov 18, 2019

angular-automatic-lock-bot bot commented Dec 20, 2019

fix(ivy): i18n - ensure that colons in i18n metadata are not rendered #33820

fix(ivy): i18n - ensure that colons in i18n metadata are not rendered #33820

Conversation

petebacondarwin commented Nov 14, 2019

ocombe left a comment

Choose a reason for hiding this comment

petebacondarwin commented Nov 14, 2019

AndrewKushnir left a comment

Choose a reason for hiding this comment

josephperrott left a comment

Choose a reason for hiding this comment

AndrewKushnir commented Nov 15, 2019

AndrewKushnir commented Nov 15, 2019

petebacondarwin commented Nov 15, 2019

AndrewKushnir commented Nov 15, 2019

IgorMinar commented Nov 15, 2019

AndrewKushnir commented Nov 16, 2019

petebacondarwin commented Nov 16, 2019

petebacondarwin commented Nov 16, 2019

AndrewKushnir commented Nov 16, 2019

AndrewKushnir commented Nov 18, 2019

gkalpak Nov 18, 2019

Choose a reason for hiding this comment

petebacondarwin Nov 18, 2019

Choose a reason for hiding this comment

gkalpak Nov 18, 2019

Choose a reason for hiding this comment

angular-automatic-lock-bot bot commented Dec 20, 2019