i18n: $localize extraction should handle ES5 concatenated strings

[bravier]

🐞 bug report

Affected Package

The issue is caused by package @angular/localize.

Description

I know using $localize in TypeScript is still unofficially supported and undocumented but it works fine for us in most cases except for multi-line strings.

When using the $localize tag function on a multi-line template literal, whitespaces are taken into account in the computation of the message ID.

This results in different IDs for the same string and multiple trans-units in the XLIFF file for the same translation.

In our use case, we need to split long translated strings to keep our code correctly formatted.

I am not sure this is an expected behavior or an issue.

If this is the expected behavior, is there a way to keep using multi-line template literals and ignoring whitespaces to keep the ID unique?

One solution would be to remove whitespaces by formating template literal to start at the beginning of lines. But it breaks code formatting (see title3 in app.component.ts).

🔬 Minimal Reproduction

Minimal reproduction available at: https://github.com/bravier/localize-multi-line

I've generated this example using ng new and added @angular/localize using the official documentation.

See app.component.ts where I translate the same string with only one space added in title2 to keep the lines correctly indented.

Build steps:

• npm install
• npm run build
• http-serve dist/localize-multi-line

🌍 Your Environment

Angular Version:

Angular CLI: 9.1.4
Node: 12.16.3
OS: linux x64

Angular: 9.1.4
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, localize, platform-browser
... platform-browser-dynamic, router
Ivy Workspace: Yes

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.901.4
@angular-devkit/build-angular     0.901.4
@angular-devkit/build-optimizer   0.901.4
@angular-devkit/build-webpack     0.901.4
@angular-devkit/core              9.1.4
@angular-devkit/schematics        9.1.4
@ngtools/webpack                  9.1.4
@schematics/angular               9.1.4
@schematics/update                0.901.4
rxjs                              6.5.5
typescript                        3.8.3
webpack                           4.42.0

[petebacondarwin]

So I am afraid this is totally the expected behaviour. JavaScript template literal strings always include all the characters between the two back-ticks.

As I see it you have a few options. The first is to accept that the string is going to be long on a single line and replace the actual newlines with \n character strings:

title = $localize`long long long long long long long long long long long long\n                    multi-line string`;

Obviously this means that you have really long lines for some messages but at least you have full control over the characters that appear.

The other option is to use custom ids for your messages, which would ensure they receive the same translated string:

  title = $localize`:@@long-message-id:long long long long long long long long long long long long
                    multi-line string`;

  title2 = $localize`:@@long-message-id:long long long long long long long long long long long long
                     multi-line string`;

In this case the actual string will be ignored for translation purposes.

[adrienverge]

Hello, I take the liberty of intervening because I share the same problem.

It's desirable to avoid many extra spaces, like 3 ×                                     in:

              title = $localize`Long long long paragraph
                                that does NOT contain new lines (\\n)
                                but actually is 280 chars long
                                and would hardly fit in one line`;

... for various reasons:

• the ones described by @bravier,
• it confuses human (non-dev) translators,
• indenting or deintending the surrounding code block cause problems too.

Is there a way to use $localize differently (e.g. with $localize([string], attr)?) to allow breaking strings on multiple lines?
For instance, in many languages it's usual to write:

              title = _('Long long long paragraph ' +
                        'that does NOT contain new lines (\\n) ' +
                        'but actually is 280 chars long ' +
                        'and would hardly fit in one line');

[petebacondarwin]

Sooooo....

The actual syntax of $localize as a function is:

function $localize(messageParts: TemplateStringsArray, ...expressions: readonly any[]): string

So if you really wanted you could do define a function that creates TemplateStringsArray objects:

function _(cooked: string[], raw: string[] = cooked): TemplateStringsArray {
  Object.defineProperty(cooked, 'raw', {value: raw});
  return cooked as any;
}

And then you could do something like:

$localize(_('blah blah' +
           'rest of line 1\n' +
           'line 2 etc'));

And the @angular/localize translation tooling will pick this up OK. (I claim).

[bravier]

Hello Pete,
​
Thanks for your proposal, it would be a perfect compromise 👍
​
However I tried it and it doesn't work as is, because at the end of compilation, when ng build tries to replace pre-built English strings with French ones, it seems to expect every element of the node to be StringLiterals:

ERROR: Localized bundle generation failed: Unexpected messageParts for `$localize` (expected an array of strings).

From function unwrapStringLiteralArray:

angular/packages/localize/src/tools/src/source_file_utils.ts

Lines 188 to 194 in ecffc35

	export function unwrapStringLiteralArray(array: t.Expression): string[] {
	if (!isStringLiteralArray(array)) {
	throw new BabelParseError(
	array, 'Unexpected messageParts for `$localize` (expected an array of strings).');
	}
	return array.elements.map((str: t.StringLiteral) => str.value);
	}

What am I doing wrong?
(You can see the code I used in https://github.com/bravier/localize-multi-line/blob/master/src/app/app.component.ts)

[petebacondarwin]

The cooked and raw parameters must be string[]. Sorry, I misled you by my example. It should be:

$localize(_(['blah blah' +
           'rest of line 1\n' +
           'line 2 etc']));

[petebacondarwin]

Or if you were sure that you would never need to provide interpolation in your strings then you could have a simplified version of _() which looked like:

function _(cooked: string, raw: string = cooked): TemplateStringsArray {
  const result = [cooked];
  Object.defineProperty(result, 'raw', {value: [raw]});
  return result as any;
}

[bravier]

Thanks for the quick answers Pete!
​
I've tried your last proposition as I won't provide interpolations in my strings.
​
Unfortunately, I still have the same error:

Localized bundle generation failed: Unexpected messageParts for `$localize` (expected an array of strings).

​
When looking at the array the function unwrapStringLiteralArray receive when using the simplest form of $localize'Just a string' (with backticks of course), I see something like (redacted):

{
    "type": "ArrayExpression",
    ...
    "elements": [
        {
            "type": "StringLiteral",
            ...
            "extra": {
                "rawValue": "Just a string",
                "raw": "\"Just a string\""
            },
            "value": "Just a string"
        }
    ]
}

​
But when using concatenated strings like I am trying to do, it receives (redacted):

{
    "type": "BinaryExpression",
    ...
    "left": {
        "type": "BinaryExpression",
        ...
        "left": {
            "type": "BinaryExpression",
            ...
            "left": {
                "type": "StringLiteral",
                ...
                "extra": {
                    "rawValue": "Long long long paragraph ",
                    "raw": "'Long long long paragraph '"
                },
                "value": "Long long long paragraph "
            },
            "operator": "+",
            "right": {
                "type": "StringLiteral",
                ...
                "extra": {
                    "rawValue": "that does NOT contain new lines (\\n) ",
                    "raw": "'that does NOT contain new lines (\\\\n) '"
                },
                "value": "that does NOT contain new lines (\\n) "
            }
        },
        ...
    },
    ...
}

​
The code I used is updated on : https://github.com/bravier/localize-multi-line/blob/master/src/app/app.component.ts
​
Is there anything else I can try?
​
(this use case can be useful for others I think)

[petebacondarwin]

Ahah... of course. The concatenated strings are passed to the translation tooling as BinaryExpressions... We could probably make that bit of the translation code more lenient...
I.E. improve unwrapStringLiteralArray() to be able to handle arrays of binary expressions where the binary expression is a concatenation of strings.
But at some point we are venturing into partial evaluation of the code, which I'd like to avoid.

[petebacondarwin]

Actually, does anyone know? I think Babel has some partial evaluation of expressions available built-in.

[bravier]

We could probably make that bit of the translation code more lenient...
I.E. improve unwrapStringLiteralArray() to be able to handle arrays of binary expressions where the binary expression is a concatenation of strings.

It makes sense and would be awesome! Is there anything I can do to help in this way?

Actually, does anyone know? I think Babel has some partial evaluation of expressions available built-in.

I am sorry but I am not proficient on Babel to answer this question.