Faster / parellel i18n AOT builds

[itslenny]

Bug Report or Feature Request (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ X ] feature request

Versions.

@angular/cli: 1.1.3
node: 6.10.1
os: darwin x64
@angular/animations: 4.2.4
@angular/common: 4.2.4
@angular/compiler: 4.2.4
@angular/compiler-cli: 4.2.4
@angular/core: 4.2.4
@angular/flex-layout: 2.0.0-beta.8
@angular/forms: 4.2.4
@angular/http: 4.2.4
@angular/material: 2.0.0-beta.7
@angular/platform-browser: 4.2.4
@angular/platform-browser-dynamic: 4.2.4
@angular/platform-server: 4.2.4
@angular/router: 4.2.4
@angular/cli: 1.1.3

Desired functionality.

I'm working on a fairly large app that uses AOT and i18n localization, and we need to translate into 60+ languages. Currently the only option is to run ng build for each language one at a time. For my app this takes 5 hours to complete running them in series. My short-term mitigation is to run ng build in parallel which will helps, but I think we can do better.

My thought is adding something like ng build -prod --i18n-config i18n-config.json

Then, in the config file you could list all the languages you want and the destination directories for each and ng-cli would run them in parallel. Additionally, it feels like it could do the AOT build once and hold the AST in memory then generate each language without having to re-do the entire AOT build for every language. I haven't fully researched this, and I know it'd add a lot of complexity, but for enterprise scale global apps this is absolutely needed to have reasonable build times.

[chcaru]

I agree, this will enable the real world use case of having a large team with a large app that has continuous delivery and many languages. What is available right now simply doesn't scale.

[filipesilva]

Yeah I can't really argue that 5h is a lot of time to build your app for all languages. I suppose at a low level, the apps array could be leveraged to that effect and used with a ng build --all or something. Wouldn't be much different from just using a shell script like you do currently though.

Saving part of the compilation from before sounds like a better idea, but much more complex. It would also require some explicit support from ngc itself.

[intellix]

In the hopes of improving what we have, this is how we're currently compiling our site for each language. Let me know if you have anything better for parallelism :)

{
  "scripts": {
    "build": "ng build --target=production -e ${NG_ENV:-prod} --aot",
    "build:i18n": "npm run build -- --output-path=dist/${LANG:-en} --bh /${LANG:-en}/ --i18n-file=src/locale/messages.${LANG:-en}.xlf --locale=${LANG:-en}",
    "heroku-postbuild": "for lang in en de sv; do LANG=$lang npm run build:i18n; if [ $? -ne 0 ]; then echo ERROR; exit 666; fi; done",
    "xi18n": "ng xi18n --i18n-format=xlf --out-file=locale/messages.xlf",
  }
}

Upon building in Heroku, it'll run the heroku-postbuild task which will start compiling languages one by one. If one of them fails then the whole build will fail. We originally tried to do things in parallel but an error would get swallowed up and report that the build was successful when it wasn't

[ocombe]

We have the same problem at google and it is now reaching the limit of our build systems as well. We are working on improvements for v5, stay tuned!

[intellix]

@ocombe just wanted to checkup on the timeline for the above stuff as v5 landed now :)
I'm guessing this is now a CLI thing since we have AOT watch mode and baking in the languages can be done much quicker?

[ocombe]

It will not be parallel builds. We are working on something called "runtime i18n" right now, we'll bundle all translations in the same bundle and apply the translation at runtime (without needing a compiler, so this is still AOT-safe). This means only one build even if you have 100 languages.
It should land in 5.1 or 5.2 (depending if we find any blocker that requires more work).

[SURENDARREDDY]

@ocombe , Anyway we are doing this for all the languages, Why can't we add for all environments.
I mean build only one for all languages and all environments once.
any chance?

[aldo-roman]

@ocombe I think "runtime i18n" approach would allow the user to change the app language without losing app state. Currently, AFAIK, to change a language you have to reload the complete app again. You also need different bundles, which means more downloads.

Any clue on how this is going? Sounds really great.

[chcaru]

@aldo-roman I think this is being tracked as part of this issue now:
angular/angular#16477

[trakhimenok]

@ocombe is there any place we can read about reasons to have all translations in a single bundle? Why not to have a separate trans bundle per language?

[ocombe]

we won't force you to have one bundle for all languages, you can build one bundle for each (it's what google will do internally)

[CanKattwinkel]

we won't force you to have one bundle for all languages, you can build one bundle for each (it's what google will do internally)

Thats sounds promising @ocombe. Will the i18n bundles then also be feature bundled in order to support lazy load along with the correct application chunk or are we required to load all translations with initial load?

[ocombe]

It's undecided yet, as much as we want to support lazy loading it'll probably be too much work for the first iteration of runtime i18n. But that's definitely a feature that we want to add.

[ntrp]

@ocombe Sorry for asking here but I cannot find up to date info about the "runtime i18n".. What is the status of the implementation? I have to decide between ngx-translate or angular-i18n (I would prefer the core lib) but we need editable translations. If I want AoT i am forced to use ngx-translate i guess at the moment.

[ocombe]

There's no fixed date, we are aiming for v6, but since it's part of the new renderer (ivy) it can be merged at any time (if we miss the v6 deadline, we don't have to wait for v7). It'll be behind a flag, and you'll have to use the new renderer which is not finished yet.
I don't think it's realistic to do a production app using ivy before may or june probably, there will be a lot of bug fixes once the feature parity with the current renderer is achieved

[ntrp]

I decided to use ngx-translate in the end because I need dynamic AoT translations since the translation files are hosted and can be changed at any time. If I understood correctly that is not possible with angular-i18n and AoT even after the "runtime i18n" because the translations are bundled, not loaded from the xlf.

[ocombe]

no, the translations will not necessarily be bundled, we'll add a way to lazy load them, but there might be some kind of transformation necessary for those files (because we don't want to ship the xliff/xtb compiler, it's like 10kb of code just to transform those xml files into usable js structures)

[alfaproject]

I'm guessing it will just be a simple JSON file for each language that can be lazy loaded with id -> target mapping. Can't be smaller than that, I think?

[ocombe]

yes something like that, the target node might be pre-processed as well to avoid any runtime computation

[jordan112]

Any updates on this? It was discussed coming in versions 5, then 6, then 7. 8 is around the corner. Our builds take approximately 45 minutes and we have about 20 languages. We will probably triple the number of languages and are looking at multiple hour builds. We are also discussing fanning out processing to many docker containers (one language per container) for every build, but wanted to check if there is any change in timeline so we can plan accordingly. Thanks!

[sebastianzillessen]

Hi! I am looking as well forward to a solution for this.
Especially as when you are executing the build for multiple locales, you will have a lot of equal files (e.g. runtime.**.js, styles.**.css, polyfill.**.js will stay the same, only the main.**.js file will change according to the locale).

Best,
Sebastian

[MebWorker]

Is there a safe & tested solution for parallel build or speeding up languages builds now?

[Wildhammer]

my solution for i18n parallel build:
https://stackoverflow.com/a/56463676/2582581

[JanneHarju]

I'm trying to use https://www.npmjs.com/package/npm-run-all this package with --parallel flag. It works on my machine but will rise my CPU usage to 100% and in our CI machine it will throw memory allocation failed exception, but I think that it is only ram issue and we are testing if more RAM will fix it. Some say that you need to run ng command with extra flag with larger project like this: "ng-high-memory": "node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng" but yet it didn't fix my problem.

[dgp1130]

Internationalization in Ivy is a lot smarter about builds so we don't need to do a full rebuild for each language by simply using the --localize parameter. You can specify only specific languages if you want, or build all of them and re-use most of the computation. This should drastically improve build performance.

https://angular.io/guide/i18n#build-from-the-command-line

If you are still having performance/memory problems when using the Ivy process, then please open a new issue where we can look into that specific problem more closely.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}