Generating and managing the translations files for large scale applications like ERP etc is very cumbersome.

[nhwaani]

🚀 feature request.

Generating and managing the translations files for large scale applications like ERP etc is very cumbersome.

Relevant Package

This feature request is for @angular/....

Description

Using the angular's builtin translation which the angular platform provides, managing and generating the translation files for multiple locale is very cumbersome , error prone and very difficult to manage as the application size scales, as generating the translation files after adding new i18n tags , generates the a default file messages.xlf and it becomes very difficult to dig into this default file (because of size .. lines of code.) and parse out the new trans-units and then move them to each locale files manually , and paste the target tag for the new trans-units .

Describe the solution you'd like

Either the translation file should be provided for each module or component level, or there must be some version control system used to check the differences/changes in the default messages.xlf file and these changes automatically being added to other locale files . Means atleast the new trans-units must be appended automatically to the locale files that the intended developer needs to create.

Describe alternatives you've considered

We have gone through ngx-Translate etc tools that provides run-time translation using pipes, but if the angular's builtin internalization module provides additional updates in near future to ease the internationalization process for developers , its far better than other third party packages.

[nhwaani]

I had gone in past through a library which uses xliffmerge to generate the translation files from the base translation file , https://github.com/nadeem09wani/ngx-i18nsupport here is the ref to that.

[ocombe]

Agreed, we need to improve this

[nhwaani]

I suggest the repo above which i shared can be good PR for this issue, if you go through the work done by the maintainer of that repo. He has used a pretty good approach to minimize the work of translation for large scale apps.
@ocombe @kara .

[ocombe]

I've used https://github.com/martinroob/ngx-i18nsupport (instead of the fork you linked which is behind the original repo) in the past, and it works really well. I would definitely reuse @martinroob's work (if he agrees) when we work on this feature.

[nhwaani]

That would be cool @ocombe .

[martinroob]

@ocombe It would be a great honor for me if you reuse my work.
Currently I'm trying to make xliffmerge a bit more Angular stylish.
I have added some schematics and right now I am working on an architect builder replacing the command line tool. It is a bit cumbersome because the documentation of all the stuff in angular-devkit is not very well or just not existing.

[ocombe]

I've heard that devkit/architect docs are coming on angular.io but it'll probably be in a few weeks

[angular-robot]

Just a heads up that we kicked off a community voting process for your feature request. There are 20 days until the voting process ends.

Find more details about Angular's feature request process in our documentation.

[angular-robot]

Thank you for submitting your feature request! Looks like during the polling process it didn't collect a sufficient number of votes to move to the next stage.

We want to keep Angular rich and ergonomic and at the same time be mindful about its scope and learning journey. If you think your request could live outside Angular's scope, we'd encourage you to collaborate with the community on publishing it as an open source package.

You can find more details about the feature request process in our documentation.

[DaSchTour]

I've used https://github.com/martinroob/ngx-i18nsupport (instead of the fork you linked which is behind the original repo) in the past, and it works really well. I would definitely reuse @martinroob's work (if he agrees) when we work on this feature.

The mentioned repo isn't maintained anymore. It would be really helpful if angular team could provide some guidance how to solve the issue of merging xliff files. Or maybe finally integrate the xliff merge tooling into angular.

[abstractBoyd]

Unfortunately @martinroob seems to have disappeared from the web a few months after his post above - luckily he gave explicit permission to use his code in the angular implementation.

We keep having to pull in old dependencies to keep our translations up to date after every angular upgrade, and this is getting to be unmaintainable. So far ngx-i18nsupport is the only free library that merges the xliff files the way we need them merged in a reliable way.

@ocombe - is there any chance ngx-i18nsupport could be picked back up and adopted by/integrated into Angular?

[ocombe]

I no longer work for the Angular team, but I can say with 97% certitude that it will not happen

[abstractBoyd]

@ocombe - that's a bummer. Any history you can share? Or is there anyone we could appeal to, to increase that 3% to something better?

[ocombe]

As far as I know @AndrewKushnir is now in charge of everything i18n

[timothyBrake]

My new feature request #23139 was just closed as duplicate so I'm assuming this is the place to get the latest update on this feature. In my post I suggested a way forward. Anyone from the Angular team want to comment please @AndrewKushnir ? Thanks

[jessicajaniuk]

During our triage meeting, we discussed that this really is a CLI issue and should really be addressed here.

[dgp1130]

Related: angular/angular#46471

[daniel-sc]

I created a drop-in replacement for the default angular i18n builder, which should do exactly what is requested here: https://github.com/daniel-sc/ng-extract-i18n-merge

[DaSchTour]

I created a drop-in replacement for the default angular i18n builder, which should do exactly what is requested here: https://github.com/daniel-sc/ng-extract-i18n-merge

So now maybe creating a PR to get this into the angular repo could be a next step? As most of the work seams to be already done somebody from the angular team could help with the last step?

[daniel-sc]

I'm not sure if there is a real benefit to having this included directly in angular (cli). Of course you'd save the initial ng add command, but maintenance would be more difficult - unless the angular team is really dedicated too support this (which I doubt).

If future maintenance is assured, of course I'd be happy if it is included and maintained by the angular team!

[dgp1130]

We had a discussion about this within the tooling team. The current process looks like this:

1. Extract: Run ng extract-i18n and get a file with messages in the source locale.
2. Translate: Send the file to translators who add in their translations in the "target" language and send back to the developer.
3. Dump: Copy the translated messages into your repository. ("Dump" is the term we use at Google. Some might call this "merge" but Angular seems to be inconsistent about that term's meaning, so I'll call it "dump" for now to reduce confusion.)
4. Build: Subsequent localized ng build operations will use translations from the latest dump to generate JS files with translations.

See https://angular.io/guide/i18n-common-translation-files#translation-process-example-for-french for more context around XLIFF files in particular.

The original comment is a bit vague on specifics, but we identified two key problems with the current system:

1. Extracting translations does not indicate which messages are new or changed and which already have usable translations.
2. Dumping new translations has no easy way to merge them into a project with existing translations without overwriting them.

These are related but distinct problems, and we agree that the CLI can probably do more here. It shouldn't be necessary to rely on an external tool to effectively manage subsequent translation files (an external tool may be necessary to actually translate messages, but that's a distinct problem from managing translation data).

A possible solution to this (option A) would be to generate a separate XLIFF file per-locale. Currently we only generate one XLIFF in the source locale, so we can't include any existing translations. However we could generate an XLIFF per-locale and include any existing translations. If a message is new or modified, the translation would be dropped, so any messages missing a <target></target> value need to be translated, while any which already exist can be left alone. Once translators respond with completed XLIFF files which have all <target></target> values filled in, that file can be copied directly into the Angular project (dumped) as is, without any need to merge with existing translations, since those were left in the extracted XLIFF files.

I'm using XLIFF as an example, but I think the same process can work in any of the other formats (maybe not XMB since that's a different file format from XTB?)

The consequences of this are:

1. ng extract-i18n now creates N extraction files rather than just one (where N is number of locales).
2. Any translation tools need to ingest N extraction files rather than just the one that they ingest today (where N is number of locales).
   • If this were a problem for any particular tools, I imagine we would include some kind of --locales flag to limit to particular locales you want to extract. Just add --locales ${SOURCE_LOCALE} and you would get exactly one file without any translations and be equivalent to current behavior.
3. Dumping is still the same process as today, just copy the translations into your project with no additional "merging" necessary.
4. Might not play well with XMBs, since those don't typically hold locale-specific information.

An alternative we considered (option B) would be to leave ng extract-i18n mostly as it is today, generating a single XLIFF file but filtering to only include messages which are new or modified. This way translators can simply translate the entire file. The challenge then is that the translation dumped into the project need to be merged with any existing translations, so we would need some kind of ng dump-i18n to perform this merge (or rely on community tools which seems mostly equivalent to how it works today).

The consequences of this are:

1. ng extract-i18n is mostly unchanged, still just emits a single extraction file.
   • Probably includes an option to --include-already-translated-messages to skip the filter if desired.
2. Users need to call a new CLI command ng dump-i18n at dump time, which has historically not been a thing.
   • The upside of this is that the CLI would now have a hook into the dump process, which could enable future improvements in this area without breaking users' workflows. I'm not really sure what we would be able to do with it in the future, but any information lost in extraction could be restored in dump (such as existing translations) which could be useful.
3. Merging process is probably more complicated to maintain in the CLI as compared to option A.

As of now we're leaning towards option A as the easier fix since it doesn't require complex format-specific merge logic. What would be helpful to us is to get some feedback about how well these solutions work. There aren't enough lists in this comment, so I'll add another one to enumerate some questions the community can help us answer:

1. Are the two problems documented at the start of this comment representative of the pain being felt here?
2. Do options A or B resolve this pain?
3. How well would options A or B work with existing translation tools? We don't have experience with all the translation tools out there or the technical expectations in this area.
4. Are all the translation file formats compatible with both solutions? I suspect XMBs will be awkward with option A, but I don't think it's a hard blocker on that one.

If you have any feedback related to these questions, we would really appreciate hearing it.

[daniel-sc]

@dgp1130 happy to see that the angular team wants to pick this up!

I'd vote for going along "option A" as this is analogous to what existing/past tooling did successfully (compare https://github.com/daniel-sc/ng-extract-i18n-merge and https://github.com/martinroob/ngx-i18nsupport). Option B would not allow translators to have the full picture and makes it difficult to adapt existing translations with unchanged source text.

Please be aware that there are a couple of subtleties involved:

• (stable) sort order
• (whitespace) normalization
• state management for updated translations (xlf translation state attribute)
• handle changed IDs for same/similar texts (some people do not assign explicit IDs / plurals have auto IDs)

For reference you could check out the config options of existing tools:

• https://github.com/daniel-sc/ng-extract-i18n-merge#configuration
• https://github.com/martinroob/ngx-i18nsupport/tree/master/projects/xliffmerge#usage

[timothyBrake]

Great to see the Angular team is standardising the final step in i18n workflow!

Some context
we do not use translation tools or an external translation company. We generate the XLF2 files using xliffmerge and manually change the files in our IDE. See attached ZIP (see further) for most basic fully working Angular 13 project as a reference to anyone reading this and want to have a starting point to how to do this now until the Angular team has created this feature.

Also this ZIP and the xliffmerge tool can act as a reference for how I personally expect the feature to work using ng extract-i18n. So I’m assuming this is what optionA is suggesting.

To answer the questions directly:

1. Are the two problems documented at the start of this comment representative of the pain being felt here?
   Yes these 2 are the main pain points from a developer standpoint that does the translations themselves. Imagine as a developer to always manually needing to scan the generated messages.xlf file and then compare the messages to all existing messages.{locale}.xlf files and search the XML for any new, updated and deleted values. To top it off, this needs to be done periodically while the project evolves. It’s just something no human should do and luckily xliffmerge solved that missing part in Angular. However xliffmerge became an abandoned project that luckily someone else picked up but then I’m guessing if a single developer can support it then surely the Angular team can create this missing part.
2. Do options A or B resolve this pain?


• OptionA solves it yes. OptionA seems the more logical step forward for developers that don’t use external translation tools or specialised translation companies. I’m assuming optionA features is based on xliffmerge ?
• OptionB, although has benefits, adds complexity for how we are using Angular i18n and defining the i18n messages.

• OptionC would be to leave the existing Angular CLI code as is and have an official fork of xliffmerge project and create official Angular documentation how to use it and making sure the code continues yes to work on each Angular version.

3. How well would options A or B work with existing translation tools?
   We don't have experience with all the translation tools out there or the technical expectations in this area.
Don’t know. We are not using any translation tools and directly updating the XLIF files in our IDE
4. Are all the translation file formats compatible with both solutions? I suspect XMBs will be awkward with option A, but I don't think it's a hard blocker on that one.
   
Don’t know as we are only using XLIF and does the job.

Working most basic working Angular 13 i18n project that does i18n merging
This ZIP file is a working most basic Angular 13 project using HTML i18n and TypeScript $localize feature that actually merges all changes to XLIF2 localised files.
Do: npm run extract-i18n and localised message.xlf files wil be generated in de, en, nl, fr, it. See directory /src/i18n/
angular-i18-flow.zip

Hope this helps someone.

Once again thank you Angular team for standardising this feature.

[dgp1130]

Since the feedback seems to be mostly position with option A, we'll move forward with that. We agree this is something the CLI should do, though right now this particular issue is a lower priority so it might be a while before we get around to it. The harder part is making sure we understand the issue and have a path forward with a fix, which is where we are now. If anyone is interested to contribute this feature, we're also happy to work with you to make that happen.

[PowerKiKi]

@daniel-sc, would you be willing to use your experience as the maintainer of ng-extract-i18n-merge to create a PR for Angular ?

[daniel-sc]

@PowerKiKi Thx - appreciate the invitation. But..

• (Sadly) I do not have enough time. (My guess is) this takes some full time weeks, but I can only carve out a couple of hours per week.
• The minimal scope is unclear (to me). Would it suffice to implement this only for xliff 1.2/2.0 or is it necessary to include all supported formats?
• The solution I built relies on some 3rd party libs (xmldoc, js-levenshtein) - would it be reasonable to add these as dependencies?
• For me the custom builder works just fine, so I do not really see incentives to (personally) spend much time on this. Either the angular team takes it on, or - for the time being - you could endorse the custom builder via your docs - see eg docs (docs-infra): i18n steps for subsequent changes angular#46471

HTH