Proposal for a Simplified I18N Process

[HansMuller]

Update

The proposal that follows led to a new i18n system for Flutter applications. The new system is part of Flutter 1.19beta and will be part of the following stable release.

A user guide for the final system can be found here: https://flutter.dev/go/i18n-user-guide. A web tutorial is forthcoming.

Introduction

This is a proposal to provide a tool that simplifies setting up and maintaining an internationalized Flutter app. It is based on the Dart intl package. The overall goal is to reduce the number of steps required to use that package in an app with modest i18n requirements.

Current Internationalization Process

Currently, internationalized apps which are based on the intl package must follow a procedure similar to the one outlined in the stocks example. Roughly speaking there are 5 steps:

• Add calls to the static Intl.message and Intl.plural methods where message strings are needed by the app.
• Generate a template ".arb" (application resource bundle) file with intl_translation:extract_to_arb.
• Make copies of the template ".arb" file for each supported locale, and add message translations.
• Generate several source files which collectively encode all of the localizations using intl_translation:generate_from_arb.
• Before the app starts, call initializeMessages() for each supported locale.

Proposal

The proposed approach is to start with handwritten ".arb" files. One of these files will contain the standard per-message metadata required by the intl package.

A new tool will be used to generate a single class that contains one method per message.

Message lookup is based on the inherited locale, using the current BuildContext:

MyLocalizations.of(context).myMessage()

The generated class will provide a LocalizationDelegate which is to be included in the app's list of localization delegates. The generated class itself will depend on intl package; the rest of the app doesn't have to.

This approach, based on segregating the app's localizations in a single class, is similar to the stocks example.

Prototype App and Tool

A prototype of the tool is #41432.

The main elements of the app, including the generated code, are as follows.

The English locale ".arb" file, lib/l10n/demo_en.arb defines the messages. The Spanish locale arb file, lib/l10n/demo_es.arb, just defines message translations.

The generated localizations class, DemoLocalizations in lib/1l0n/demo_localizations.dart, provides all of the messages.

The app includes DemoLocalizations.delegate in its localizationDelegates list, and looks up its messages with DemoLocalizations.of(context).

[escamoteur]

Have you seen this package https://github.com/timfreiheit/r_flutter?
Also just a thought wouldn't it even more comfortable if the input would be one cvs file with all languages in it that easy can be edited with any spread sheet application, so easy to give to translators.
Also have you seen https://marketplace.visualstudio.com/items?itemName=esskar.vscode-flutter-i18n-json

To make is really comfortable you should be able to directly create a new word entry from the IDE.

[alan-knight]

The target audience for this are people who doing their own translations by directly modifying the data files? I would have thought most people were using some sort of tool or translation service, but it seems not. Given that it seems reasonable enough. It seems important to consider migration issues when someone scales up to using a tool/service. I think they could probably still generate that way, but could this reasonably support multiple classes/multiple ARB files? Is that something people need/use in larger Flutter applications?

Intl doesn't currently handle formatting of parameters, so if you have something that takes a date or a number, you either have to format it before passing it in as a parameter or it will get the default format. So the rest of the app might still need to depend on intl for that.

Not directly relevant to this proposal, but Google Translator Toolkit, which is the primary thing that supported the ARB format is deprecated. We will really need to support at least one other format. There is a pending bug on intl_translation to make it so that anyone can write a reader/writer external to the package. There's also a converter someone wrote in JS, between ARB files and "Hierarchical JSON" format and it could probably handle others.

CSV files with Unicode in them makes me nervous of escaping issues, but there are a lot of formats. As above, being able to handle lots of them seems good, but we may want to recommend a default.

Does this/can this reasonably combine with the ability to use native translations on mobile platforms? So, would people be more comfortable with editing e.g. Android resource files directly?

There's been a bit of work ongoing on being able to read ICU MessageFormat strings at runtime and format them, as Closure does. That would be another alternative, although it's an entirely different (and much less typed) API so would be a significant change. And those would still have to be embedded in a file format.

[escamoteur]

@alan-knight what is the problem with CVS and Unicode?
I still think a format that can be read by office applications would make it much easier.

Another point not to forget is how can we support singular / plural versions of words in different languages?

[HansMuller]

It seems important to consider migration issues when someone scales up to using a tool/service. I think they could probably still generate that way, but could this reasonably support multiple classes/multiple ARB files? Is that something people need/use in larger Flutter applications?

The tool could definitely support generating multiple classes for different sets of ARB files. That does seems like a nice kind of flexibility. I don't know if developers need that kind of support now, although I'm hoping we'll hear from them here.

Intl doesn't currently handle formatting of parameters, so if you have something that takes a date or a number, you either have to format it before passing it in as a parameter or it will get the default format. So the rest of the app might still need to depend on intl for that.

Yes, that's correct.

Not directly relevant to this proposal, but Google Translator Toolkit, which is the primary thing that supported the ARB format is deprecated. We will really need to support at least one other format. There is a pending bug on intl_translation to make it so that anyone can write a reader/writer external to the package. There's also a converter someone wrote in JS, between ARB files and "Hierarchical JSON" format and it could probably handle others.
...
Does this/can this reasonably combine with the ability to use native translations on mobile platforms? So, would people be more comfortable with editing e.g. Android resource files directly?

The ARB format isn't essential to this proposal, however it's what we've recommended that Flutter developers use to date. We could support native message catalog formats, per #37031, as well.

There's been a bit of work ongoing on being able to read ICU MessageFormat strings at runtime and format them, as Closure does. That would be another alternative, although it's an entirely different (and much less typed) API so would be a significant change. And those would still have to be embedded in a file format.

It would be nice to support a platform-independent and widely accepted standard. Assuming that the right API to expose to an app was still a set of methods that returned strings, supporting ICU (or ICU too) sounds like a win.

[Tomucha]

Hi! I'm all for handwritten message files and code generation. The compile-time check of the message existence is great. But don't forget that occasionally you need to decide the message at runtime, maybe like this:

var vehicleTypeMessageKey = "VehicleType.${data['type']'}";
... so having just generated methods is not enough.

You will find a few more ideas like this in my internationalization package :-)

https://pub.dev/packages/i69n

[ashour]

Hello, I hope you're all well. Thanks for your work on Flutter btw. It's an awesome piece of tech and I wish I had Flutter when I was co-founding a mobile app startup years ago. I was recently emailed by @HansMuller, who asked me to chime in. I research and write about i18n. I wrote an article on Flutter + Intl last year (I think that's how Hans found me).

I think this is a good proposal because the current Intl workflow is a bit clunky for smaller projects. This is especially true for projects that don't have automation and sophisticated tooling around i18n. Given this leaner alternative, I think two main pain points from a developer perspective warrant a look:

• Reducing manual script running as much as possible
• Using quick, lean file formats for messages that can be easily shared with translators

RE reducing manual script running, I've always wondered while using Flutter + Intl if there could be a watcher that generated and updated message files on the fly while developing. I've actually never seen this in any i18n library and I've always wanted it. I don't know how feasible this is given available Dart code analysis tools VS what would need to be written from scratch for such a watcher.

As for file formats, there are a few I think worth considering (not exclusively, of course, just ones that come to my mind):

• ICU
• PO
• CSV
• JSON
• YAML

ICU and PO are widely used. PO has good adoption among i18n tools and platforms for developers & translators. For a smaller team, however, I think the overhead of generating and compiling ICU or PO is more of a hassle than it's worth. The other options are much more lean and to-the-point for smaller projects.

CSV is attractive for sharing with translators, although I can see it being a pain for developers. CSV headers can be far removed from their respective values, which can make managing messages tricky. Think of a plural message in Arabic with the language's 6 plural forms. How could a developer tell them apart? You could add an identifier to each message component, but that would take away from translators' experience.

JSON would be nice for us coders, but it may be a bit of a hassle for translators.

YAML could be a nice in-betweener for both programmers and translators. YAML has a nice, intuitive syntax with significant white space that I think would be picked up easily by translators. It also allows for various types and hierarchies, which make it flexible enough accommodate plural forms and more complex messages.

It could be worthwhile to pull in some translators that work in software and get their thoughts. They may well have their own bread-and-butter tools that, say, work well with JSON and have wide adoption. In that case JSON would be a very attractive option since we'd really only be solving the developers' problem, not the translators'.

These are my two cents so far. I hope they help a bit.

[HansMuller]

@Tomucha - Thanks for the feedback. I think we've taken a similar approach to simplifying building i69n applications :-).

But don't forget that occasionally you need to decide the message at runtime, maybe like this:

var vehicleTypeMessageKey = "VehicleType.${data['type']'}";
... so having just generated methods is not enough.

I agree that this sort of thing isn't necessary to solve with generated methods. I would think the app would have to include a method that switched on vehicleTypeMessageKey and returned a localized string.

String vehicleType(String vehicleTypeMessageKey) {
  final MyLocalizations localizations = MyLocalizations.of(context);
  switch (vehicleTypeMessageKey) {
    case 'sedan': return localizations.sedan();
    case 'truck': return localizations.truck();
    ...
  }
}

[Tomucha]

At this point i69n generates the exact switch statement you are suggesting ;-)

[szotp]

@ashour what's wrong with .arb files? Flutter already uses them, and they are esentially jsons.

EDIT: Also, the watcher you mentioned kinda exists already. Packages using code generation ( https://github.com/flutter/flutter/wiki/Code-generation-in-Flutter ) already auto update when app is running. The only problem is that analyzer does not seem to pick it up immediately and displays fake errors.

@HansMuller I agree that current state is a mess. Did you consider doing flutter generate builder package? I implemented this in r_flutter and it worked decently well (although IDE & analyzer are not really prepared to handle it). Such package could be eventually embraced by flutter team.

[HansMuller]

@ashour - Your article on internationalizing Flutter apps is an excellent introduction to this topic!

I think two main pain points from a developer perspective warrant a look:

• Reducing manual script running as much as possible
• Using quick, lean file formats for messages that can be easily shared with translators

RE reducing manual script running, I've always wondered while using Flutter + Intl if there could be a watcher that generated and updated message files on the fly while developing. I've actually never seen this in any i18n library and I've always wanted it. I don't know how feasible this is given available Dart code analysis tools VS what would need to be written from scratch for such a watcher.

That's an excellent point. We can definitely do this, even in the context-of hot-reload, in roughly the same way that Flutter handles asset changes.

As for file formats, there are a few I think worth considering (not exclusively, of course, just ones that come to my mind):

• ICU
• PO
• CSV
• JSON
• YAML

Currently the "ARB" format, which is used by the Dart intl package and the proposed tool, is mostly just a JSON container for a collection of ICU messages. Additional support (beyond what's in the prototype) for dates, currencies, numbers, gender would move us from the realm of "modest" i18n Flutter apps, to more ambitious ones. I think that would be a good direction to take in the long run.

Support for other message catalog formats came up earlier. We could support other formats with additional code generators or perhaps with a scheme that converts other formats to a common one. XLIFF is another popular alternative.

These are my two cents so far. I hope they help a bit.

Thanks for your feedback!

[HansMuller]

@szotp - thanks for your feedback.

I agree that current state is a mess. Did you consider doing flutter generate builder package? I implemented this in r_flutter and it worked decently well (although IDE & analyzer are not really prepared to handle it). Such package could be eventually embraced by flutter team.

I'll take a look at r_flutter.

[escamoteur]

Looking back when I did.Net programming there was a powerful free Application to manage the resource files including text resources in different languages. That could also in and export different formats.

So the used format might not be so important for users if someone writes a flutter desktop app for this.

So maybe

[alan-knight]

Dart has a lot of things that rely on code generation and mechanisms for automatically re-generating on changes, e.g. build_runner. So I would expect automating that to be fairly easy. But I don't think anything is integrated with flutter right now, so I guess you would have to manually run build_runner on the side and have it watch the files.

[jonahwilliams]

I don't think we want to require a dependency on build_runner to generate translations.

[alan-knight]

No, build_runner is pretty heavy for that. But I don't think it would be difficult to do something custom that did file watching and triggered a re-generate, or to use other mechanisms.

[escamoteur]

Just one thing that came to my mind. If this gets sort of service think about adding an interface that can be easily used by the IDE extensions for VS code and IntelliJ.
With a function in the editor to insert a new identifier together with a default language that then creates an entry in the ARB(or whatever) files would make you really productive with internationalisation.

[orestesgaolin]

Support for ARB is coming to more and more tools as this is the default for Flutter. Recently there was new open source tool announced called Arbify. This is a self-hosted tool that can be run in Docker container, supports multiple projects, plural and genders.

[HansMuller]

@orestesgaolin - thanks for the pointers and for the reminder that it's time to acknowledge that the work proposed in this PR has been completed. The last i18n tooling change was #57510, and the completed system part of 1.19 beta and will land in the next stable release as well.

There's a user guide here: https://flutter.dev/go/i18n-user-guide

[jbojcic1]

@HansMuller when can we expect the next stable release which will include this?

[HansMuller]

@csells - can you let @jbojcic1 et al. know when to expect the next stable release?

[csells]

@jbojcic1 we haven't announced the next stable release, but you should expect it in the next month or two.

[jsroest]

@shihaohong and @HansMuller
Error: Could not resolve the package 'args' in 'package:args/args.dart'.

When following this guide https://flutter.dev/go/i18n-user-guide step-by-step (on the beta channel), I ran into the same problem as described in flutter/samples#331

Running: flutter update-packages in ~/development/flutter/packages/flutter, solved it for me.
flutter/samples#331 (comment)

[kuhnroyal]

Running flutter update-packages in the SDK is not really a solution. Especially in CI when switching between Flutter versions a lot. This should be fetched when installing by default.

[kuhnroyal]

@HansMuller @shihaohong Is this now the promoted way for i10n with Flutter 1.20.0?

A couple questions, maybe someone can answer some.

1. I need to manually run pub get in flutter/dev/tools for the generator to work - Will this automatically be installed in the future? Since the whole process starts by itself when I add the i10n.yaml file, I would expect it to be ready to use at that point.
2. What is the recommendation for the generated Dart files, should they be committed to VCS?

[shihaohong]

@kuhnroyal

1. I need to manually run pub get in flutter/dev/tools for the generator to work - Will this automatically be installed in the future? Since the whole process starts by itself when I add the i10n.yaml file, I would expect it to be ready to use at that point.

I'm currently looking into fixing this. This is a result of the gen_l10n tool itself being a package, which is why it requires a pub get call since its dependencies change between flutter versions. I'm looking into seeing if I can pre-compile the dependencies with the tool itself so that it does not require re-caching and updating its dependencies.

2. What is the recommendation for the generated Dart files, should they be committed to VCS?

If you're going the synthetic package route (which we're trying to make the default way of doing things through #62395), then the goal is that generated Dart files will not need to be committed to VCS. If you'd still prefer to check in the generated code into VCS, you can turn off the synthetic package option with the ways mentioned in the PR depending on whether you're using an l10n.yaml file or running the tool directly via the command line.

[kuhnroyal]

@shihaohong Thanks for the clarification and your work on this!
I would love to not commit the generated files but in my initial tries I ran into problems running flutter test in CI. The files are not generated before running tests, that would currently leave me with the option to run the tool directly beforehand or committing the files.
Are the plans to make this tool available as generator so it can be tied into a build_runner call?

[shihaohong]

@kuhnroyal Actually, the current issue I'm trying to resolve is getting flutter analyze to figure out this situation. What's happening is that flutter analyze fails sometimes because the l10n code had not been generated beforehand. So, I'm trying to get flutter analyze to run flutter pub get as a step (which I'm trying to have call the synthetic package generation code) so that it does not fail for this reason.

I had not considered that if would fail with flutter test. Could you share more about the kind of problems you're running into? Also, what does build_runner do with code generation that you would like to see? (I'm not as familiar with using build_runner).

[kuhnroyal]

@shihaohong From my understanding with the synthetic route (i10n.yaml) I would not commit the files to VCS. When I checkout a project I am usually able to run flutter pub get and flutter test which would compile the code and run the tests. But now the localization files are missing as they are only being generated during flutter run/build so the tests don't compile.

The question about build_runner is related to my test problem. Many projects use build_runner generators to generate code, mostly *.g.dart files for all kinds of libraries, built_value, moor, freezed and lots of other i18n libraries. In most cases the *.g.dart files are also not committed so that a dedicated build step is required before running tests or building the project (flutter pub run build_runner build) and many CI pipelines in a lot of projects already use this. There is already an established config file format build.yaml via https://pub.dev/packages/build_config. I was just wondering if there are plans to provide a generator for build_runner.

[shihaohong]

@kuhnroyal Great, thanks for the feedback/explanation! I'll take a look into how we can address that in the next few days. I might create a separate issue to track it as well since this one is already closed and will loop you in once I do that.

[shihaohong]

@kuhnroyal My synthetic package PR (#62395) incorporates synthetic package generation into flutter test and flutter analyze invocations when flutter pub get is required, so that should address the issue you mentioned with running tests and compiling code.

As for providing a generator for build_runner, we're currently not planning on incorporating synthetic package generation into build_runner, but this shouldn't be a problem as long as there isn't a problem with the synthetic package generation code. Please feel free to file a new issue highlighting any concerns or problems you face with the gen_l10n tool, I'd be happy to look into and try to address them.

[Sominemo]

https://flutter.dev/go/i18n-user-guide

I noticed the output-dir parameter from the docs doesn't exist in code

[shihaohong]

@Sominemo Thanks for the heads up! I need to make some updates to the user guide, some of the instructions have changed with some updates to gen_l10n.

Edit: I made the updates to the user guide. output-dir definitely exists, but it's only relevant if you're not using the synthetic package strategy that is on by default. Meaning, the generated code will always go into {flutter_project}/.dart_tool/flutter_gen/gen_l10n by default. If you turn off synthetic packages (synthetic-package: false), it will generate the localizations code in output-dir.

[kuhnroyal]

@shihaohong Did this make it into the new beta?

[shihaohong]

Yes, it looks like the synthetic packages commit made it into beta.

[kuhnroyal]

@shihaohong The synthetic package seems to break build_runner as it finds the .dart_tool/flutter_gen folder and expects a pubspec.yaml there.

Bad state: Unable to generate package graph, no `/Users/foo/workspace/test/.dart_tool/flutter_gen/pubspec.yaml` found.
#0      _pubspecForPath (package:build_runner_core/src/package_graph/package_graph.dart:235:5)
#1      _parsePackageDependencies (package:build_runner_core/src/package_graph/package_graph.dart:208:21)
#2      PackageGraph.forPath (package:build_runner_core/src/package_graph/package_graph.dart:97:33)
<asynchronous suspension>
#3      PackageGraph.forThisPackage (package:build_runner_core/src/package_graph/package_graph.dart:110:20)
#4      main (file:///Users/foo/.../build_runner-1.10.2/bin/build_runner.dart:27:49)
#5      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#6      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)
pub finished with exit code 255

Other things I found:

• I can't get synthetic-package: false to work, no code is generated.
• In a small example with the synthetic package, the import for the AppLocalizations class can not be found/imported automatically in IntelliJ. Have to write it by hand to make it work.

[shihaohong]

@kuhnroyal The first issue you mentioned was also mentioned/filed in dart-lang/build#2835.

For the next two, could you create separate issues for them and tag me? It's easier for us to keep track of new issues with the tool that way since these problems can get lost in this issue's comment thread.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.