Suggestion: i18n support

[vzahradnik]

I'm currently working on a firmware which will eventually support multiple languages. Eventually I will need a way how to make it work with tcMenu. Perhaps we can come up with a unified solution which could serve others as well.

Arduino library with gettext support

Due to memory constraints, i18n support is not that easy. I'd like to make it compatible with GNU gettext, especially because there are tools from translators already available (see Poedit for example). Another great benefit is deduplication of strings - If I write in two places the same string, gettext stores it only once. Not to mention other cool stuff like detection of changed strings so that translators check and update just some of the lines.

I'm not aware of any Arduino library dealing with this. My guess is that people use that mostly for hobby or small projects and they just don't have the need for supporting users from different countries.

At this point, I introduced a simple macro to mimic the gettext API. I don't have any implementation now.

#ifndef LUTEMI_STRINGS_H
#define LUTEMI_STRINGS_H

#define _(String) (String)
#define N_(String) String
#define textdomain(Domain)
#define bindtextdomain(Package, Directory)

#define EMPTY_STRING ""

#endif // LUTEMI_STRINGS_H

It is used in code like this:

// Instead of regular string "Network manager..." we wrap it with _("...")
Log.noticeln(F(_("Network manager: Connecting to WiFi %s...")), storeManager.getNetwork().getWifiSsid().c_str());

Now we come to TcMenu...

TcMenu limitations

• It doesn't take such a usecase (multilanguage support) into account
• For example, boolean entries can have values TRUE/FALSE, ON/OFF, etc. There is no way how to localize them.
• Strings are stored in the project file *.emf from which TcMenu generates C++ files
• Strings have length limits. This is not strictly locale issue but try to imagine a scenario where the menu is displayed on a smartphone and on a small OLED display. Smartphone can display much more text. Perhaps TcMenu could offer two variants of labels - one for the embedded part, the other one more verbose. These strings can be pulled via TagVal protocol and smartphones just load the text dynamically
• Project file contains strings only from a single language, usually English

My workaround

• I created two project files (one for each language). Now I need to manually generate files for each language and compile the code. This is tedious, especially when I still develop the menu and change things frequently.
• My smartphone app will have its strings properly stored for each language. From the TcMenu I will be interested only in types, IDs and values. It solves the above mentioned issues - strings can be longer and multiple languages are supported

How to deal with it properly?

This could be subject of our further discussion.

[davetcc]

Agreed on all fronts that this is important and comes up quite frequently, in the Java UI / generator domain, the work could be quite easily done with resource bundles, the language has built-in. On the C++ side, agreed something needs to be looked at here, but for the use cases I usually have, it has so far not been a priority. What we must not lose sight of is Uno support, along with many others, I still have a couple of Uno boards running in production, so whatever we do would need to be memory efficient or at least optional.

For the name, from the next release, 3.0 you'll be able to change the name length and the unit length very easily at compile time, along with quite a few other compile flags that have been added. These will be documented properly once the 3.0 release goes ahead.

What I have prioritized in this release is full UTF-8 Unicode font support that works across every display library. It works really well, better than I expected, and is compatible with nearly all TcMenu display plugins. #256 and #238

The problem with String as I understand is it will allocate memory at runtime, many people really don't want that on embedded boards, and try to avoid allocations where possible. What about smaller boards with 2K, 8K, or 32K RAM where the heap will fragment very quickly?

[davetcc]

I think what I'm trying to say is that as long as the way it's done doesn't rely on allocation in the main loop, except for allocating one-off buffers and stuff, I'm open to most possible solutions. Given how I tend to use tcMenu I'm probably not best placed to spec this one out. Unlike the unicode support, where how it should work is pretty clear, with this there is more complexity.

Maybe the best thing to do is to try and put together some kind of showcase "example" build, where you just hack and move things around until it does something pretty close, and we could discuss. It's what I often do to be honest to get started on something. I'd be happy to make the changes on the Java side and help with the changes on the C++ side.

[vzahradnik]

My idea how to keep things efficient was to leverage code generation and plug it somehow into PlatformIO so that the necessary code will be generated as part of the automated build. This could work also for TcMenu, as we can call the generator via CLI. I would love to have generated code separate from main code... but it's just nice to have.

I don't know yet how to make it gettext-compatible but I will do some experiments.

First things first... Python/Dart library is my top priority. Later I will look into this issue as well.

[davetcc]

The first step to this has gone into tcMenuLib at V3.1, all the hardwired strings such as "Yes" "No" etc are now loaded using a TC_LOCALE and selectable, I've added English and had a go at French. See TcMenu/tcMenuLib#176

This is only for the constants within the library itself, of which there are not many, hence the very simple solution for now.

[vzahradnik]

Sounds good. I made a PR with support for Slovak and Czech language.

• Do these constants affect how the message is encoded on the wire with TagVal?
• What about client libraries? We could include similar support to the Java/Python version as well. During initialization we could specify an optional parameter Locale defaulting to English.US. Libraries would not share the translations but considering the number of strings is quite low, I don't see any problem with that approach. We can just create similar lang structure in each library.

[vzahradnik]

By the way, I like the approach you chose. This could be adapted for the menu strings as well. Instead of generating different cpp files containing code with specific language strings, we could instead generate files with all language strings included. Users could define the language using build flag like in the case of TcMenuLib.

This would involve generating similar structure to the tcMenuLib:

• TcMenu would generate /i18n/ directory containing language translations
• A proper language would be included based on build flag, defaulting to English
• We need to figure out how to generate unique defines which would be referenced in the generated code

[vzahradnik]

Hi @davetcc , I'd like to move with this issue and here is my proposal:

TcMenu Designer:

• In project settings we'll set the list of supported languages. If not specified, Users could select multiple languages from the list. By default, English will be pre-selected and used.
  
• Or as an alternative, just allow users add new language by hand. Like when they configure IO expanders. What we need is language code, e.g. en_us.
  
• Each menu item will have a new field, Language selector. By selecting a language, you can edit the Name attribute for the given language.
  

Generating the code:

• TcMenu will generate files inside <project-src>/i18n/ directory
• Each defined language will have a corresponding file, e.g. en_us.h
• If user didn't specify a translation, a default language string will be used

Generated code proposal

• Let's say we support two languages, English and Slovak
• TcMenu will generate filenames: en_us.h, sk_sk.h
• The content of the files could be as follows:

# define <MenuVariableName>_STRING "Translated string"

Now it's simple. We just need to include proper file based on some flags (your initial support already works with them). And the variables need to be modified from:

const PROGMEM SubMenuInfo minfoDigitalInputs = { "Digital inputs", 58, 0xffff, 0, NO_CALLBACK };

to

const PROGMEM SubMenuInfo minfoDigitalInputs = { DigitalInputs_STRING, 58, 0xffff, 0, NO_CALLBACK };

This solution is not perfect, but it should be easy to implement. For most projects I think it's more than enough. We're not developing a new business applications with thousands of lines of strings.

Let me know your thoughts on this and also I'm willing to help because I obviously need this feature.

Thanks!

[davetcc]

Agreed, this is a good solution. I've just released 3.1 yesterday so given this is an important feature requested by many people it should be next. On the designer side, if we use the Java inbuilt Locale and resource bundle support, we get a good deal of what we need for free.

Depending on how much time you have I could go through the Java generator code with you and build the core support into the designer and the generator. Feel free to ping me by email to discuss..

I think this should be optional so that if only one language exists, it can still work exactly as it does now.

[vzahradnik]

I reserved my weekends for this so definitely I'm available. I'll ping you via email.

[vzahradnik]

I think this should be optional so that if only one language exists, it can still work exactly as it does now.

I agree. You are obviously more familiar with the code of generator but function-wise it makes sense.

[davetcc]

So, I've started off by applying resource bundles to the UI itself, so I better understand how they work. They are really straightforward and there are a good number of editing options, although the file format is trivial (properties) and well-supported by many editors. They are stored and read in UTF-8 by default, making editing most languages easy. I now know how to apply it to both the plugins and menu editing once I've got the UI finished up.

[davetcc]

Feel free to reach out by email @vzahradnik if you'd like to go through the plans in more detail, and thanks for the continued support, it helps me with the costs of keeping things running.

[vzahradnik]

I appreciate you work on my suggestions. Your library saves me a ton of work, so I contribute back.

Let me reach out by email.

[vzahradnik]

I see you are moving strings into separate files. Once you finish this, just write here. I'll provide translations for Slovak and Czech so that it gets mainlined in the same release. It looks like there's not that much work with the translation.

[davetcc]

I'm going to open a discussion around internationalization

[davetcc]

#316

[davetcc]

Closing this for now, from this point on, the releases will be incremental releases mainly fixing outstanding issues. In each one, I'll try and internationalize extra dialogs. The menu support for I18N is in place now and initial testing shows it to be quite simple to work with.