i18n plans?

[zyf722]

I'm working on a project that requires me to allow users to edit and run code in multiple languages in the browser. Then I found this project and it works perfectly for my needs.

I notice that currently all messages and texts are hardcoded in the source code. I think it would be better if we can move all these texts to a separate file so that it's easier to maintain and translate. Any plans or thoughts on this i18n feature?

[hatemhosny]

Thank you, @zyf722 for the suggestion.
This is indeed a good idea.

This is also related to providing a RTL layout, since a significant percentage of LiveCodes users have Arabic as their first language.

[hatemhosny]

I have opened an issue for this here:
#548

[zyf722]

Glad to see this is being considered. If possible, I would like to have a try to work on this.

I think i18next could be a good choice for i18n library based on following reasons:

• It allows to use translations type-safe with TypeScript.
• It supports pluralization and dynamic translations out of the box.
• There is multiple extraction tools to build translation files based on current source code from scratch.
• Online translation services like Crowdin have integration support for it.
• As for implementing RTL layout, it provides a dir function to get the writing direction based on the current language.

I might take some time to get familiar with the codebase and other current existing i18n implementations, then try to make a demo to validate the feasibility of this proposal.

[zyf722]

After some research, I believe the hardest part of this challenge is to find out all hard-coded UI strings out of other string literals. String concatenation shall also be considered since it is indeed a common way to dynamically generate strings.

Frustatingly, there seems to be no easy way to do this. Existing tools either are not designed to handle this task (instead, they basically extract keys like i18n.t('some.key') from source code to json files, not original hard-coded strings) or might get messy with code as they try to turn code into AST and then convert it back.

The only choice left is to manually search for hard-coded strings. But ahead of that, it is necessary to understand where to look for. After a quick look at the codebase, I wonder if all we have to check is src/livecodes/html and src/livecodes/UI folders.

@hatemhosny Any idea if this is the right approach? If so, I will start working on it. Otherwise, please let me know what you think. Thanks!

[hatemhosny]

yes, @zyf722 , I see what you mean.

i18n was not put in mind when developing LiveCodes, so no effort was made to isolate user-facing text.

The only choice left is to manually search for hard-coded strings. But ahead of that, it is necessary to understand where to look for. After a quick look at the codebase, I wonder if all we have to check is src/livecodes/html and src/livecodes/UI folders.

exactly as you say, most UI strings can be found in those 2 folders.

html files in src/livecodes/html are imported as strings by src/livecodes/html/index.ts (using esbuild text loader). So string manipulations can be done there before re-exporting.

Other much fewer strings will be elsewhere, like tool names: https://github.com/live-codes/livecodes/blob/develop/src/livecodes/toolspane/tools.ts#L29C1-L42C5

If you start, I can help or guide for the rest.

Please note that I would prefer lazy-loading language resources for the selected language only.
Also the language should be added as a user config so that it can be saved for later sessions (I would be happy to do that).

Thanks a lot for willing to help.

[zyf722]

Here's some progress about infrastructure and HTML-related work. All changes could be checked here in my fork.

Infrastructure

1. i18next with its backend (mainly to implement lazy loading) and browser language detector are added as dependencies.
2. New build script scripts/i18n.js is added to generate locales folder. Changes are also made in scripts/build.js to integrate this script into current build process.
3. New commit type i18n is used to mark commits related to .json translation files and other non-code-level changes.

HTML

html files in src/livecodes/html are imported as strings by src/livecodes/html/index.ts (using esbuild text loader). So string manipulations can be done there before re-exporting.

As for static HTML, I checked replaceValues function in src/livecodes/html/index.ts, however found it seemingly impossible to apply translation here, as the translation resource might not be loaded yet at this moment.

Current workaround is to add i18n class to the elements that will be localized, and change their inner text to translation key in HTML files. (However, this probably make the HTML files less readable and harder to maintain. I'm considering to use an attribute to store translation key instead, while keeping the inner text as fallback.)

Then, these elements will be processed in their container's corresponding handler in src/livecodes/core.ts. To avoid code duplication, maybe it is better to create a new function to handle this kind of localization and move selector-related code to src/livecodes/UI/selector.ts.

To preview, you may want to change fallbackLng in src/livecodes/i18n/index.ts to zh-CN to see the effect. I don't include that since I'm using Chinese as my browser language so it's not necessary for me.

That's all for now. Please let me know if you have any suggestions or questions.

By the way, is it still available to assign the issue by commenting .take? If so, I will do that soon.

[hatemhosny]

@zyf722
Your approach does work. Nice job.

I have some comments:

• I agree. An attribute (e.g. data-i18n) for translation key is better than putting the key as inner text. The fallback is an added benefit.

• Instead of scattering text replacement logic in each corresponding handler, all screens are loaded in modal.show(). This would be a better place.

• Not all text is inner text. e.g. Tooltips are in data-hint attributes. We may add another attribute for the element property that needs to be replaced (e.g. data-i18n-prop which defaults to innerText).

• Adding i18next library and dependencies adds ~77 kb to bundle (~22kb gzipped). This is probably acceptable for the full app. However, I do not want to add this for embedded playgrounds. Most UI screens with most text are not available for embedded playgrounds anyway. I currently provide 4 bundles (app, embed, lite & headless). I utilize tree-shaking to allow smaller bundles for un-needed features. I think i18n should only be available for the full app and not the others. So may be others, get a mock that just uses the fallback text similar to this.
  To be able to utilize tree-shaking, './i18n' should not be imported anywhere else, and any modules that needs to utilize it (e.g. modal) should get a reference to it (or the mock) as dependency injection.

• I add hash for all files (file checksum) to enable aggressive caching across versions. The hash changes only when file content changes regardless which app version. Then I cache it in the browser. We will need to do the same for locale files. This can be done later. Let's just get it working and we can add that later.

Let me know what you think.

edit: Of course, I would be happy to assign you to this issue. Just add a comment there so that I can assign you.

[zyf722]

Thanks for such a detailed explanation and guidance. I'll move the current logic to modal.show() and switch to use attributes to make it more clear.

Regarding the size and performance considerations, my workaround is indeed a bit rough as I didn't take into account the impact on the bundle size. I agree to enable i18n only for the full app bundle. However, I must admit that I'm not entirely familiar with the concepts of dependency injection and tree-shaking. Could you possibly provide some further information on how to implement these?

Once again, thank you for your assistance!

[hatemhosny]

I will open a PR in your repo with the proposed changes when I have some time.

[hatemhosny]

After checking bundle size, I found that the app minified bundle size increased from 313kb to 385kb after adding the i18n dependencies.
I still think this is unnecessary for most users who will use the English language.
So I built the i18n module separately and lazy-loaded it when another language is selected (in Config.appLanguage or browser settings).
The module is not loaded in embedded playgrounds.

The variable i18n either gives access to the module when loaded or is undefined. TypeScript will force us to check that it is not undefined before using it.

A helper function translate allows us to populate HTMLElements with translations. It can be used like that.

The type AppLanguage strongly types the list of supported languages for use in the SDK.

I have opened a PR in your repo with these changes

[zyf722]

All changes look great to me.

I only have one question about language detection in core.ts when no language is explicitly set. This seems to do the same thing as the language detector plugin (which has more ways to detect the language besides navigator.language). If we decide to use this simpler approach, we could remove the plugin and simplify the setup. What do you think?

Another thing to mention is that i18next offers namespace support which allows to further separate translations into different files. How many localized strings do you roughly estimate in the application? Should separate files be considered for better maintainability?

If there's no other major changes, I shall merge the PR and continue to work on the next steps once we reach a consensus on the above points.

[hatemhosny]

I agree on removing language detection plugin. Other ways for language detection are either not needed or already provided by the app.
e.g. query param - you can override detected browser language like this ?appLanguage=ar.
When we add a UI for manually changing language, the setting will be saved in the browser storage (TODO: we need to add appLanguage here for this to happen.)

Something else we need to consider. Currently the translate helper function only allows to replace a single property (e.g. textContent or title). Some few elements have text in more than one property. We need to think of an elegant way to handle this. However, we can keep this for later.

I think keeping each translation in a single file will be easier to manage and easier for contributors to provide more translations. However, the language-info file is quite large. Can we keep this file in a separate namespace for lazy-loading and have everything else together?

[zyf722]

If we choose to keep language-info in a separate namespace, then there is more steps to make sure it would be loaded when access the screen. i18next.loadNamespaces(ns, callback) provides a way to load the namespace on demand.

[hatemhosny]

To deal with multiple properties, is it possible to use data-i18n as a parent key and store properties needs to be translated space-separatedly in data-i18n-prop?

Nice 👌
Let's go with this

If we choose to keep language-info in a separate namespace, then there is more steps to make sure it would be loaded when access the screen. i18next.loadNamespaces(ns, callback) provides a way to load the namespace on demand.

I think this is reasonable. We probably need to do this once here (we may need to turn it to async). It is then called there.

[zyf722]

I think I found another issue that needs to be discussed.

How shall we handle elements like this (nested elements)? Apparently it is not a good choice to separate them into different keys totally based on tag since the context matters and word order may vary in different languages.

React integration for i18next has a component called Trans that can be used to handle this kind of situation. We might need to implement a similar mechanism in our project.

Are we going to use innerHTML in data-i18n-prop? Would this method pose any potential security risks? If we choose this method, we also need to change escapeValue in i18n.ts to false to prevent escaping.

[hatemhosny]

In this case, I believe setting innerHTML is safe enough since we have full control on the injected HTML in translation files. That would be much simpler than implementing such mechanism.
We are already injecting HTML from source files. This is probably the same.

[zyf722]

Alright, I believe there are no further questions as of now.

The PR will be merged soon and changes we discussed will be carried out. I will return here to provide an update once there is new progress.

[hatemhosny]

We may use Weblate to allow contributions for translation to different languages.
They offer the hosted plan for free for open-source projects. LiveCodes seems to be eligible.

This is a demo for the-algorithms.com: https://github.com/TheAlgorithms/website?tab=readme-ov-file#translating-the-website

We should consider this after extracting text to be translated.

[zyf722]

I only have some experience with Crowdin and another similar platform, but Weblate seems to be a good choice as well 👍

[hatemhosny]

Crowdin also looks awesome. I have no experience with either. You choose the platform to use.

[zyf722]

OK, then I might try starting with Crowdin first.

[zyf722]

New progress here.

Features discussed before have now been implemented, and some new features have been added.

Changes

• Synced with upstream develop branch in livecodes/livecodes repo.
• Removed i18next-language-detector plugin.
• Added appLanguage to getUserConfig in src/livecodes/config/config.ts. Query parameter appLanguage should work now.
  
• Modified translate function as discussed.
• Add lazy-loading support for language-info namespace. The json file will be loaded only when needed.
  
• Changed escapeValue to false in i18n.ts file to support HTML tags in translations.
  • TODO: Need testing.
• Change translation file from .json to .ts in order to support type-safe translations.
  • This will allow us to have some degree of IntelliSense for translations, to check if the attribute is valid. An key object must have at least one of the attributes defined in _I18nAttributes interface (an empty object or using properties not defined will lead to error).
  • Extend _I18nAttributes interface in template.ts to add more possible attributes that can be translated.

    interface _I18nAttributes {
      textContent?: string;
      title?: string
    }

  • i18n.js will now try to load .ts files in src/livecodes/locales folder and use esbuild to compile them in memory, then save the module's default export (the translation object) as .json file in build/livecodes/locales folder.

Comments

Type-safe translations

This is a feature that I come up with when implementing other changes. Please let me know if you have any thoughts on this.

Pre-replaced values in HTML tags

As for HTML tag support, I now find another thing to think about. Take the about message for example, it contains multiple HTML tags with pre-replaced values like {{REPO_URL}}. How should we handle this?

Abstract tags proposal

Besides, containing a large <a> element like <a href="{{DOCS_BASE_URL}}why#client-side" target="_blank" rel="noopener">client-side</a> in translation file seems to be redundant. I'm still thinking using abstract tags and creating a helper function to overwrite them when translating.

For example, we use <0>client-side</0> in translation file, and when translating, we can replace <0> with <a href="{{DOCS_BASE_URL}}why#client-side" target="_blank" rel="noopener"> and </0> with </a>. If necceary (like different target urls for different languages), we can also add attributes in the abstract tags and later use them to overwrite the fallback values.

This is just a thought, and I'm not sure if it's a good idea because more efforts to parse HTML and replace tags are needed.

[zyf722]

As for pre-replaced values, we could utilize interpolation to replace the values in the string. Codes might behave similarly to the replaceValues function. Any ideas to prevent code duplication?

[zyf722]

@hatemhosny Could you please take a look here at your convenience?

[hatemhosny]

Thank you.
I will have a good look and get back to you.

[hatemhosny]

Great work @zyf722 . Thank you.

I like the type-safe solution.

I have started another PR with the following changes:

• extended type-safety to include translation keys.
• renamed template.ts to models.ts to be consistent with other type-only files.
• reduced redundency in translation files. Instead of requiring to supply attribute for all values (and only use them a few times), now the value can also be a string which will then be assumed to be textContent.
  e.g.

const translation = {
 welcome: 'Welcome',
 hello: { textContent: 'Hello', title: 'Hello' }
}

• I have made some changes to the build script:
  • calling esbuild.build inside a nested loop will lead to starting many processes. Instead, it is much more performant to pass all files to a single call as entryPoints.
  • now the output files are placed in the build/livecodes build directory with the pattern translation-{locale}-{filename}.json. TODO: we need to add hash to file names.

Regarding your question, I like your proposal (<0>client-side</0>). Please go ahead implementing that if you think this is the way to go.
If we need to reuse replaceValues function, we can move it to utils module and export it from there.

[zyf722]

Thanks for pointing out these issues and making the changes. I have merged the PR and will continue with finding a solution to replace those predefined values and implementing the abstract tag part.

PS: As there is a .vscode folder in the repo, I suppose you are currently using VSCode. I'm considering making full use of Custom Data Format to provide better Intellisense for data-i18n and data-i18n-props attributes. I'll come back here once there is something could be shown.

[zyf722]

Done with abstract tags and intellisense enhancement. Please check commits for details.

• replaceValues-related code has been refactored to re-use in i18n.
• Two i18n namespaces has been separated to two types in order to have better type checking.
• A new vscode-intellisense npm script has been added to package.json to generate html.html-data.json file in .vscode folder for VSCode intellisense enhancement. Currently it only contains data-i18n and data-i18n-prop attributes and value set for data-i18n-prop attribute, which is generated by extracting fields from I18nAttributes in src/livecodes/i18n/locales/models.ts using regex. Further extensions could be added here to provide better intellisense support as well for other parts of the project, like data-hint and other custom attributes.
• Abstract tags are introduced using DOMParser, which supports overriding specified attributes or adding new ones to corresponding element defined in .html files (e.g. <2 href="https://example.org">open-source</2> will override href attribute of the original element).
  
  • Add data-i18n key for the element and check console to see the abstractified innerHTML and array of extracted tags. Since this is implemented using missingKeyHandler, so make sure the key has not been added to the translation files yet.
    

I believe i18n integration is now functional enough for use. Please let me know if you have any suggestions or questions on above changes. If there is no further feature planned, I will mainly focus on marking elements that need to be translated in HTML files next.

[hatemhosny]

wow! this is impressive.

some minor comments:

• replaceValues-related code has been refactored to re-use in i18n.

Nice 👍

• Two i18n namespaces has been separated to two types in order to have better type checking.

This is fine. You may want to update the comment here to use the new type I18nLangInfoTranslation.

• A new vscode-intellisense npm script has been added ...

Beautiful 😍

You may actually add data-hint as you suggested.

I think the regex used here is too specific and can easily break with minor code change (e.g. if we later use Partial instead of RequireAtLeastOne or change it to interface instead of type). I suggest using a more generic regex like this: I18nAttributes.+?{([\s\S]*?)}

• Abstract tags are introduced using DOMParser, which supports overriding specified attributes or adding new ones to corresponding element defined in .html files (e.g. <2 href="https://example.org">open-source</2> will override href attribute of the original element).

I like this indeed 👍
Just a minor stylistic comment. I tend to use early return pattern, if there is a condition that wraps the rest of a function. It decreases indentations and I feel it is more readable.
So instead of:

const replaceElement = (node: HTMLElement) => {
  if (node.nodeType === Node.ELEMENT_NODE) {
    node.childNodes.forEach((child) => {
      replaceElement(child as HTMLElement);
    });

    const name = node.tagName.toLowerCase();
    if (name !== 'body') {
      // implementation
    }
  }
};

I usually do this:

const replaceElement = (node: HTMLElement) => {
  if (node.nodeType !== Node.ELEMENT_NODE) return;

  node.childNodes.forEach((child) => {
    replaceElement(child as HTMLElement);
  });

  const name = node.tagName.toLowerCase();
  if (name === 'body') return;
  
  // implementation
};

also, instead of:

const attributes =
(node.attributes.length > 0) ?
  Array.from(node.attributes).reduce((acc, attr) => {
    acc[attr.name] = attr.value;
    return acc;
  }, {} as Record<string, string>) : undefined;

I would reverse the condition to deal with the short path (undefined) early and continue with the logic, like this:

const attributes =
  node.attributes.length === 0
    ? undefined
    : Array.from(node.attributes).reduce(
        (acc, attr) => {
          acc[attr.name] = attr.value;
          return acc;
        },
        {} as Record<string, string>,
      );

just to keep the same style with the rest of the codebase.

• Add data-i18n key for the element and check console to see the abstractified innerHTML and array of extracted tags. Since this is implemented using missingKeyHandler, so make sure the key has not been added to the translation files yet.

Very nice 💯

In my local setup, I have prettier configured to run on-save. I noticed some formatting changes (mostly whitspace formatting).
You may want to run npm run fix before commits to run formatters and linters with autofix, to avoid later failure in CI when merging.

Apart from these minor comments, I think we are ready. Please go ahead.

[zyf722]

Got it. Thanks for suggestions on style and readability and I'll fix that soon :)

[hatemhosny]

TODO: we need later on to add some e2e tests for significant cases.

e.g.:

• no i18n -> use the hard coded strings in files
• embed builds (?embed and ?lite) -> no i18n
• i18n with string value
• i18n with object for different props
• abstract tags
• interpolation for predefined values

etc.

Also, whenever you see suitable, you may start a [WIP] PR, so that we can run all CI checks early and fix anything that fails. And then you can keep pushing more changes.

[hatemhosny]

We will also need to add something in the docs.

A point for thought:
Where do we allow the user to manually change appLanguage in the UI?

May be add another screen for global app settings.
This can also include global app themes (see #415 and #539 ). What else?

[hatemhosny]

Or just change "Editor Settings" to "App Settings" and add them there?

[zyf722]

Personally I'd prefer to have an independent App Settings screen with these appearance customization, including languages, about the app itself, while the Editor Settings continues to serve as options for the external editor only.

This also reminds me that we need to take editors into consideration when it comes to i18n. Currently the context menu of monaco remains in English. Maybe some changes need to be done about it.

[hatemhosny]

Needs some thinking.
Let's keep that for later.

[zyf722]

I think some changes here are worth discussion.

Further type-safety and nested keys

Type-safety and intellisense of i18n.t has been further improved.

• Now we can use i18n.t with auto-completion and key and interpolation validation.
• Nested keys are allowed to make translations more hierarchical and organized. As we discussed before, we use translation (the default one) and language-info as two namespaces. I think we could manage keys in the form of namespace:file.key1.key2... to make it more readable and maintainable, as illustrated here. Type-checking still works for nested keys - it will throw an error if two objects are not isomorphic.

Translation process

During add keys and translations for welcome.html, I soon found that it is laborious and error-prone to manually add keys in translation files. So I wrote a new script to automatically generate English .ts template files from HTML files. It works just fine with nested keys which is mentioned above.

I also take collaboration and online translation platform into consideration. The main problem here is that I think it'd be much better if translators can see the actual HTML elements for abstract tags in the form of context, description or etc. (as shown in the screenshot)

However, i18next's key-translation JSON schema doesn't support this kind of comment information, which then drives me to come up with the following localization process:

1. Write .html files with data-i18n and data-i18n-prop attributes and .ts files with concrete i18n.t("namespace:file.key1.key2...", "fallback") calls. These files are considered as actual English "source" of translations.
2. Generate .ts template files and .lokalise.json files for English from .html and .ts source-code files, which respectively offers type-safety and get used later to get pulled in Lokalise for online translation. These special .lokalise.json files are structured in order to provide context information for translators, and please note that they are different from .json files generated during building - those are in i18next JSON schema.
3. Translators directly doing translations on Lokalise without cloning the repo, building the app or making changes to any source code files.
4. After translations are done, Lokalise provides Github integration to automatically create pull requests for new translations once build is triggered by an admin. These new translations are not ready to be merged until they are checked by the action described in the next step.
5. An Github Actions workflow is set up to automatically run an import script when PR for new translations is created. This script will convert Lokalise's .json files back to .ts translation files we use in the project, and then commit these changes. If everything is fine, the PR could be then manually reviewed and merged.

Even myself must admit that I seem to have made this too complicated. However, there seems to be no better way to add context information except manually maintaining it as we use a custom format for translation files to have better type-safety and developing experience - we either have to give up providing context information for translators or make the process more complicated. Any ideas or suggestions shall be greatly appreciated.

PS: After doing some research, I believe Crowdin has been a little bit too strict for open-source projects, so I then turn to Lokalise.

[zyf722]

Please check full commits here to get more details.

[hatemhosny]

Some significant changes here :)
Let me have a good look and get back to you.
Thank you.

[hatemhosny]

@zyf722
Sorry for the delay, I was a bit busy.

So I have to admit, this is impressive. However, it is getting quite complex.

I like that contributors to translations can have much simpler workflow using Lokalise while having context, without having to mess with code. I also like the proposed GitHub actions workflow to automate incorporating the translations back into the code base.
This however comes at the expense of making things a lot more complex for maintaining this workflow.

If there is no simpler way, we may just accept that, and clearly document the workflow in contribution docs here, and refer to it under "Specific topics" here.

In this documentation we need to explain the full workflow (including when running scripts are required), with guides for:

• How to add a new translation?
• How to contribute to a tranlation in Lokalise?
• How to trigger merging the translation back to code?

I also have some comments for specific topics:

Now we can use i18n.t with auto-completion and key and interpolation validation.

This is great in principle. However, I see we are not making use of this since we are just looping through the keys in translate function.
I aknowledge that this does not add much to the complexity, and is a-nice-to-have feature. Well done by the way :)

During add keys and translations for welcome.html, I soon found that it is laborious and error-prone to manually add keys in translation files. So I wrote a new script to automatically generate English .ts template files from HTML files. It works just fine with nested keys which is mentioned above.

This is nice.
I'm not sure what this does:

    const type = namespace === 'translation' ? 'I18nTranslationTemplate' : 'I18nTranslationTemplate';

Also I think this is error-prone:

    const definition = JSON.stringify(trans[namespace], null, 2)
      .replace(/"([^"]+)":/g, '$1:')
      // .replace(/"(.*)\/\* COMMENTS \*\/(.*)"(,?)/g, "'$1'$3 // $2")
      .replace(/"/g, "'");

These are examples for how this can break: https://livecodes.io/?x=id/rxmym34u9va
I would suggest to use a proper parser for such things. Prettier is already a dependency in the project and it can do this much better. Example: https://livecodes.io/?x=id/zz7ss8qk3zg

This type does not allow a tranlation file to be missing top-level keys while the translation is still in progress:

type UnAsConst<T> = {
  readonly [K in keyof T]: T[K] extends I18nAttributes | I18nTranslationTemplate
    ? RequireAtLeastOne<UnAsConst<T[K]>>
    : string;
};

consider wrapping it with Partial or RequireAtLeastOne

This is indeed becoming a big project. However, I'm really grateful for you making this effort.
Thank you.

[zyf722]

Oh, I could be sure that the const type statement is a typo. I must have forgotten to edit it after some copy-pasting. 😂

You're right about using parsing instead. I indeed didn't notice that babel is a dependency of prettier. Such being the case, I will try to use prettier and babel instead. Thanks for pointing out this.

I'll also fix the definition for UnAsConst<T>. This should be a small mistake.

This is great in principle. However, I see we are not making use of this since we are just looping through the keys in translate function.
I aknowledge that this does not add much to the complexity, and is a-nice-to-have feature. Well done by the way :)

I don't think that translate function is the only place where we handle translations, since it is designed for HTML files. However we still have some strings in src/livecodes/UI, so I guess this feature is still needed there.

In summary, thank you for your detailed reply.

PS: I noticed that v27 was just released yesterday. Congrats!

[hatemhosny]

Thank you. Yes, that was a relatively big release 😀

I don't think that translate function is the only place where we handle translations, since it is designed for HTML files. However we still have some strings in src/livecodes/UI, so I guess this feature is still needed there.

It might be more efficient to run the translate function once on document.body after the UI DOM is loaded. May be somewhere in initializePlayground‎ which runs once when app is Initialized or in bootstrap which runs every time a new project is loaded.

[hatemhosny]

We need to think of a way to keep the translations in sync.

Consider the case when some text in HTML is changed. We need a check that detects that this key in all translations is no longer valid, so it should not be used and defaults to the text in HTML till the translations are updated.

May be add a checksum for the source string with translation key. This check can only run with production build. The build should fail if this is the case till we mark that this key should be ignored, so that we are aware of this situation. Specially that maintainers may not be able to read different translations or update them on their own.

[hatemhosny]

FYI
https://x.com/yijie4188y/status/1781062881553080435
https://github.com/NWYLZW/i18ner-releases

[zyf722]

Looks great! Actually I've discovered a similar plugin (lokalise/i18n-ally) for VSCode before. Glad to see that there's a similar one for WebStorm. Thanks for sharing! 👍

[zyf722]

These days I fixed few typo alongside small mistakes, and more importantly, briefly reviewed the entire thread and now have a to-do list below (ones with higher priority are listed first):

• Rewrite i18n-export.js with Prettier and Babel
• Find out a solution to add translation to src/livecodes/UI/*.ts files
• Add checksum for translation keys to keep sync
• Setup workflow on Lokalise
  • Apply for open-source plan
  • Create the project
  • Find a proper way to pull translations from source on Github automatically
    • Maybe there's something like timer trigger or commit trigger
  • Setup integration with GitHub to create pull requests automatically
• Write a i18n-import.js script to import translations from Lokalise
  • Integrate it with GitHub Actions
• Add hash for translations and put them in cache
• Add e2e tests
• Create a PR to trigger CI check
  • Solve issues if any found
• Add related documentation about
  • appLanguage
  • make i18n contribution on Lokalise
  • add a logo with a link to Lokalise in the credits section

Here are some comments on part of the tasks and need your help to clarify them:

Solution for src/livecodes/UI/*.ts files

It might be more efficient to run the translate function once on document.body after the UI DOM is loaded. May be somewhere in initializePlayground‎ which runs once when app is Initialized or in bootstrap which runs every time a new project is loaded.

I agree with the solution. So as I understand it, for codes like following (from src/livecodes/UI/assets.ts):

const type = document.createElement('div');
type.classList.add('light');
type.textContent = 'Type: ' + item.type;
detailsContainer.appendChild(type);

We basically just add data-i18n and data-i18n-prop (if necessary) to the elements, and move translating logic (using translate function) to initializePlayground or bootstrap function.

But I'm still a bit confused about why this could work. From my point of view, given that these elements are created dynamically, the translate function should be called in some event listener, but these two functions seem to only run once when the app is initialized (which means the elements are not created yet). Could you please further explain your idea and mechanisms behind that?

Checksum for translation keys & Hash for translations files

To keep the translation in sync with the source files (HTML, JS, as mentioned in my previous post), we need to add checksums for translation keys. But I'm not sure about the following questions:

• How should we generate these checksums?
  • Apply the MD5 digest on the combination of translation key + its value in source files?
• When should we create or update them?
  • In pre-build process?
• How and where should we store these checksums?
• Where can we use these checksums?
  • To check at runtime (in translate func) whether the string in the current language has not been updated, or to directly replace the translation values of the corresponding language with defaults during building?

Also, you mentioned adding hash for translations and putting them in cache before. Is it still necessary once we have checksums for translation keys?

Lokalise

I've already sent a message to Lokalise support to apply for an open-source plan. After the plan is approved, I will try to create a project, set up the workflow as mentioned in the to-do list and if needed, transfer the ownership of the project or the organization account to you.

I'm looking forward to your reply and further guidance on these tasks. If you have any other suggestions or comments, please feel free to let me know. Thank you!

[zyf722]

Thanks for the explanation and I'll try looking into these two opitons.

The list will be updated once relavant changes are finished and commited :)

[hatemhosny]

I'm actually more in favour of the custom events approach.
It can allow us to apply the translation anywhere anytime in a simple way, without having to change any architecture or keep passing references around while watching for tree-shaking.
This can also be used in modal.show().

You may even add a helper function (similar to the translate function) that takes a reference to the DOM element (defaulting to document.body) and will then create and dispatch the custom event. But in this case there is no need to pass a reference of the i18n object everywhere.

[zyf722]

Here is my event-driven translation solution.

I add a custom event type livecodes-i18n and add a listener to document.body to handle the translation, alongside with a dispatchTranslationEvent helper function to dispatch the event when needed.

Whenever a new element is added to the DOM, we call dispatchTranslationEvent to trigger the translation process. By event bubbling, the event will be caught by the listener on document.body and the translation will be done.

How do you think about this solution? Any suggestions or improvements?

[hatemhosny]

This is nice. Thank you.
I have added some minor comments:
zyf722@a6c60cf#r141397868
zyf722@a6c60cf#r141397761
zyf722@a6c60cf#r141397938

[zyf722]

Thank you for your suggestions. I have fixed these issues in the latest commit.

[zyf722]

Got a response from Lokalise support team. There are a couple of questions that need to be clarified before they can approve the open-source plan:

• Is there any paid versions of LiveCodes?
• Is any part of LiveCodes not open source?
• Are you funded?
• Are you generating revenue?
• Is it acceptable to put the URL and logo of Lokalise on Github page and landing page of LiveCodes?

They also points out that it will be appreciated if we could spread the word about Lokalise on X and other media.

---

Original message:

[hatemhosny]

Is there any paid versions of LiveCodes?
Is any part of LiveCodes not open source?

No, LiveCodes is fully open-source under MIT license.
However, please note that MIT license allows anybody to use it commercially.

Are you funded?
Are you generating revenue?

LiveCodes has a sponsorship program to support its development: https://livecodes.io/docs/sponsor

Is it acceptable to put the URL and logo of Lokalise on Github page and landing page of LiveCodes?

I'm happy to add a logo with a link to Lokalise in the credits section in GitHub readme and the docs website.

[zyf722]

The Lokalise support has granted the open source plan for us, and I will start integration related work soon. I suggest you register a Lokalise account (maybe using the organization e-mail) so that I can add you to the project and set you up as the team owner.

I've also asked the support team about the renewal and whether the plan will expire after a certain period of time:

The Billing page indicates that the current plan expires on Dec 31, 2025. Does this mean that the plan will be automatically renewed after that date, as long as we continue to meet the open-source requirements and remain active on Lokalise?

Hey Max, after a while we will perform a check to make sure you are still complying with the requirements.

[hatemhosny]

That's great. Thank you.
I have signed up at lokalise with the account: hatem (at) livecodes .io

[zyf722]

Added 🆗

[zyf722]

Hmm. Found something tricky to deal with.

Some values of attributes in those dynamically-added elements are only determined at runtime (like here), which means that there is no simple way to extract their default value through a static analysis. This makes it hard to integrate with current i18n-export.js script and force us to manually maintain the translation source hard-coded in code files unless we find a better solution.

Moreover, our current i18n mechanism is element-level (using translate) instead of string-level, which means that we can't directly translate the strings that shall be passed to external components (like the autocomplete in deploy screen or the snackbar notification system). This is another issue that needs to be addressed. Maybe using i18n instance through dependency injection is the only solution here?

---

PS. The work on Lokalise is progressing smoothly. I turned abstract tags in .lokalise.json into the form of <tag-xxx></tag-xxx> in order to make group highlighting work, and I also requested the support team to grant us the access to the segmentation feature to make the translation process clearer.

[hatemhosny]

Some values of attributes in those dynamically-added elements are only determined at runtime (like here), which means that there is no simple way to extract their default value through a static analysis. This makes it hard to integrate with current i18n-export.js script and force us to manually maintain the translation source hard-coded in code files unless we find a better solution.

Moreover, our current i18n mechanism is element-level (using translate) instead of string-level, which means that we can't directly translate the strings that shall be passed to external components (like the autocomplete in deploy screen or the snackbar notification system). This is another issue that needs to be addressed. Maybe using i18n instance through dependency injection is the only solution here?

My suggestion for both of these problems is to add a translateString helper function. This takes the full string as a key.

This can then be added to this object which is globally available in the window object:

livecodes/src/livecodes/core.ts

Line 4514 in b66c2f8

window.deps = { showMode };

const initApp = async (config: Partial<Config>, baseUrl: string) => {
  window.deps = { 
    showMode,
    translateString, // pass here the helper function (only in the app build)
  };
  // rest of code
};

const initEmbed = async (config: Partial<Config>, baseUrl: string) => {
  window.deps = { 
    showMode,
    translateString: (str: string) => str, // use a mock in other builds
  };
  // rest of code
};

// also in initLite and initHeadless

this allows using it elsewhere without having to pass reference to the i18n object, like this:

logOutLink.dataset.hint = window.deps.translateString('Logged in as ') + displayName;

PS. The work on Lokalise is progressing smoothly. I turned abstract tags in .lokalise.json into the form of in order to make group highlighting work, and I also requested the support team to grant us the access to the segmentation feature to make the translation process clearer.

That's very nice indeed.

[zyf722]

Thanks for pointing out this solution and this is a great way to handle the translation.

In my opinon, instead of using concatenation:

window.deps.translateString('Logged in as ') + displayName;

I think it could be better to use both key and value while utilizing interpolation:

window.deps.translateString('login.loggedIn.hint', 'Logged in as {{name}}', {name: displayName})

With translateString's definition basically like this:

export const translateString = (i18n: typeof import('./i18n').default, key: string, value: string, interpolation?: any) => {
  // ...
  return i18n.t(key, value, interpolation) as string;
}

Using a fixed key can prevent unexpected access to a string with same value but different context. Also, it's easier to maintain and update the translation file using the i18n-export.js script.

Besides, it seems that it is still necessary to pass i18n instance to the function so that it can call i18n.t. I'd like to wrap it up in core.ts like this:

window.deps = { 
    showMode,
    translateString: (key: string, value: string, interpolation?: any) => translateString(i18n, key, value, interpolation)
};

[hatemhosny]

I agree.
We can use something like this as the mock version:

const translateString = (
  _key: string,
  value: string,
  interpolation: { [s: string]: string },
) => {
  for (const [k, v] of Object.entries(interpolation)) {
    value = value.replaceAll(`{{${k}}}`, v);
  }
  return value;
};

[zyf722]

New progress in these days. Check here to see all changes.

String-level translation

Added the string-level translation and enhance its intellisense and static checking with type-safety (Reused some code from i18next.d.ts which was done before, so acutally that part is not totally useless 😏 ).

Now we can use window.deps.translateString like this:

window.deps.translateString('namespace:file.key1.subkey1', 'default <strong>value</strong>, {{interpol}}', {
  isHTML: true,
  interpol: "abc"
})

The function is completely type-safe:

• The first argument is key, which is validated with valid keys in i18n/locales/en/.
• The second argument is value. With key provided, the type of value will be narrowed to the value in the translation file to ensure all occurrences of the key have the same value. If this is a HTML string (with isHTML attribute set to true), the value will automatically be abstractified when exporting translation.
• The third argument is interpolation, which could be omitted when the value doesn't contain any interpolation. Otherwise, it should be an object whose attributes are inferred from the value. Moreover, an additional isHTML boolean attribute is added to indicate whether the value contains HTML tags and should be abstractified when exporting translation.

To provide better readability and maintainability, only string-level translation will be used in .ts files when it comes to dynamic content. For static content, element-level translation is still the best choice.

Marked all strings in src/livecodes/UI/*.ts files

Done with almost all known strings of files in src/livecodes/UI and core.ts.

There might still be some strings that are not marked yet, so I'd appreciate it if you could help me review the files and point out anything missing.

Support for .ts files in i18n-export.js

Now the i18n-export.js script can handle .ts files as well. It will use Babel to extract all strings marked with window.deps.translateString and export them to .lokalise.json and .ts files in i18n/locales/tmp.

To add new keys, just mark strings with window.deps.translateString in source files (don't worry if type check complains about missing keys, as these new keys haven't been exported yet), run npm run i18n-export and copy generated .ts files to i18n/locales/en to update the translation files and everything will be fine.

[zyf722]

PS. Don't know if keys (here to take a glance) are elegant and neat enough. Please feel free to suggest any improvements.

[zyf722]

@hatemhosny Could you please take some time to review this? Don't mean to bother, but it coule be better if you could give me some feedback so that I can continue to work on HTML files. Thanks in advance!

[hatemhosny]

hi @zyf722
Sorry for the delay. I have been busy the last few days, and was able to finish a difficult and important update.
I will have a good look and get back to you, hopefully by tomorrow.

[hatemhosny]

wow!
That was a lot of work. I really appreciate the effort and thought you put in this project.
Thank you.

I like the added type-safety and extracting data from ts files (using babel!). Impressive.

I reviewed the changes and have some comments:

• When I start the app I get an error: Cannot read properties of undefined (reading 'translateString')
  This is related to importing this variable which tries to run window.deps.translateString before initApp runs.
  The solution for this would be to change the noUserTemplates variable to a function and then call it here and there.

• When the welcome screen loads with a previous unsaved project, I get this error: TypeError: Cannot set properties of null (setting 'textContent')
  This is related to the translation of welcome.recover.lastModified, does not include the child element: <span id="unsaved-project-last-modified"></span>

PS. Don't know if keys (here to take a glance) are elegant and neat enough. Please feel free to suggest any improvements.

I have no problem with the namings of the keys. Actually, I find them descriptive.
However, I'm a bit concerned about using the file name in the key. Although this makes it easier to find and group keys, but this means that refactoring app code will result in the need to make significant changes in translations. I'm not sure what is the best approach here. I'm just thinking with you.

The big concern I'm having now is that I'm starting to feel overwhelmed with this feature. I'm not 100% sure I know what is going on where.
I know you are making a really impressive effort. I'm just worried about the burden on maintaining this, adding translation for new features, changing old text (and translations), the process of using the cloud service and getting back the translations to the code base.
I'm not sure what would be the solution for this. These are some suggestions:

• Can we simplify the workflow, even if we lose some power?
• Adding clear documentations for different workflows
• Add some code comments describing why and when each script should run.

I want to know your thoughts.
Again, thanks a lot. Much appreciated.