metaframework agnostic i18n library

[samuelstroschein]

Problem

Building (sophisticated) i18n for metaframeworks is a month/year-long effort.

For example, SvelteKit's most upvoted issues are about i18n and have been open for years sveltejs/kit#553, sveltejs/kit#1274. I18n in the context of metaframeworks is especially problematic because routing, SSR, SEO, "code splitting", and more needs to be considered.

• no metaframework has a "one click/simple" i18n solution, sometimes even lacking i18n support entirely
• every metaframework builds its own solution, leading to duplicate efforts

Proposal

Provide a framework-agnostic i18n library for metaframeworks with the possibility for deep integration via adapters.

We have multiple components in @inlang/core to provide an i18n library already. We can invest resources to relieve metaframework authors from months/years of work to build i18n features for their frameworks. By providing such a library, we simultaneously come closer to our goal to make i18n easy.

• relieving metaframework authors from months/years of work
• make every framework better regarding i18n
• the rest of inlang's dev tools like the ide extension work out of the box, leading to a DX beyond of what a i18n library could provide

[samuelstroschein]

Features that immediately come to mind in the context of metaframeworks:

• i18n routing (/en/site, /de/site)
• localized routing (/en/buy/chair/, /de/kaufen/stuhl)
• resource splitting per route (minimize KB shipped)
• compiler-based to avoid a heavy runtime (shipping 1KB or less should be possible)
• typesafety (due to compiler)
• localized head tags (for SEO)
• localized sitemaps (for SEO)

[adiguba]

Hello,

I see also these features :

• Domain-based localized routing (https://www.my-domain.co.uk/buy/chair/, https://www.my-domain.de/kaufen/stuhl)

• A way to retrieve routes in other languages, in order to :

  • Generate alternate links (ex: <link rel="alternate" hreflang="de" href="https://www.my-domain.de/kaufen/stuhl">)
  • Displaying a language menu, without losing the current page.
  • Manage redirections in case of incorrect link (ex: /de/buy/chair/ is a 404 error, but we should be able to redirect to /de/kaufen/stuhl)

• Restrict some routes to locale.
  I would like a page to be available in certain locales, but not in another.
  Ex: having a /en/local-shops, with no equivalence on /de/

• I'm not sure this fits within the scope of this metaframeworks, but it's could be nice to have distinct template based on locale.
  Ex : an /en/about-us page that would be drastically different from /de/uber-uns

[stalkerg]

Just in case, we shouldn't forget to get the language names from query like ?lang=ja

[ivanhofer]

And also locale detection without a marker in the URL needs to be considered:

• auto detection based on
  • ip address
  • browser language
• storing the locale in a cookie
• storing the locale in the user object (in a DB)

If it is a SPA, then the locale is probably stored:

• in localStorage or sessionStorage
• using indexedDB
• the HTML lang attribute
• reading navigator.language
• as a part of the hash in the URL

[iolyd]

For param-based locale detection, could routes without any [locale] param fallback to the defined default locale without adding the locale param? Say we have a project with two locales, fr and en, but we want to valorize fr as the original language since we expect most our users will want french content. I would enjoy having the possibility of letting users access the fr content both through /fr/... and /... routes since it is a somewhat common practice to have their browsers settings configured for en-CA while expecting french content (french browser UIs often use weird/unfamiliar nomenclatures).

[ivanhofer]

Usually the / route should be the entry point to your website. It should detect the user's language and make a redirect to e.g. /fr. If you really want, you will also be able to show the reference language on the / route. But then you need to implement the logic yourself how to to show the correct language version for your users. If 95% of your users speak french, this could be fine, but still I would aim for 100% if I could. Inlang will provide sane defaults, but you should be able to configure everything if you relly want.

[brillout]

That's great. As the author of vite-plugin-ssr, I'm very much interesed in this. Looking forward to playing around with a first draft of the library.

[shripadk]

Really like the idea of it being compiler based. Would this mean it will scan the codebase, extract strings and autogenerate ids? Something along the lines of what formatjs is doing? Also, please keep edge in mind. Most i18n libraries I know use fs module from Node to fetch locales. fs isn't available in edge runtimes. Also, I am not a fan of "fetching" locale files after browser loads (which some libraries implement). Not good for SEO. I guess bundling locales is one option (but can easily shoot up in size and cross the worker size limitations imposed by edge runtimes - like Cloudflare Workers has a soft 1MB limit and hard 5MB limit AFAIK). Maybe provide adaptors to deploy to R2/S3 buckets?

Just thinking out loud.

[samuelstroschein]

Would this mean it will scan the codebase, extract strings and autogenerate ids?

Yes.

Also, please keep edge in mind. Most i18n libraries I know use fs module from Node to fetch locales. fs isn't available in edge runtimes.

Noted. In general, inlang is "node agnostic" because everything we are building is running in the browser as part of the editor.

Also, I am not a fan of "fetching" locale files after browser loads [...] I guess bundling locales is one option

The compiler approach with SSR support should eliminate both issues by only bundling what a component/page requires. The translations themselves will likely be compiled to minimum JS instead of being string-based.

Maybe provide adaptors to deploy to R2/S3 buckets?

If the need arises, an adapter should make that possible.

Just thinking out loud.

Keep it coming! :)

[ul8]

@samuelstroschein Sounds promising! The Sveltekit team seems to perpetually remain in analysis/paralysis with this, and with good reason. It's just way, way more complex than anyone would first suspect. Especially so if you want to support use cases of larger apps involving many languages, professional translation workflows etc.

I think it actually makes a lot of sense for a dedicated team to work on this in a framework-agnostic, mostly reusable way because i18n is such a deep rabbit hole. Most i18n frameworks I've seen are just the same mistakes being made again and again due to the author's lack of experience with how translation actually works professionally.

I do think however it would be problematic if this adapter and framework is tightly coupled in a not-so agnostic way with how the translation content is stored and translated.

Let's say sveltekit is integrated with inlang. Could we then extract and import again all translations to other translation management systems, export into industry standard formats like XLIFF etc.? Meaning do we have to buy in to the "built on git" idea? I personally like it but for this to become truly universally useful and accepted it would need to work with everyone's own choice of translation workflow.

[samuelstroschein]

Meaning do we have to buy in to the "built on git" idea?

Nope.

I do think however it would be problematic if this adapter and framework is tightly coupled in a not-so agnostic way with how the translation content is stored and translated.

Agree.

1. every component of inlang (editor, ide extension, i18n library) is designed to work independently of other
2. inlang is storage agnostic. You could write a config file that fetches translations from a database:

export async function defineConfig(env){ 
  return {
    readResources: fetchFromDatabase(),
    writeResources: updateDatabase()
  }
}

Let's say sveltekit is integrated with inlang. Could we then extract and import again all translations to other translation management systems, export into industry standard formats like XLIFF etc.?

Yes.

How translations are stored is not a concern of inlang and must be provided through the config/plugins. Even better, because inlang is file agnostic and defines a common AST to parse and serialize from, you could store your translations as JSON files and convert them on the fly in a CI/CD process to XLIFF and sync them with your translation management system.

[vnphanquang]

Your rationale and proposal in on point. i18n has been an issue for too long, proven to be a complex issue bigger than any framework; so having a dedicated team / effort makes a lot of sense to me. Thank you for taking the initiative.

I believe there is already a lot of effort going on for this, maybe not necessarily with the same philosophy as yours but certainly with the same intention of unifying the toolsets for i18n. Reminds me of @ivanhofer with his pipeli18ne project (sorry for tagging, thought you might have some thoughts here).

[ivanhofer]

No worries, I'm actually helping to shape this concept.
When I shared my ideas with pipeli18ne @samuelstroschein and I had a discussion and saw that we are working on similar problems and we plan to combine our efforts.

[vnphanquang]

That is great to hear. I did not notice you're already a collaborator.

[cliffordfajardo]

Provide a framework-agnostic i18n library for metaframeworks with the possibility for deep integration via adapters.

Love this idea to try and solve i18n via adapters, kind of like how meta frameworks are solving deployment target via adapters;

Regarding i18n

@samuelstroschein, have you seen the message format Github Repo which implements the ICU Message format which is a standard used in other programming languages for i18n & is a project of the Open JS Foundation. We should see if it makes sense to build on standards so we can increase compatibility of tooling across the ecosystem, if possible.

Shoutout to @ivanhofer who's exploring i18n as well with his pipeli18ne which is in RFC/exploratory phases

Random aside

This metaframework agnostic effort reminds me of recent initiative(s) which are also trying to solve problems in framework agnostic ways & helping metaframeworks & authors consolidate on solution(s) together. Plugging these projects, below in case people want to reach out to other framework creators

• authjs - framework agnostic authentication for web (formerly nextauth by vercel)
• bling - Framework agnostic transpilation utilities for client/server RPCs, env isolation, islands, module splitting, and more ---- from the creators of SolidStart and react-query

[ivanhofer]

We should see if it makes sense to build on standards so we can increase compatibility of tooling across the ecosystem, if possible.

The concept behind inlang takes into account that any message format could be used. It should not enforce a certain message format, but it could recommend one. If somene would write a parser & serializer that converts a given string i a certain format to the AST inlang uses, anything is possible. By building a modular code structure like in the concept of pipeli18ne any part of that process can be replaced and customized depending on a projects specific needs.

[CanRau]

The AST sounds like a fantastic solution, love this a lot about the unified, remark, rehype ecosystem 👏

[cliffordfajardo]

ahh AST solution 🤔

[victormihalache]

For metaframeworks that use a file-based router, how would routes be localised?

Having to keep translations spread across multiple files (say that each route gets its own file) is tedious, also having to manually register a route in a single file is tedious.

[samuelstroschein]

A deeper integration with the routing system of the framework would be required and is what we are aiming for.

For example, we could automatically prefix routes with the locale or map routes to a different translated route.

[ivanhofer]

Hey folks, we have some great news to share!
We will soon start with our SDK. We will implement the first version for SvelteKit!

I have written a few lines on how we could implement it. See here.
We are looking for feedback. It would be great if you could take some time to review the RFC and share your ideas.

[stalkerg]

@ivanhofer is it allowed by Rich? I don't want to have a situation where he will deploy a new solution next month and all your work will be gone.

[vnphanquang]

uh is there a conversation between inlang team & Rich (Rich Harris from svelte team?) that I don't know of? Sounds unrelated imo no?

[ivanhofer]

@ivanhofer is it allowed by Rich? I don't want to have a situation where he will deploy a new solution next month and all your work will be gone.

As far as I know they won't build their own solution. But I can't speak for the SvelteKit team. You will need to ask them yourself if you want a confirmation.

uh is there a conversation between inlang team & Rich (Rich Harris from svelte team?) that I don't know of? Sounds unrelated imo no?

We are talking with the SvelteKit team because we hit a few limitations & need to find creative workarounds. One of the members sayed he would like to help us (and other projects that build on top of SvelteKit) to find a good solution.

And yes, I agree, it is unrelated. Even if something is official, doesn't mean it is the best solution or it fits your needs. Inlang is not just building a solution for SvelteKit. Other metaframeworks will follow in the future.

[vnphanquang]

Very much agreed. I'm extremely happy that conversation is taken place between two teams and that inlang team is supporting sveltekit first 👍🏼. I have hight hopes for what's coming, but please don't feel pressured to put out a release day 🙏🏼

[ivanhofer]

well, we have put the pressure on ourselfs ^^
https://www.sveltesummit.com/ 👀

[omar2205]

Hey @ivanhofer, is there a good guide to use typesafe-i18n with inlang? I know the plugin exists, but can't seem to get it working.

[samuelstroschein]

@omar2205 Nope. Inlang works with plugins. You can simply use the typesafe-i18n plugin as described in https://github.com/ivanhofer/inlang-plugin-typesafe-i18n#usage

[omar2205]

I meant setting up typesafe-i18n itself. Do I follow https://github.com/ivanhofer/typesafe-i18n#get-started and then follow Inlang's getting started?

[ivanhofer]

Yes

[samuelstroschein]

@omar2205 can you pinpoint where your confusion is coming from?

We are about to update our communication. Ideally, you should have known how to set up inlang with typesafe-i18n.

[omar2205]

I've opened this issue ivanhofer/inlang-plugin-typesafe-i18n#4
My confusion is how to set up typesafe-i18n with Inlang. I've gotten a couple of errors when I start the server.

[NiklasBuchfink]

@samuelstroschein I think we now have the solution with Paraglide JS :)