Backend loading of supported languages and language fallbacks

[Shepard]

Hi! I'm having a bit of a problem with language fallbacks and whitelists. I'm not sure if I'm just using things wrong or if my use cases are not supported. Maybe you can advise.

We're using i18next with the i18next-browser-languagedetector and i18next-xhr-backend plugins. In using this combination I noticed that the backend plugin will request resource files for languages we don't actually support and when it notices that that's not successful, falls back to the next language and attempts to load that.

So I thought I would try to restrict it to languages we know we support and did that via the whitelist option. By setting it to whitelist: ['en', 'de'] I was able to get the backend plugin to not request languages anymore that are not in the whitelist.

Of course I still want a user with a language of "en-GB" to get the "en" translations we provide and a user with "de-DE" to get the "de" translations we provide. However, the whitelist option messes with the language detection, making this impossible. In particular, because the browser language detector plugin will only detect languages that are whitelisted, it will never return "en-GB" or "de-DE". Instead it will return the first configured fallback language (which we have set to "en").

So maybe I shouldn't use the whitelist for this case? How can I tell the backend plugin to only attempt loading certain languages while still having a locale fallback mechanism?
I looked at the load option but I don't think it works for us either because I can only restrict to loading pure language codes only. However in the future the list of locales supported by us might also have such things as "zh-TW", "pt-BR", or even "es-419-x-informal".

Another use case that the browser language detection doesn't seem to support at the moment (if I don't use a whitelist at least) is considering the full list of languages set in the browser. If, e.g., I set my browser to request the following languages, in this order: "fr", "de", "en" - then the browser language detector plugin will return the first entry in this list that is whitelisted only (which, if I haven't configured a whitelist, is just the first entry). In this case "fr". The backend plugin will request this, fail because we don't have translation resources for "fr", and then probably just consider fallbackLng instead ("en"). The language detection plugin doesn't get asked anymore even though it could provide "de" as the next best match that's actually supported by us, instead of falling back to "en" already.

I know that I can define more complex cases in fallbackLng but that would mean listing every possible locale that we don't support, which is not doable.

This is, roughly, our setup at the moment:

i18next
	.use(i18nextBrowserLanguageDetector)
	.use(i18nextXHRBackend)
	.init({
		fallbackLng: 'en',
		whitelist: ['en', 'de'],
		ns: ['main'],
		defaultNS: 'main',
		detection: {
			order: ['navigator'],
			caches: null
		},
		backend: {
			loadPath: '/translations/{{lng}}/{{ns}}.json'
		}
	})
	.then(function() {
		console.log(`I18Next initialized. Current language is '${i18next.language}'.`);
		// Do other stuff depending on i18n being initialized...
	})
	.catch(fatalErrorHandler);

// Adding English main translations manually so that we can bundle them with the JavaScript. Doing this here will
// prevent them from getting loaded via the backend which i18next will otherwise do even if the language isn't set to
// English because it preloads the set language *and* the fallback language (which we set to English).
i18next.addResourceBundle('en', 'main', englishMainTranslations);

[jamuhl]

yes, the language detector can't detect multiple languages...it will only pass the one back with highest prio in case of navigator...a limitation of the current system...

If you need to mix locales with and without region -> zh-TW, pt-BR, fr, de, en is to add all you support to the whitelist...eventual nonExplicitWhitelist https://www.i18next.com/overview/configuration-options#languages-namespaces-resources

But:

I noticed that the backend plugin will request resource files for languages we don't actually support

The question is - what's the problem with that? It's not a runtime error it's a network error that the file was not found...nothing harmful with that

[Shepard]

My problem is with the unnecessary request on every page load that slows down the initialisation of the page as well.

And the whitelist, as I described, doesn't help because it messes with the language detection.
If we have language resource files for "en" but not "en-GB" then either I can add "en-GB" to the whitelist, which means I still get the unnecessary request, so I don't actually gain anything (and I would have to do that for every other locale-variant of our supported languages that you can imagine - which is unmanagable). Or I can not add "en-GB" to the whitelist but then there's also no automatic fallback from that to "en" anymore because it will just get rejected.
Or I don't use a whitelist at all and still have the unnecessary requests.

I was just hoping for a way to influence the backend plugin so that it won't make requests for languages which we know we don't have files for, without breaking the language detection.

[jamuhl]

How should the backend do so? there is no difference from en-GB to zh-TW both are the same language-Region -> so you would have to define a whitelist (but you don't want to maintain one as it's to much work)

Not sure if it helps but you might add an option to not check whitelist here: https://github.com/i18next/i18next-browser-languageDetector/blob/master/src/index.js#L71

[Shepard]

No, I'm happy to maintain a whitelist - of the languages where we do have translation files! That's a list that the backend could check before making a request to our server.

However I don't want to lose the ability for a user to have their browser set to "en-GB" and us giving them the translations for "en" in that case. Or they have it set to "de-CH" and even though we don't support Swiss German explicitly we can still give them the translations for "de" which we do have.

What is unmaintainable is to have a list of all possible locales that a user could have set their browser to. Maybe the user is in Jamaica and speaks English, so they have their browser set to "en-JM". Me as a developer didn't anticipate this locale. But automatisms can decide that this should fall back to "en". And i18next does seem to have these autmatisms, but by using a whitelist the locale gets rejected by the browser language detector before the code can even get to the automatism.

[Shepard]

I hope I'm making enough sense here. :-)

I'm currently thinking that I could get around most of the problems by putting a proxy object in front of the backend plugin that will have the list of supported languages and just not hand the call on to the backend if a requested language is not supported.

However it seems to me that this is not an unusual use case and support for this in the library would be nice.

[jamuhl]

like said...making that optional should do the trick: Not sure if it helps but you might add an option to not check whitelist here: https://github.com/i18next/i18next-browser-languageDetector/blob/master/src/index.js#L71

en-US will still be detected even if not in the whitelist -> later the loading will check the whitelist and fallback to en

[Shepard]

I think it might do the trick too. Do you suggest I fork the plugin?

[jamuhl]

No doing a PR making that check optional optional ;) -> option checkWhitelist: true so you can set that to false and omit that whitelist check

[Shepard]

Sorry, I was away for a bit, but I submitted a PR now: i18next/i18next-browser-languageDetector#190