[Python] xbmc.getLanguage(xbmc.ISO_639_1) returns empty string for English(US), English(AU), French(CA) etc

[scott967]

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

The python function from xbmc, xbmc.getLanguage(xbmc.ISO_639_1) returns an empty string when the Kodi GUI language is set to one of the "compound" languages, for example English(US), English(AU), French(CA). The nominal language version, English or French return "en" or "fr" as expected. Also, xbmc.ISO_639_2 and xbmc.ENGLISH_NAME return the expected results. Went back and it has been like this since at least matrix.

Expected Behavior

Here is a clear and concise description of what was expected to happen:

GUI language such as English (US) should return the proper 2-char ISO code, "en".

Actual Behavior

function returns an empty string

Possible Fix

Haven't had time to run Kodi in a debugger to examine more fully.

To Reproduce

Steps to reproduce the behavior:

1. Install and select a GUI language with region variant, such as English(US)
2. Use python xbmc.getLanguage(xbmc.ISO_639_1) in an addon and log results or display in GUI notification

Debuglog

The debuglog can be found here:
https://paste.kodi.tv/usejafariq.kodi

In this log I installed a simple program script to exercise the function. First with default resource.language.en_gb in use, then installing and switching to resource.language.en_us and repeating.

import xbmc
import xbmcgui

langs = {'Kodilanguage639_1': xbmc.getLanguage(xbmc.ISO_639_1),
'Kodilanguage639_2': xbmc.getLanguage(xbmc.ISO_639_2),
'Kodilanguage_eng': xbmc.getLanguage(xbmc.ENGLISH_NAME)}
notestring = ''
for key in langs:
    if langs.get(key) == '':
        langs[key] = 'Empty str returned'
    xbmc.log(f'script.testlang: the current language is {key} {langs.get(key, "NONE returned")}')
    notestring = f'{notestring} {key} {langs.get(key, "NONE returned")}'
xbmcgui.Dialog().notification('Kodi getLanguage Test', notestring, time=10000)`

Screenshots

Here are some links or screenshots to help explain the problem:

Additional context or screenshots (if appropriate)

Here is some additional context or explanation that might help:

Discovered looking at user forum issue that script.tv.show.next.aired 8.0.4 doesn't work. Function is used in dependency script.module.thetvdb and when run was throwing exception.

Your Environment

Used Operating system:

• Android

• iOS

• tvOS

• Linux

• macOS

• Windows

• Windows UWP

• Operating system version/name: Win 10 x64 22H2

• Kodi version: Windows x64 desktop 22 nightly Git:20240318-40c09d0c93

note: Once the issue is made we require you to update it with new information or Kodi versions should that be required.
Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

[scott967]

OK, the problem appears to be in utils/LanguageCodeExpander.cpp

xbmc/xbmc/utils/LangCodeExpander.cpp

Line 294 in 497beac

bool CLangCodeExpander::ConvertToISO6391(const std::string& lang, std::string& code)

The Englishname is used in the call to

bool CLangCodeExpander::ReverseLookup(const std::string& desc, std::string& code)

ReferseLookup tries to match the Englishname in the user assigned codes, g_iso639_1 table and g_iso639_2 tables
which fails for "English (New Zealand)" so the final attempt is in

xbmc/xbmc/utils/LangCodeExpander.cpp

Lines 395 to 397 in 497beac

	code = g_langInfo.ConvertEnglishNameToAddonLocale(descTmp);
	if (!code.empty())
	return true;

which for "English {New Zealand)" returns "en-nz" in str tmp.

ConvertToISO6391 continues:

xbmc/xbmc/utils/LangCodeExpander.cpp

Lines 338 to 353 in 497beac

	{
	if (tmp.length() == 2)
	{
	code = tmp;
	return true;
	}
	if (tmp.length() == 3)
	{
	// there's only an iso639-2 code that is identical to the language name, e.g. Yao
	if (StringUtils::EqualsNoCase(tmp, lang))
	return false;
	return ConvertToISO6391(tmp, code);
	}
	}

which expects the returned code to be 2 chars (except for a single case of 3 chars) so "en-nz" fails.

Not sure how best to fix. Maybe in ConvertToISO6391 strip the hyphen region off the returned results of ReverseLookup.

I guess CLocale can parse the string and return the ISO 639-1 portion of it.

[scott967]

After thinking about this, I think changing ModuleXbmc.cpp is the place to do this, since it's closest to the caller and not likely to have side effects. Then in getLocalizedString for the
case CLangCodeExpander::ISO_639_1
handle an english name by splitting at "(", or possibly " (" not sure if there is a standard we use for the english name?

[scott967]

I did a one-line fix for this issue, but during testing I set the "region" parameter to "True" in the python test script, and what I get back for ISO639_2 and ENGLISH_NAME was a mess for the test case of UI language set to English (New Zealand).

For example, for ISO639_2 the code is calling
CLangCodeExpander::ConvertToISO6392B
with a Windows locale (on my windows system) of "nzl" which does appear to be the expected windows locale (from ISO 3166-1 alpha-3) , but what that function is returning is "nzl" instead of "eng". Unfortunately so far I can't figure out how that function was designed. Suspect this might be Windows unique, but unsure. I could probably work around it by changing the upstream so it converts 639-1 code of "en" to "eng", but that doesn't fix what seems broken here.

As it is the Python is returning "en-nz-nzl" when what is expected is "eng-nzl".

[scott967]

While testing fixes, discovered related issue when switching language resource addons in the UI see #25034

[scott967]

Also would help if a requirements engineer (anyone with interest in setting/validating the requirement) could specify what is supposed to be returned for the six possible cases in xbmc.getLanguage(). In particular, for case xbmc.ISO_639_2 with region=True and xbmc.ENGLISH_NAME with region=True what exactly should be returned?