setlocale(): Specified locale name is too long

[ewallah]

Since a week I see these errors in GitHub Actions when I use language related functions in behat tests :

Exception - Warning: setlocale(): Specified locale name is too long in [dirroot]/admin/tool/langimport/classes/locale.php on line 82
        Error code: generalexceptionmessage
        	* line 157 of /lib/behat/lib.php: Exception thrown
        	* line ? of unknownfile: call to behat_error_handler()
        	* line 82 of /admin/tool/langimport/classes/locale.php: call to setlocale()
        	* line 68 of /admin/tool/langimport/classes/locale.php: call to tool_langimport\locale->set_locale()
        	* line 139 of /admin/tool/langimport/index.php: call to tool_langimport\locale->check_locale_availability()

I suspect that the locale used in behat test is not en_AU.UTF-8 but the Ubuntu default.

A possible workaround for avoiding these errors is forcing en_AU.UTF-8 in your behat tests:

    name: Behat features
    if: ${{ always() }}
    env:
         LC_ALL: en_AU.UTF-8
    run: moodle-plugin-ci --ansi behat --profile chrome

With these additional lines, my behat tests pass successfully.

[abias]

Thanks @ewallah for raising this issue.

Some days ago, I stumbled upon the same issue in my Behat tests which broke after adopting the latest proposed changes of moodle-plugin-ci to my Github actions configuration.

I asked about this issue in Moodle dev chat, trying to figure out how this can be solved and if this error might be related to MDL-74512, and @stronk7 and @andrewnicols answered helpfully, but I couldn't resolve the issue after all.

Now I noticed that you have already reported this issue here and have proposed a workaround. This helped me to get my tests up and running again. Thanks for that!

In my case, the error was thrown in the Behat step And I navigate to "Language > Language packs" in site administration. Before that step, I did not do anything with the language pack in the scenario (see https://github.com/moodle-an-hochschulen/moodle-local_navbarplus/blob/master/tests/behat/local_navbarplus.feature#L38 for details).

I also noticed that the error is only thrown if I run Github actions on ubuntu-22.04 as it's recommended now. If I run the same Github actions configuration file on ubuntu-18.04, the tests pass without any problems.

I assume that the default locale might have changed between the Ubuntu versions, but I am unsure how and where to fix this properly and sustainably for moodle-plugin-ci now.

Cheers,
Alex

[gjb2048]

I have this sudo locale-gen en_AU.UTF-8 in the plugin initialise step - https://github.com/gjb2048/moodle-theme_foundation/blob/master/.github/workflows/ci.yml#L70 - as without it, the PHPUnit tests fail to run. Might help here?

[stronk7]

But, that has been recommended since the very beginning of the GHA integration, isn't it?

https://github.com/moodlehq/moodle-plugin-ci/blob/master/gha.dist.yml#L57

I'm not sure we need to add anything else. Or are you saying that, no matter that line, tests continue failing?

Ciao :-)

[stronk7]

Another issue is that the 4.2 lang packs... don't exist yet... so if your tests include any test trying to import lang-packs in master (4.2dev), they will fail. Not sure if that's your problem or no.

References (we have them disabled in CIs till the new lang packs are available):

• downloads: https://download.moodle.org/langpack/4.2/
• own CI runner: Re-enable lang upgrades after the 4.1 on-sync period moodle-ci-runner#89 (comment)

Ciao :-)

[abias]

I'm not sure we need to add anything else. Or are you saying that, no matter that line, tests continue failing?

Thanks, @gjb2048 . But I have this line in my config as it's recommended (see https://github.com/moodle-an-hochschulen/moodle-local_navbarplus/blob/master/.github/workflows/moodle-plugin-ci.yml#L56) and tests are failing on ubuntu-22.04

Another issue is that the 4.2 lang packs... don't exist yet... so if your tests include any test trying to import lang-packs in master (4.2dev), they will fail. Not sure if that's your problem or no.

No, that's not my issue. My tests are targetting Moodle 4.0 Stable.

[stronk7]

Thanks @abias, I'll try to clone your plugin and give to it some runs here, to try to understand what's really going. I'm not sure to understand why the Ubuntu version makes any difference or why the solution proposed by @ewallah does the trick.

Nice mystery!

[gjb2048]

@abias and @stronk7 Ah ok :). Not twigged that. I run Behat tests on Collapsed Topics -> https://github.com/gjb2048/moodle-format_topcoll/blob/master/.github/workflows/ci.yml - and that works on ubuntu-22.04 or at least did on the last run. Has something changed?

[stronk7]

Hi @gjb2048,

right now I don't know much. I also reran 1-2h ago all the local_codechecker own tests to check if this problem was something global:

https://github.com/moodlehq/moodle-local_codechecker/actions/runs/3558869848

But they passed ok, so it seems that the problem is related with behat steps trying to import a language.

That's the reason I warned about the inexistent yet 4.2 language packages.

And then @abias confirmed that it happens to him not with 4.2, but with older branches.

So all we know right now is:

• Only happens with Ubuntu 22.04 run.
• Only happens with behat tests using (or navigating) to the import lang-packs tool page.

Let's see if we can trace this down...

[gjb2048]

@abias Given the stack trace and looking at the code (how the called get_list_of_translations() string_manager method seems to work), I don't think its to do with the environment but rather the actual Moodle files etc. Do you have any custom plugins with different languages in their lang folders?

[stronk7]

Aha, @abias , now I remember the previous discussion! And remember that I looked to this:

https://github.com/stronk7/moodle/blob/master/admin/tool/langimport/classes/locale.php#L62

And I only could imagine some connectivity glitch being the problem, somehow corrupting the $locale returned.

In the other side, I've been able to reproduce it now (@ GHA):

https://github.com/stronk7/moodle-local_navbarplus/actions/runs/3676900419

So now I'm going to try to:

1. Reproduce it locally, and if I cannot...
2. Make moodle-plugin-ci to use some customised moodle.git branch of me, with some extra debugging added to show which language is the one causing the exception and which contents are coming to that $locale variable.

Wish me luck! Ciao :-)

PS: I still cannot imagine how setting LC_ALL helps, maybe it does by setting back the original LC_ALL that with old Ubuntus was set and not with 22.04... in any case, that's after my 1-2 above. I want to check first if the problem happens because of some exact language and wrong locale within its configuration.

[stronk7]

Ok,

so the problem doesn't happen when trying to set the locale from the lang file (line #65). It happens 3 lines below, when trying to restitute the original ($currentlocale) locale (line #68).

So, then the line suggested by @ewallah makes sense if it happens that Ubuntu 22.04 comes with it undefined, and older versions had it defined. Going to check that now...

[stronk7]

Wow, this is a real surprise. I just modified the workflow to output both:

• $LANG env variable (as is in shell).
• setlocale(LC_ALL, 0) (to see which default locale PHP is picking).

And, or I'm missing something... or the default locale (C.UTF-8 is perfectly ok). And it's exactly the same in Ubuntu 20.04 and 22.04. Another story is Ubuntu 18.04 that seems to be buggy, but we are not using it.

Run: https://github.com/stronk7/moodle-local_navbarplus/actions/runs/3677316897

So, how is it possible that 22.04 is failing and 20.04 is passing? They use exactly the same default locale, that for sure is installed in the system. I cannot understand which the difference can be.

Note that I'm starting to imagine that we can workaround that from within moodle-plugin-ci, by setting the env variable (LANG or LC_ALL) before invoking behat, so this seems to be fixable.

The problem is that I don't understand why that fix is needed, and I don't like to apply for mysterious fixes. I'm going to debug a little bit more against custom moodle to see if there is any difference between the tests above (returning C.UTF-8) and what real Moodle gets. There is something, somewhere... that we need to discover.

Ciao :-)

[stronk7]

I've added some debugging to the tool_langimport\locale::check_locale_availability() method. And, then, just visited the Language import utility page.

And testing it locally (Mac)... I get this (only English is installed by default:

[12-Dec-2022 18:24:32 Europe/Madrid] langpack = en
[12-Dec-2022 18:24:32 Europe/Madrid] $currentlocale = en_AU.UTF-8/en_AU.UTF-8/C/C/en_AU.UTF-8/C
[12-Dec-2022 18:24:32 Europe/Madrid] $locale = en_AU.UTF-8
[12-Dec-2022 18:24:32 Europe/Madrid] set_locale to lang pack result = en_AU.UTF-8
[12-Dec-2022 18:24:32 Europe/Madrid] set_locale back to current result = en_AU.UTF-8/en_AU.UTF-8/C/C/en_AU.UTF-8/C

So, when processing the "en" lang pack, first it fetches $currentlocale (that is a log string separated by slashes), then tries to set the locale to the lang pack one (en_AU.UTF-8) and does is successfully. And, finally, it sets the locale back to the $currentlocale fetched at the beginning, and that operation is successful too.

It would be great if you can share here what do you get in apache logs with your Ubuntus or any other linux flavour, just to be able to compare stuff.

Ciao :-)

[gjb2048]

Hi @stronk7,

No idea if this helps, but I'm running a custom WAMP on Windows 10 and get:

[12-Dec-2022 17:40:56 Europe/London] langpack = en
[12-Dec-2022 17:40:56 Europe/London] $currentlocale = LC_COLLATE=English_Australia.1252;LC_CTYPE=English_Australia.1252;LC_MONETARY=C;LC_NUMERIC=C;LC_TIME=English_Australia.1252
[12-Dec-2022 17:40:56 Europe/London] $locale = English_Australia.1252
[12-Dec-2022 17:40:56 Europe/London] set_locale to lang pack result = English_Australia.1252
[12-Dec-2022 17:40:56 Europe/London] set_locale back to current result = LC_COLLATE=English_Australia.1252;LC_CTYPE=English_Australia.1252;LC_MONETARY=C;LC_NUMERIC=C;LC_TIME=English_Australia.1252

So possibly interesting for comparison? I have a Ubuntu VM machine so if I can get it working...

Cheers,

Gareth

[stronk7]

Yeah, that's correct too. The code doesn't fail for you. In the Mac I was surprised about the "/" (slashed) formatting, but it worked ok when using it to set back the original locale.

In your windows, it's better formatted, with semi-colon separated information. And also worked ok when using it to set back the original locale.

Note that if any set_locale fails we should see a 0/empty (false) in some of the logged lines.

Let's see with Ubuntu (or any other linux).

[stronk7]

Thanks to @sarjona I got it run against Ubuntu 20.04:

[Mon Dec 12 18:46:29.122150 2022] langpack = en
[Mon Dec 12 18:46:29.122227 2022] $currentlocale = LC_CTYPE=en_AU.UTF-8;LC_NUMERIC=C;LC_TIME=en_AU.UTF-8;LC_COLLATE=en_AU.UTF-8;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=en_AU.UTF-8;LC_NAME=en_AU.UTF-8;LC_ADDRESS=en_AU.UTF-8;LC_TELEPHONE=en_AU.UTF-8;LC_MEASUREMENT=en_AU.UTF-8;LC_IDENTIFICATION=en_AU.UTF-8
[Mon Dec 12 18:46:29.122282 2022] $locale = en_AU.UTF-8
[Mon Dec 12 18:46:29.122299 2022] set_locale to lang pack result = en_AU.UTF-8
[Mon Dec 12 18:46:29.122326 2022] set_locale back to current result = LC_CTYPE=en_AU.UTF-8;LC_NUMERIC=C;LC_TIME=en_AU.UTF-8;LC_COLLATE=en_AU.UTF-8;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=en_AU.UTF-8;LC_NAME=en_AU.UTF-8;LC_ADDRESS=en_AU.UTF-8;LC_TELEPHONE=en_AU.UTF-8;LC_MEASUREMENT=en_AU.UTF-8;LC_IDENTIFICATION=en_AU.UTF-8

It also worked ok, no error at all and all the setlocale() calls returned information.

[gjb2048]

Hi @stronk7,

Fired up a Ubuntu 22.04 and get:

[Mon Dec 12 18:00:26.778592 2022] [php:notice] [pid 32787] [client x.x.x.x:50351] langpack = en, referer: https://ubu2204servercoursemoodle./admin/search.php
[Mon Dec 12 18:00:26.778613 2022] [php:notice] [pid 32787] [client x.x.x.x:50351] $currentlocale = C, referer: https://ubu2204servercoursemoodle./admin/search.php
[Mon Dec 12 18:00:26.778624 2022] [php:notice] [pid 32787] [client x.x.x.x:50351] $locale = en_AU.UTF-8, referer: https://ubu2204servercoursemoodle./admin/search.php
[Mon Dec 12 18:00:26.778634 2022] [php:notice] [pid 32787] [client x.x.x.x:50351] set_locale to lang pack result = , referer: https://ubu2204servercoursemoodle./admin/search.php
[Mon Dec 12 18:00:26.778638 2022] [php:notice] [pid 32787] [client x.x.x.x:50351] set_locale back to current result = C, referer: https://ubu2204servercoursemoodle./admin/search.php

But I don't think the sever is quite right! ->

But I get:

locale -a
C
C.utf8
en_GB.utf8
POSIX

Cheers,

Gareth

[stronk7]

I think you're missing the en_AU.UTF-8 locale and it's warning you about that, in fact the set_locale to lang pack result = failed (empty/zero/false), and that causes the notification to happen. So surely you need the sudo locale-gen en_AU.UTF-8 to be run there to make that locale available.

In any case, you did not get the "Specified locale name is too long" error/exception. And it's 22.04!

[gjb2048]

@stronk7 Thanks, I found https://askubuntu.com/questions/76013/how-do-i-add-locale-to-ubuntu-server before I saw your reply! So after reboot the error goes away and:

[Mon Dec 12 18:10:15.521587 2022] [php:notice] [pid 924] [client x.x.x.x:50449] langpack = en, referer: https://ubu2204servercoursemoodle./admin/search.php
[Mon Dec 12 18:10:15.521628 2022] [php:notice] [pid 924] [client x.x.x.x:50449] $currentlocale = LC_CTYPE=en_AU.UTF-8;LC_NUMERIC=C;LC_TIME=en_AU.UTF-8;LC_COLLATE=en_AU.UTF-8;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=en_AU.UTF-8;LC_NAME=en_AU.UTF-8;LC_ADDRESS=en_AU.UTF-8;LC_TELEPHONE=en_AU.UTF-8;LC_MEASUREMENT=en_AU.UTF-8;LC_IDENTIFICATION=en_AU.UTF-8, referer: https://ubu2204servercoursemoodle./admin/search.php
[Mon Dec 12 18:10:15.521656 2022] [php:notice] [pid 924] [client x.x.x.x:50449] $locale = en_AU.UTF-8, referer: https://ubu2204servercoursemoodle./admin/search.php
[Mon Dec 12 18:10:15.521671 2022] [php:notice] [pid 924] [client x.x.x.x:50449] set_locale to lang pack result = en_AU.UTF-8, referer: https://ubu2204servercoursemoodle./admin/search.php
[Mon Dec 12 18:10:15.521682 2022] [php:notice] [pid 924] [client x.x.x.x:50449] set_locale back to current result = LC_CTYPE=en_AU.UTF-8;LC_NUMERIC=C;LC_TIME=en_AU.UTF-8;LC_COLLATE=en_AU.UTF-8;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=en_AU.UTF-8;LC_NAME=en_AU.UTF-8;LC_ADDRESS=en_AU.UTF-8;LC_TELEPHONE=en_AU.UTF-8;LC_MEASUREMENT=en_AU.UTF-8;LC_IDENTIFICATION=en_AU.UTF-8, referer: https://ubu2204servercoursemoodle./admin/search.php

and the server is:

Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-56-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Mon 12 Dec 18:10:37 UTC 2022

  System load:  0.99658203125     Processes:               127
  Usage of /:   70.0% of 9.75GB   Users logged in:         0
  Memory usage: 31%               IPv4 address for enp0s3: x.x.x.x
  Swap usage:   0%

 * Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s
   just raised the bar for easy, resilient and secure K8s cluster deployment.

   https://ubuntu.com/engage/secure-kubernetes-at-the-edge

0 updates can be applied immediately.

G

[stronk7]

Thanks @gjb2048 !

So it's working ok on your local Ubuntu 22.04 too. Ok, I'm going to do some more tests against GHA, trying to output the web server logs containing the debugging as part of the workflow.

Let's see if we find why 22.04 on GHA makes any difference!

[gjb2048]

No worries @stronk7 and just in case:

gareth@ubu2204servercourse:/var/log/apache2$ php --version
PHP 8.0.26 (cli) (built: Dec  3 2022 19:21:17) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.26, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.26, Copyright (c), by Zend Technologies
gareth@ubu2204servercourse:/var/log/apache2$ apache2 -v
Server version: Apache/2.4.52 (Ubuntu)
Server built:   2022-09-30T04:09:50

G

[stronk7]

Ok, by hacking a little bit moodle-plugin-ci and pointing to it from [my fork of moodle-local_navbarplus... I've been able to get the web server logs in output. See:

https://github.com/stronk7/moodle-local_navbarplus/actions/runs/3684508663/jobs/6234367284#step:7:277

And, certainly... the set_locale() call switching back to the original local it failing there. Although the original one looks really correct (exactly the same than the ones reported here for 20.04 and 22.04). So it continues being a good mystery.

[Tue Dec 13 10:33:17 2022] langpack = en
[Tue Dec 13 10:33:17 2022] $currentlocale = LC_CTYPE=en_AU.UTF-8;LC_NUMERIC=C.UTF-8;LC_TIME=en_AU.UTF-8;LC_COLLATE=en_AU.UTF-8;LC_MONETARY=C.UTF-8;LC_MESSAGES=C.UTF-8;LC_PAPER=en_AU.UTF-8;LC_NAME=en_AU.UTF-8;LC_ADDRESS=en_AU.UTF-8;LC_TELEPHONE=en_AU.UTF-8;LC_MEASUREMENT=en_AU.UTF-8;LC_IDENTIFICATION=en_AU.UTF-8
[Tue Dec 13 10:33:17 2022] $locale = en_AU.UTF-8
[Tue Dec 13 10:33:17 2022] set_locale to lang pack result = en_AU.UTF-8
[Tue Dec 13 10:33:17 2022] Default exception handler: Exception - Warning: setlocale(): Specified locale name is too long in [dirroot]/admin/tool/langimport/classes/locale.php on line 88

Now that I've access to the web server logs... I'm going to try with 20.04 to be able to compare results...

Let's see, this is being fun!

Ciao :-)

[stronk7]

Uhm... interesting.... it seems that locale max length is 255 (and the failing logs above are 268 if I've counted them correctly), have found various reports out there. Example:

horde/horde@13d2342#diff-43cdf63cea18bc549d232f55b13b3b14536151d45f8e771a8e1e398762456d34R141-R148

Ok, going to compare with 20.04 and then decide if we have to apply for something similar...

[stronk7]

And just for reference, because we'll need that... in Macs (that is a BSD unix), the "slashed" locales are perfectly ok. Each one corresponds to one of six LC variables. Fixed.

Source: https://man.openbsd.org/setlocale.3#RETURN_VALUES

On OpenBSD, if setlocale() was never called with a non-NULL locale argument, the string "C" is returned. Otherwise, if the category was not LC_ALL or if the locale is the same for all categories, a copy of the locale argument is returned. Otherwise, the locales for the six categories LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC, LC_TIME are concatenated in that order, with slash (‘/’) characters in between.

So one less mystery... now let's see how the multi-ubuntu tests end...

https://github.com/stronk7/moodle-local_navbarplus/actions/runs/3684814565

Back in a few hours... ciao :-)

[stronk7]

Ok, so from the run commented above:

• 22_04 Failed => the current locale is 268 chars long and it fails when we try to set it back.
• 20_04 Failed => the current locale is 268 chars long and it fails when we try to set it back.
• 18_04 Passed => the current locale is 250 chars long and setlocale() was able to set it back.

And the differences are because some of the elements have "C" in 18_04, but "C.UTF-8" in 20_04 and 24_04, leading to > 255 chars.

Windows "may" be exposed to the same problem (theoretically), if there is some long-enough locale which total length becomes > 255.

Macs will be hardy affected by this, because it always returns a fixed 6-elements list (as explained above) and no combination of 6 locales (42 chars each, they should be) can lead to > 255.

So, conclusions:

• Until it's fixed in core, by allowing us to set the locales 1 by 1. That will be done @ MDL-76666 (nice number, LOL!).
• Setting 'LC_ALL=xxxxx' (with xxxx being a correct locale) is a valid workaround, because when LC_ALL is set, then ALL the LC locales get the same value. And then, when asking for the current locale a single xxxxx is returned (instead of all individual locales).

I'm going to leave this open till the issue in the tracker is fixed. Good we were able to find the root cause, yay!

Ciao :-)

[abias]

@stronk7 - Kudos for going down the rabbit hole that far!
I appreciate your work and am looking forward to the fix in MDL-76666 :)

[stronk7]

The very first candidate branches have been put @ MDL-76666. They are passing ok under mac/win/linux and they cover the specific linux case.

Also, I've modified my fork of navbarplus to point to those candidate branches, and it has passed ok, so far:

https://github.com/stronk7/moodle-local_navbarplus/actions/runs/3693007313

So it's looking good, aiming to send it to PR soon (just a few more tests to run and done!)

So I'm going to close this now, thanks all for your support, it made easier to find the root cause (plus, we all have learnt a lot more about those "$·/&%$&% locales) 😆

Ciao :-)