Tests are dependent on environment's locale

[galaxy4public]

On a clean clone of the repository one could affect the outcome of integrated tests by changing their system locale environment, e.g.:

 [galaxy@archlinux pelican]$ LANG=C invoke tests | tail -5
=========================== short test summary info ============================
FAILED pelican/tests/test_pelican.py::TestPelican::test_basic_generation_works
FAILED pelican/tests/test_pelican.py::TestPelican::test_custom_generation_works
FAILED pelican/tests/test_pelican.py::TestPelican::test_custom_locale_generation_works
================== 3 failed, 232 passed, 20 skipped in 5.83s ===================

Unfortunately, on my system, there is some other environmental issue that slightly fails these three tests (the content is generated correctly, but class names differ in a minor way, e.g. class="testingbp" becomes class="testingkc", I am still trying to figure this out)

Now, if we provide an invalid LANG value, suddenly, French tests will start to fail:

[galaxy@archlinux pelican]$ LANG=invalid invoke tests | tail -5
FAILED pelican/tests/test_pelican.py::TestPelican::test_custom_locale_generation_works
FAILED pelican/tests/test_settings.py::TestSettingsConfiguration::test_default_encoding
FAILED pelican/tests/test_utils.py::TestDateFormatter::test_french_locale - l...
FAILED pelican/tests/test_utils.py::TestDateFormatter::test_french_strftime
================== 6 failed, 229 passed, 20 skipped in 6.02s ===================

They also fail with a valid locale such as ru_RU (Russian):

[galaxy@archlinux pelican]$ LANG=ru_RU invoke tests | tail -5
FAILED pelican/tests/test_pelican.py::TestPelican::test_custom_locale_generation_works
FAILED pelican/tests/test_settings.py::TestSettingsConfiguration::test_default_encoding
FAILED pelican/tests/test_utils.py::TestDateFormatter::test_french_locale - l...
FAILED pelican/tests/test_utils.py::TestDateFormatter::test_french_strftime
================== 6 failed, 229 passed, 20 skipped in 5.97s ===================
[galaxy@archlinux pelican]$

Although, I think it should not really matter for a self-contained test suite, but I have ensured that all locales are installed on my system (despite that I have no use for the French locale):

[galaxy@archlinux pelican]$ locale -a
C
en_AU.utf8
en_US.utf8
fr_FR.utf8
POSIX
ru_RU.utf8
[galaxy@archlinux pelican]$

I need some help here since I am just starting to learn how Pelican's test suite is designed. Thanks!

[galaxy4public]

For the three failed test where class names differ - it is a result of a change in the upstream pygments module re: how it parses python. I think this is another indication that the tests need to be fixed to be less brittle.

[avaris]

You don't have ru_RU so it is not a valid locale for your system, ru_RU.utf8 is.

Well, we kind of assume you have a valid locale. python does that as well. Errors are basically from trying to reset the locale to system settings with this:

locale.setlocale(locale.LC_ALL, '')

at which point, python chokes because you have an invalid locale. We could try to safeguard against it but... I mean, why would someone have an invalid locale?

[galaxy4public]

Fair, enough. I am closing this one and will open an issue for things related to the updated pygments then (since those 3 failed tests took a chunk of my life to find the root cause) :)

[avaris]

Pygments is a bit notorious for causing some headache in this manner (see #1916). That's why it is fixed to a specific version in CI configuration, perhaps we should do the same with dev environment setup.

[justinmayer]

@galaxy4public: I am sorry to hear that this problem caused you some time, and undoubtedly some frustration as well. I hope you will take some small comfort from knowing that you tracked down a problem that also affected my local workstation, but I hadn't had time to figure out the source of the test result discrepancy. So I very much appreciate you taking the time to identify this issue. 🙇

And thanks to @avaris for helpfully linking back to where I pinned this previously. I just submitted a PR that updates and pins to the current Pygments 2.6.1 version and also moves said pinning to a requirements file for (hopefully) more consistent testing behavior across environments. Comments on that PR are welcome.