qa: set locale to C.UTF-8 in tox.ini #49321

tchaikov · 2022-12-08T06:47:39Z

as ansible is using UTF-8 encoded characters in the file names, so, to avoid failures like:

File "/home/jenkins-build/build/workspace/ceph-pull-requests/qa/.tox/py3/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 217, in untar_file
with open(path, "wb") as destfp:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 137-140: ordinal not in range(256)

we have to set a locale which is able to handle UTF-8.

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

tchaikov · 2022-12-08T06:48:34Z

this change should address the first failure in https://jenkins.ceph.com/job/ceph-pull-requests/107859/console

tchaikov · 2022-12-08T13:35:32Z

@adk3798 hi Adam, could you help review this change as well?

adk3798 · 2022-12-08T14:06:20Z

still seeing

usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Missing target module, package, files, or command.
mypy: exit 2 (0.22 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr> mypy --config-file=../../mypy.ini pid=3198086
mypy: FAIL ✖ in 35.1 seconds

but thought this looked alright.

tchaikov · 2022-12-08T14:13:24Z

that's a different issue from the one fixed by this change. i think this is also a fallout of upgrading tox -- it could be interpreting the newlines in a variable differently.

adk3798 · 2022-12-08T14:14:15Z

that's a different issue from the one fixed by this change. i think this is also a fallout of upgrading tox -- it could be interpreting the newlines in a variable differently.

okay, well this PR looks good then anyway

phlogistonjohn

Hi @tchaikov can we use the C.UTF-8 locale instead? As part of my efforts to build and test in containers, many of the images only have a minimal set of locales included. For example I had to change install-deps.sh to prevent errors when running in the container (see cd1cd1b)

Thanks!

phlogistonjohn · 2022-12-08T14:27:15Z

I'd also mention that I think my build-and-test-in-containers effort could help avoid these sudden scrambles to fix things when packages on the build hosts change ;-)
If I understand the cause of the sudden need to update our tox.ini files correctly.

as ansible is using UTF-8 encoded characters in the file names, so, to avoid failures like: File "/home/jenkins-build/build/workspace/ceph-pull-requests/qa/.tox/py3/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 217, in untar_file with open(path, "wb") as destfp: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 137-140: ordinal not in range(256) we have to set a locale which is able to handle UTF-8. see also ceph/teuthology#1671 Signed-off-by: Kefu Chai <tchaikov@gmail.com>

tchaikov · 2022-12-08T14:40:23Z

Hi @tchaikov can we use the C.UTF-8 locale instead? As part of my efforts to build and test in containers, many of the images only have a minimal set of locales included. For example I had to change install-deps.sh to prevent errors when running in the container (see cd1cd1b)

Thanks!

@phlogistonjohn hi John, done.

tchaikov · 2022-12-08T14:41:24Z

I'd also mention that I think my build-and-test-in-containers effort could help avoid these sudden scrambles to fix things when packages on the build hosts change ;-)
If I understand the cause of the sudden need to update our tox.ini files correctly.

looking forward to it!

phlogistonjohn

LGTM

otherwise it is mising when running test, and we'd have following failure: py3: exit 2 (0.00 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/qa> pytest --assert=plain test_import.py Signed-off-by: Kefu Chai <tchaikov@gmail.com>

tchaikov · 2022-12-08T16:16:47Z

jenkins test make check

py3 tries to import all python modules to ensure that they are python3 compatible. but the installation fails on jenkins test node: Resolved https://github.com/ceph/teuthology.git to commit 4da97cf64e542f347ec47b7bdbe5eca99759f9b7 Installing build dependencies: started error: subprocess-exited-with-error Signed-off-by: Kefu Chai <tchaikov@gmail.com>

as we always test with ubuntu jammy, which does not provide python3.7: py37: skipped because could not find python interpreter with spec(s): py37 so there is no point testing with python3.7. also, in tox v4, it is not able to handle "key = value" anymore, where value has newlines in it. so we need to find a better way passing command line options to the test command. this change partially reverts 2dd86c9 Signed-off-by: Kefu Chai <tchaikov@gmail.com>

When creating ceph#49359 I was testing on an outdated branch and didn't realize part of what I was fixing had already been fixed in ceph#49321. Basically ended up changing what a variable "mypy_args" is set to but that variable is no longer being used. It has no actual effect but we should remove the extraneous code. Signed-off-by: Adam King <adking@redhat.com>

tchaikov force-pushed the wip-qa-install-with-utf-8 branch from 7900f05 to 8bd2890 Compare December 8, 2022 06:47

tchaikov changed the title ~~qa: set local to en_US.UTF-8 in tox.ini~~ qa: set locale to en_US.UTF-8 in tox.ini Dec 8, 2022

trociny mentioned this pull request Dec 8, 2022

mgr/cephadm: try to avoid pull when getting container image info #48973

Merged

14 tasks

tchaikov force-pushed the wip-qa-install-with-utf-8 branch from 8bd2890 to 0cba7a3 Compare December 8, 2022 13:35

tchaikov assigned adk3798 Dec 8, 2022

adk3798 approved these changes Dec 8, 2022

View reviewed changes

phlogistonjohn requested changes Dec 8, 2022

View reviewed changes

tchaikov force-pushed the wip-qa-install-with-utf-8 branch from 3a5aa7c to e6a6bbd Compare December 8, 2022 14:39

tchaikov force-pushed the wip-qa-install-with-utf-8 branch from e6a6bbd to 82dbfaf Compare December 8, 2022 14:40

phlogistonjohn approved these changes Dec 8, 2022

View reviewed changes

tchaikov changed the title ~~qa: set locale to en_US.UTF-8 in tox.ini~~ qa: set locale to C.UTF-8 in tox.ini Dec 8, 2022

qa: add pytest to deps of py3 env

127dd54

otherwise it is mising when running test, and we'd have following failure: py3: exit 2 (0.00 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/qa> pytest --assert=plain test_import.py Signed-off-by: Kefu Chai <tchaikov@gmail.com>

tchaikov force-pushed the wip-qa-install-with-utf-8 branch from 82dbfaf to db0f16d Compare December 8, 2022 15:58

github-actions bot added build/ops mgr pybind labels Dec 8, 2022

tchaikov added 2 commits December 9, 2022 01:07

tchaikov force-pushed the wip-qa-install-with-utf-8 branch from 534e98a to c67106b Compare December 8, 2022 17:07

tchaikov merged commit af7d7d3 into ceph:main Dec 9, 2022

tchaikov deleted the wip-qa-install-with-utf-8 branch December 9, 2022 00:31

This was referenced Dec 9, 2022

mgr/pybind: fix mypy and autopep8 arg parsing #49359

Merged

quincy: pybind/mgr: fixup after upgrading tox versions #49361

Merged

pacific: pybind/mgr: fixup after upgrading tox versions #49363

Merged

adk3798 mentioned this pull request Dec 10, 2022

pybind/mgr: partial reversion of #49359 #49375

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

qa: set locale to C.UTF-8 in tox.ini #49321

qa: set locale to C.UTF-8 in tox.ini #49321

tchaikov commented Dec 8, 2022

tchaikov commented Dec 8, 2022

tchaikov commented Dec 8, 2022

adk3798 commented Dec 8, 2022 •

edited

tchaikov commented Dec 8, 2022

adk3798 commented Dec 8, 2022

phlogistonjohn left a comment

phlogistonjohn commented Dec 8, 2022

tchaikov commented Dec 8, 2022

tchaikov commented Dec 8, 2022

phlogistonjohn left a comment

tchaikov commented Dec 8, 2022

qa: set locale to C.UTF-8 in tox.ini #49321

qa: set locale to C.UTF-8 in tox.ini #49321

Conversation

tchaikov commented Dec 8, 2022

Contribution Guidelines

Checklist

tchaikov commented Dec 8, 2022

tchaikov commented Dec 8, 2022

adk3798 commented Dec 8, 2022 • edited

tchaikov commented Dec 8, 2022

adk3798 commented Dec 8, 2022

phlogistonjohn left a comment

Choose a reason for hiding this comment

phlogistonjohn commented Dec 8, 2022

tchaikov commented Dec 8, 2022

tchaikov commented Dec 8, 2022

phlogistonjohn left a comment

Choose a reason for hiding this comment

tchaikov commented Dec 8, 2022

adk3798 commented Dec 8, 2022 •

edited