Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ repos:
rev: v0.45.0
hooks:
- id: markdownlint
exclude: docs/source/include-toplevel-*


- repo: local
Expand Down
14 changes: 8 additions & 6 deletions README-Unicode.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ for i in 2.7 3.{6,7};do echo "$i:";
LC_ALL=C python$i -c 'open("/usr/share/hwdata/pci.ids").read()';done
```

```
```text
2.7:
3.6:
Traceback (most recent call last):
Expand All @@ -106,20 +106,22 @@ UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 97850: ordi
3.7:
```

This error means that the `'ascii' codec` cannot handle input ord() >= 128, and as some Video cards use `²` to reference their power, the `ascii` codec chokes on them.

It means `xcp.pci.PCIIds()` cannot use `open("/usr/share/hwdata/pci.ids").read()`.
The `'ascii'` codec fails on all bytes >128.
For example, it cannot decode the bytes representing `²` (UTF-8: power of two) in the PCI IDs database.
To read `/usr/share/hwdata/pci.ids`, we must use `encoding="utf-8"`.

While Python 3.7 and newer use UTF-8 mode by default, it does not set up an error handler for `UnicodeDecodeError`.

As it happens, some older tools output ISO-8859-1 characters hard-coded and these aren't valid UTF-8 sequences, and even newer Python versions need error handlers to not fail:
Also, some older tools output ISO-8859-1 characters
These aren't valid UTF-8 sequences.
For all Python versions, we need to use error handlers to handle them:

```sh
echo -e "\0262" # ISO-8859-1 for: "²"
python3 -c 'open(".text").read()'
```

```
```text
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<frozen codecs>", line 322, in decode
Expand Down
4 changes: 2 additions & 2 deletions tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,9 @@ commands =
coverage xml -o {envlogdir}/coverage.xml --fail-under {env:XCP_COVERAGE_MIN:78}
coverage lcov -o {envlogdir}/coverage.lcov
coverage html -d {envlogdir}/htmlcov
coverage html -d {envlogdir}/htmlcov-tests --fail-under {env:TESTS_COVERAGE_MIN:96} \
coverage html -d {envlogdir}/htmlcov-tests --fail-under {env:TESTS_COVERAGE_MIN:95} \
--include="tests/*"
diff-cover --compare-branch=origin/master --exclude xcp/dmv.py \
diff-cover --compare-branch=origin/master \
{env:PY3_DIFFCOVER_OPTIONS} --fail-under {env:DIFF_COVERAGE_MIN:92} \
--html-report {envlogdir}/coverage-diff.html \
{envlogdir}/coverage.xml
Expand Down