diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index ac73ec2b..ba7cb9cb 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -93,6 +93,7 @@ repos: rev: v0.45.0 hooks: - id: markdownlint + exclude: docs/source/include-toplevel-* - repo: local diff --git a/README-Unicode.md b/README-Unicode.md index 3f21dff9..c7ddb018 100644 --- a/README-Unicode.md +++ b/README-Unicode.md @@ -95,7 +95,7 @@ for i in 2.7 3.{6,7};do echo "$i:"; LC_ALL=C python$i -c 'open("/usr/share/hwdata/pci.ids").read()';done ``` -``` +```text 2.7: 3.6: Traceback (most recent call last): @@ -106,20 +106,22 @@ UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 97850: ordi 3.7: ``` -This error means that the `'ascii' codec` cannot handle input ord() >= 128, and as some Video cards use `²` to reference their power, the `ascii` codec chokes on them. - -It means `xcp.pci.PCIIds()` cannot use `open("/usr/share/hwdata/pci.ids").read()`. +The `'ascii'` codec fails on all bytes >128. +For example, it cannot decode the bytes representing `²` (UTF-8: power of two) in the PCI IDs database. +To read `/usr/share/hwdata/pci.ids`, we must use `encoding="utf-8"`. While Python 3.7 and newer use UTF-8 mode by default, it does not set up an error handler for `UnicodeDecodeError`. -As it happens, some older tools output ISO-8859-1 characters hard-coded and these aren't valid UTF-8 sequences, and even newer Python versions need error handlers to not fail: +Also, some older tools output ISO-8859-1 characters +These aren't valid UTF-8 sequences. +For all Python versions, we need to use error handlers to handle them: ```sh echo -e "\0262" # ISO-8859-1 for: "²" python3 -c 'open(".text").read()' ``` -``` +```text Traceback (most recent call last): File "", line 1, in File "", line 322, in decode diff --git a/tox.ini b/tox.ini index 82d52514..bcfdeab4 100644 --- a/tox.ini +++ b/tox.ini @@ -119,9 +119,9 @@ commands = coverage xml -o {envlogdir}/coverage.xml --fail-under {env:XCP_COVERAGE_MIN:78} coverage lcov -o {envlogdir}/coverage.lcov coverage html -d {envlogdir}/htmlcov - coverage html -d {envlogdir}/htmlcov-tests --fail-under {env:TESTS_COVERAGE_MIN:96} \ + coverage html -d {envlogdir}/htmlcov-tests --fail-under {env:TESTS_COVERAGE_MIN:95} \ --include="tests/*" - diff-cover --compare-branch=origin/master --exclude xcp/dmv.py \ + diff-cover --compare-branch=origin/master \ {env:PY3_DIFFCOVER_OPTIONS} --fail-under {env:DIFF_COVERAGE_MIN:92} \ --html-report {envlogdir}/coverage-diff.html \ {envlogdir}/coverage.xml