Test suite needs adjustments for Expat >=2.4.5

[hartwork]

BPO	46811
Nosy	@ned-deily, @ambv, @mgorny, @hartwork, @miss-islington
PRs	bpo-46811: Make test suite support Expat >=2.4.5 #31453 [3.9] bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453) #31469 [3.8] bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453) #31470 [3.7] bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453) #31471 [3.10] bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453) #31472

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-20.19:54:05.845>
labels = ['3.8', '3.9', '3.10', '3.11', 'expert-XML', '3.7']
title = 'Test suite needs adjustments for Expat >=2.4.5'
updated_at = <Date 2022-02-22.20:58:09.013>
user = 'https://github.com/hartwork'

bugs.python.org fields:

activity = <Date 2022-02-22.20:58:09.013>
actor = 'lukasz.langa'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['XML']
creation = <Date 2022-02-20.19:54:05.845>
creator = 'sping'
dependencies = []
files = []
hgrepos = []
issue_num = 46811
keywords = ['patch']
message_count = 6.0
messages = ['413605', '413654', '413658', '413659', '413669', '413743']
nosy_count = 5.0
nosy_names = ['ned.deily', 'lukasz.langa', 'mgorny', 'sping', 'miss-islington']
pr_nums = ['31453', '31469', '31470', '31471', '31472']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46811'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[hartwork]

It has been reported at https://bugs.python.org/issue46794#msg413587 that the current CPython test suite needs some adjustments for Expat >=2.4.5. Since that is somewhat separate from updating the bundled copy of Expat to >=2.4.6, I am creating this dedicated ticket. I pull request for discussion will follow shortly.

[ambv]

New changeset 2cae938 by Sebastian Pipping in branch 'main':
bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453)
2cae938

[ambv]

New changeset 7da97f6 by Miss Islington (bot) in branch '3.10':
bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453) (GH-31472)
7da97f6

[ambv]

New changeset 336a916 by Miss Islington (bot) in branch '3.9':
bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453) (GH-31469)
336a916

[ned-deily]

New changeset d4f5bb9 by Miss Islington (bot) in branch '3.7':
bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453) (GH-31471)
d4f5bb9

[ambv]

New changeset fdfd7a9 by Łukasz Langa (Sebastian Pipping) in branch '3.8':
bpo-46811: Make test suite support Expat >=2.4.5 (GH-31453)
fdfd7a9

[vfazio]

If python is built to use the system libexpat, the version check used in the listed PRs will not necessarily be effective if the distribution ports patches back:

# dpkg -l libexpat1
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version         Architecture Description
+++-===============-===============-============-===============================
ii  libexpat1:arm64 2.2.6-2+deb10u4 arm64        XML parsing C library - runtime

# python3 -c "import pyexpat; print(pyexpat.version_info); from xml.dom.minidom import parseString; parseString('<element xmlns:abc=\"http:abc.com/de f g/hi/j k\"><abc:foo /></element>')"
(2, 2, 6)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/xml/dom/minidom.py", line 1998, in parseString
    return expatbuilder.parseString(string)
  File "/usr/local/lib/python3.10/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
  File "/usr/local/lib/python3.10/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
xml.parsers.expat.ExpatError: syntax error: line 1, column 0

# dpkg -i libexpat1_2.2.6-2+deb10u1_arm64.deb
dpkg: warning: downgrading libexpat1:arm64 from 2.2.6-2+deb10u4 to 2.2.6-2+deb10u1
(Reading database ... 22507 files and directories currently installed.)
Preparing to unpack libexpat1_2.2.6-2+deb10u1_arm64.deb ...
Unpacking libexpat1:arm64 (2.2.6-2+deb10u1) over (2.2.6-2+deb10u4) ...
Setting up libexpat1:arm64 (2.2.6-2+deb10u1) ...
Processing triggers for libc-bin (2.28-10+deb10u1) ...


# dpkg -l libexpat1
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name            Version         Architecture Description
+++-===============-===============-============-===============================
ii  libexpat1:arm64 2.2.6-2+deb10u1 arm64        XML parsing C library - runtime

# python3 -c "import pyexpat; print(pyexpat.version_info); from xml.dom.minidom import parseString; parseString('<element xmlns:abc=\"http:abc.com/de f g/hi/j k\"><abc:foo /></element>')"
(2, 2, 6)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/xml/dom/minidom.py", line 1998, in parseString
    return expatbuilder.parseString(string)
  File "/usr/local/lib/python3.10/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
  File "/usr/local/lib/python3.10/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
  File "/build/python/3.10.2/python-3.10.2/Modules/pyexpat.c", line 416, in StartElement
  File "/usr/local/lib/python3.10/xml/dom/expatbuilder.py", line 738, in start_element_handler
    uri, localname, prefix, qname = _parse_ns_name(self, name)
  File "/usr/local/lib/python3.10/xml/dom/expatbuilder.py", line 129, in _parse_ns_name
    raise ValueError("Unsupported syntax: spaces in URIs not supported: %r" % name)
ValueError: Unsupported syntax: spaces in URIs not supported: 'http:abc.com/de f g/hi/j k foo abc'

[ambv]

@vfazio, system expat which isn't truly any upstream version but a series of cherry-picked patches on an old version, is not something upstream Python is able to support. In this case you should be using your distribution's Python to get an internally consistent state.

[vfazio]

@vfazio, system expat which isn't truly any upstream version but a series of cherry-picked patches on an old version, is not something upstream Python is able to support. In this case you should be using your distribution's Python to get an internally consistent state.

@ambv Using a distribution's package isn't always feasible, especially if there's no port for the latest upstream version (Debian Bullseye does not provide 3.10.x) so many people will compile their own. A number of places document configuring --with-system-expat to take advantage of the system provided library to receive security updates vs using CPython's in-tree version.

I'm just calling out that people may see these tests fail if they built Python to leverage the external libraries

[hartwork]

With all the linked pull requests merged, should this issue be closed as fixed?