pyexpat changes in 3.10.14 broke Supervisor

[unlike777]

Bug report

Bug description:

Link for supervisor issue Supervisor/supervisor#1636

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/supervisord", line 33, in <module>
    sys.exit(load_entry_point('supervisor==4.2.4', 'console_scripts', 'supervisord')())
  File "/usr/bin/supervisord", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/supervisor/supervisord.py", line 38, in <module>
    from supervisor.medusa import asyncore_25 as asyncore
  File "/usr/lib/python3.10/site-packages/supervisor/medusa/asyncore_25.py", line 58, in <module>
    from supervisor.compat import as_string, as_bytes
  File "/usr/lib/python3.10/site-packages/supervisor/compat.py", line 69, in <module>
    import xmlrpclib
ModuleNotFoundError: No module named 'xmlrpclib'
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/supervisor/compat.py", line 67, in <module>
    import xmlrpc.client as xmlrpclib
  File "/usr/lib/python3.10/xmlrpc/client.py", line 138, in <module>
    from xml.parsers import expat
  File "/usr/lib/python3.10/xml/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ImportError: Error relocating /usr/lib/python3.10/lib-dynload/pyexpat.cpython-310-x86_64-linux-gnu.so: XML_SetReparseDeferralEnabled: symbol not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/supervisord", line 33, in <module>
    sys.exit(load_entry_point('supervisor==4.2.4', 'console_scripts', 'supervisord')())
  File "/usr/bin/supervisord", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/supervisor/supervisord.py", line 38, in <module>
    from supervisor.medusa import asyncore_25 as asyncore
  File "/usr/lib/python3.10/site-packages/supervisor/medusa/asyncore_25.py", line 58, in <module>
    from supervisor.compat import as_string, as_bytes
  File "/usr/lib/python3.10/site-packages/supervisor/compat.py", line 69, in <module>
    import xmlrpclib
ModuleNotFoundError: No module named 'xmlrpclib'

CPython versions tested on:

3.10

Operating systems tested on:

Linux

[Eclips4]

Hello!
From e2caef7:
The :mod:`xmlrpclib` module has been renamed to :mod:`xmlrpc.client` in Python 3.0.
So, there's no problem, just use a xmlrpc.client instead of xmlrpclib.
It was renamed long time ago, but name of supervisor issue says "Fatal error after update python from 3.10.13 to 3.10.14", I'm sure that this cannot be true, import xmlrpclib cannot work on any of 3.10 versions, probably you're missing something.

[unlike777]

Hello!
Thanks for the prompt feedback, but I don’t agree with you

The supervisor definitely works on 3.10.13 - verified.
And it stops working on 3.10.14.
It turns out that the minor version influences the supervisor.
If you look at what was done in 3.10.14, this is exactly a fix for an xml vulnerability in Python. https://github.com/python/cpython/commits/3.10/

If you look at the supervisor code, they solved the problem you describe
https://github.com/Supervisor/supervisor/blob/main/supervisor/compat.py#L66

try: # pragma: no cover
    import xmlrpc.client as xmlrpclib
except ImportError: # pragma: no cover
    import xmlrpclib

Also, the supervisor hasn’t changed anything for a long time, and the breakdown occurred precisely after the release of Python 3.10.14

The problem is on your side

[Eclips4]

A closer look at the traceback makes it clear that the problem is probably related to version of the pyexpat library bundled in your system. I'm not expert in this kind of issues, so I'm re-open the issue.

[AlexWaygood]

Cc. @hartwork @serhiy-storchaka

[hugovk]

Highlighting the most relevant part of the traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/supervisor/compat.py", line 67, in <module>
    import xmlrpc.client as xmlrpclib
  File "/usr/lib/python3.10/xmlrpc/client.py", line 138, in <module>
    from xml.parsers import expat
  File "/usr/lib/python3.10/xml/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ImportError: Error relocating /usr/lib/python3.10/lib-dynload/pyexpat.cpython-310-x86_64-linux-gnu.so: XML_SetReparseDeferralEnabled: symbol not found

Diff between the releases: v3.10.13...v3.10.14

Had three expat changes:

• [3.10] gh-115133: Fix tests for XMLPullParser with Expat 2.6.0 (GH-115164) #115525
• [3.10] Upgrade bundled libexpat to 2.6.0 (GH-115399) (GH-115468) #115473
• [3.10] gh-115398: Expose Expat >=2.6.0 reparse deferral API (CVE-2023-52425) (GH-115623) #116270

[hugovk]

And the related issues:

• test.test_xml_etree*.XMLPullParserTest.test_simple_xml fails with (system) expat 2.6.0 #115133
• Please upgrade bundled Expat to 2.6.0 (e.g. for the fix to CVE-2023-52425) #115399
• Please expose Expat >=2.6.0 API function XML_SetReparseDeferralEnabled #115398

This last one seems most relevant, to expose XML_SetReparseDeferralEnabled, as the traceback says XML_SetReparseDeferralEnabled: symbol not found.

We don't provide Linux installers (and don't provide any installers for 3.10 because it's in the security-only/source-code only phase).

Could it be a problem with a distro-provided Python installation?

(See also jupyterhub/repo2docker-action#113 for a similar report.)

[hartwork]

From a quick look my understanding is that @unlike777 is using Python binaries compiled on a host that did have symbol XML_SetReparseDeferralEnabled present in libexpat — through a backport or version >=2.6.0 — but the target system has installed binaries that lack these symbols. If that's what's going on, the fix is to get these two sides back in sync.

[hartwork]

@unlike777 which distro is this on, which release of the distro, and how did you install CPython and libexpat and which version are they now?

[unlike777]

@hartwork

Thank you very much for being included!

I use alpine:3, 3.16, 3.16.3, latest

Install the supervisor:
apk add supervisor

[supervisor 1/3] RUN apk add supervisor
2.126 (1/10) Installing mpdecimal (2.5.1-r1)
2.437 (2/10) Installing python3 (3.10.14-r0)
3.440 (3/10) Installing py3-appdirs (1.4.4-r3)
3.483 (4/10) Installing py3-more-itertools (8.13.0-r0)
3.532 (5/10) Installing py3-ordered-set (4.0.2-r3)
3.575 (6/10) Installing py3-parsing (2.4.7-r3)
3.629 (7/10) Installing py3-six (1.16.0-r1)
3.672 (8/10) Installing py3-packaging (21.3-r0)
3.720 (9/10) Installing py3-setuptools (59.4.0-r0)
3.818 (10/10) Installing supervisor (4.2.4-r0)

[hartwork]

@unlike777 thanks. I cannot reproduce the issue with Alpine 3.16. This will need a robust minimal reproducer. What's a way to reliably reproduce the issue?

[hartwork]

It's a bug in Alpine's packaging of Python 3.10.14-r0, not an upstream issue.

[AlexWaygood]

Great — thanks @hartwork!