"ModuleNotFoundError: No module named 'pyexpat'" when run pip install on embedded Python

[bebound]

Bug report

ModuleNotFoundError: No module named 'pyexpat' is raised when use pip install on embedded Python.

But .\python.exe -c 'import xml.parsers.expat' works fine.

Steps to reproduce:

1. Download https://www.python.org/ftp/python/3.10.9/python-3.10.9-embed-win32.zip and extract it, delete python310._pth
2. curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
3. python get-pip.py
4. .\python.exe -m pip install pwinput
   Output:

Collecting pwinput
  Using cached pwinput-1.0.2.tar.gz (4.5 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\cli\base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\cli\req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\commands\install.py", line 400, in run
    requirement_set = resolver.resolve(
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_vendor\resolvelib\structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 155, in __bool__
    return any(self)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 297, in __init__
    super().__init__(
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 162, in __init__
    self.dist = self._prepare()
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 231, in _prepare
    dist = self._prepare_distribution()
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 308, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\operations\prepare.py", line 491, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\operations\prepare.py", line 577, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\operations\prepare.py", line 69, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\distributions\sdist.py", line 48, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\distributions\sdist.py", line 118, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\distributions\sdist.py", line 95, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_internal\utils\misc.py", line 685, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_vendor\pep517\wrappers.py", line 173, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_vendor\pep517\wrappers.py", line 319, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "C:\Users\kk\Downloads\python\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
  File "importlib\__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\kk\AppData\Local\Temp\pip-build-env-wz_0zi4g\overlay\Lib\site-packages\setuptools\__init__.py", line 16, in <module>
    import setuptools.version
  File "C:\Users\kk\AppData\Local\Temp\pip-build-env-wz_0zi4g\overlay\Lib\site-packages\setuptools\version.py", line 1, in <module>
    import pkg_resources
  File "C:\Users\kk\AppData\Local\Temp\pip-build-env-wz_0zi4g\overlay\Lib\site-packages\pkg_resources\__init__.py", line 32, in <module>
    import plistlib
  File "plistlib.py", line 61, in <module>
  File "xml\parsers\expat.py", line 4, in <module>
ModuleNotFoundError: No module named 'pyexpat'

Related issue: Azure/azure-cli#24781, pypa/pip#11670

Your environment

• CPython versions tested on:3.10.9 win32 embedded
• Operating system and architecture:Win11 amd64

[sergeyyurkov1]

I was able to fix the issue in my project with embedded Python running inside a virtual environment by prepending _ to pyexpat.pyd -> _pyexpat.pyd. Embedded Python 3.10.9. You may need to keep both versions, so just make a copy of the file and rename it.

Edit: It broke again.

[sergeyyurkov1]

I made a batch script to create a fully portable venv that will run on any Windows system without system-wide Python installations. It outputs a sample project folder named MyProject (adjustable in settings). You can choose which Python version to embed. After that, cd to the newly created project folder, and click run.bat. You can put your dependences in requirements.txt and click install.bat. These batch scripts are convenient ways to set everything up. I created this for a client who didn't want to juggle different versions of Python on their system but wanted the convenience of pyinstaller, which is complicated to use and rarely produces a working executable - it is very error prone. Hope this helps.

https://github.com/sergeyyurkov1/make_portable_python_env

Edit: the main issue is absolute imports that make modules such as pyexpat invisible to the environment. My solution is to replace those with relative paths and everything works as expected. You can look at my script to see the steps and redo them manually if you wish to.

[bebound]

I found the cause of this error, see pypa/pip#11724
I don't know whether it should be fixed by pip or Python...

Workaround:

1. Create DLLs folder in embedded Python root folder
2. Move pyd and dll (don't touch python3**.dll) files in root folder to DLLs.