Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add root path to sys.path in build env for embedded Python #11724

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

bebound
Copy link

@bebound bebound commented Jan 13, 2023

Fix #11670, related issue: python/cpython#100399, Azure/azure-cli#24781

In embedded python, the pyd files are put in root folder.
In standard python, they are put into DLLs folder.

  Directory: C:\Users\kk\AppData\Local\Programs\Python\Python310\DLLs

-a----        10/11/2022     16:59        1121152 unicodedata.pyd
-a----        10/11/2022     16:59          29560 winsound.pyd
-a----        10/11/2022     16:59          64384 _asyncio.pyd
-a----        10/11/2022     16:59          83320 _bz2.pyd
-a----        10/11/2022     16:59         123264 _ctypes.pyd
  Directory: C:\Users\kk\Downloads\python_emb

-a----        12/21/2022     14:38        1113976 unicodedata.pyd
-a----        12/21/2022     14:38          90440 vcruntime140.dll
-a----        12/21/2022     14:38          26488 winsound.pyd
-a----        12/21/2022     14:38          55680 _asyncio.pyd
-a----        12/21/2022     14:38          79736 _bz2.pyd
-a----        12/21/2022     14:38         106368 _ctypes.pyd

In build_env.py, the root folder is excluded by this logic, as root folder is in system_paths/system_sites.

original_sys_path = [
path for path in original_sys_path
if os.path.normcase(path) not in system_paths
]
sys.path = original_sys_path

If these pyd files need to be imported when install new package, ModuleNotFoundError is raised.
Currently, I find pwinput and pendulum can't be installed in embedded Python.

I'm not sure whether this change brings some side effect. Let me know if you have any concern.

Repro step

  1. Download https://www.python.org/ftp/python/3.10.9/python-3.10.9-embed-win32.zip and extract it
  2. curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
  3. .\python.exe get-pip.py
  4. .\python.exe -m pip install pwinput
  5. ModuleNotFoundError: No module named 'pyexpat' raises

It can be installed successfully with this PR.

@uranusjr
Copy link
Member

This looks like an issue in the embedded distribution? pip simply uses the sysconfig layout to reference the modules, and if a distribution fails to report the site locations, it’s on the distribution to fix that.

Comment on lines +106 to +107
import pyexpat
if os.path.dirname(pyexpat.__file__) == os.path.dirname(sys.executable):
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any other way to detect embedded Python?

@bebound
Copy link
Author

bebound commented Jan 13, 2023

Yes, it only affects embedded distribution.
I don't know why embedded Python decides to put pyd into root folder. DLLs is in sys.path in build env, everything works fine if pyd is in DLLs folder.
Since we can handle pyd correctly in DLLs folder, I think it's okey to make the logic compatible with pyd in root folder.
Before embedded Python changes its mind, can we fix it in pip?

@pfmoore
Copy link
Member

pfmoore commented Jan 13, 2023

In addition to the point @uranusjr made that this is an issue for the embedded distribution, I will note that the embedded distribution is explicitly documented as not being for general use - see the comment here

Using pip to manage dependencies as for a regular Python installation is not supported with this distribution, though with some care it may be possible to include and use pip for automatic updates.

So this is not intended or supported use anyway, and IMO pip should not try to special case it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"ModuleNotFoundError: No module named 'pyexpat'" when run pip install on embedded Python
3 participants