pypy.exe -m ensurepip does not work when invoked from inside Windows Services for Linux

[gitlab-importer]

In Heptapod by @tekknolagi on Sep 24, 2023, 20:41

(posted on behalf of Andi McClure)

I use WSL (Windows Services for Linux) as my command line shell on Windows 10. I use basic tools such as grep, git etc inside WSL; for coding tools such as Rust cargo, Python, etc I usually (since WSL has some limitations) install Windows versions and invoke the Windows .exes from the Linux command line, which works fine, I just have to manually type the .exe at the end when I invoke them. In this respect I basically use WSL as if it were MSYS.
Today I decided to install Pypy. I downloaded the Win64 .zip and unzipped it to a convenient location. I then of course needed to run pip to install some modules. Pip was not included, but a helpful message in the interpreter, as well as the Pypy documentation, recommended to run pypy with -m ensurepip to install a pip. Fine.

When I did this, I got a long and unhelpful exception, ending with "The parameter is incorrect".

After wasting some time fiddling, it occurred to me that WSL could be somehow causing the problem, so I tried the command in a Windows CMD.EXE shell. This worked on the first try:

C:\Users\Andi>work\usr\pypy3.10-v7.3.12-win64\pypy.exe -m ensurepip
Looking in links: c:\Users\Andi\AppData\Local\Temp\tmpue1h28og
Processing c:\users\andi\appdata\local\temp\tmpue1h28og\setuptools-65.5.0-py3-none-any.whl
Processing c:\users\andi\appdata\local\temp\tmpue1h28og\pip-23.0.1-py3-none-any.whl
Installing collected packages: setuptools, pip
  WARNING: The scripts pip3.10.exe and pip3.exe are installed in 'C:\Users\Andi\work\usr\pypy3.10-v7.3.12-win64\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-23.0.1 setuptools-65.5.0

Expected behavior

Once an exe has been invoked inside WSL, it's just a normal Windows app running on normal Windows. So pypy should work the same if invoked from WSL Bash as if invoked from CMD.EXE, especially for a basic/"install process" kind of task like -m ensurepip.

If basic pypy functionality like ensurepip requires something special about the environment in which it is run— it's possible for example that my WSL was somehow creating janked environment variables, I don't know— then it should print a more helpful error message than "invalid parameter", and the environment requirements should be documented in the install instructions.

Configuration

I am using Windows 10 Version 10.0.19045 Build 19045. My WSL is a WSL1 VM running Ubuntu 20.04.6 LTS. My pypy is pypy3.10-v7.3.12-win64. I have a file in which I have saved a copy of os.environ of pypy when invoked from WSL and a copy of same when invoked from CMD.EXE, in case this is helpful for debugging this error, but I would prefer not to post this file on a public bug report for privacy reasons.

[gitlab-importer]

In Heptapod by @mcc on Sep 24, 2023, 20:42

Note, this was originally my bug, Max had to post it for me because regardless of how I worded it any issue I try to post is being "recognized as spam and discarded"

[gitlab-importer]

In Heptapod by @mcc on Sep 24, 2023, 20:48

Also note, this may be a general problem with pypy pip rather than ensurepip specifically. Once my pypy had a working pip, I tried pypy.exe -m pip install pdm and this resulted in a similar "parameter" error installing one of the dependencies.

Other packages I tried to install (pyreadline3, etc) did not have this problem. When I tried re-running the pdm install from a CMD.EXE window, it worked fine.
I have never had this problem or anything like it with Vanilla Python3 (I specifically did these same steps with 3.11) when invoked on Windows from inside WSL.

[gitlab-importer]

In Heptapod by @mcc on Sep 24, 2023, 20:50

pdm itself (which works when installed in Vanilla Python 3.11 from inside WSL) also fails with a similar error.

Andi@Alita:/mnt/c/Users/Andi/work/f/analogue-core-template-amaranth$ (cd src/fpga/amaranth_core/ && /mnt/c/Users/Andi/work/usr/pypy3.10-v7.3.12-win64/pypy.exe -m pdm install)
The saved Python interpreter doesn't match the project's requirement. Trying to find another one.
python.use_venv is on, creating a virtualenv for this project...
See C:\Users\Andi\AppData\Local\Temp\pdm-install-resolve-wohjxzvv.log for detailed debug log.
[OSError]: [WinError 87] The parameter is incorrect
Add '-v' to see the detailed traceback

This only occurs when installing with pdm (which I believe relies on pip), running scripts with pdm does not have the problem.

[gitlab-importer]

In Heptapod by @mattip on Sep 26, 2023, 09:11

It seems there is something off with the way PyPy-for-windows calls CreateProcess when running under wsl. The code in the cffi-based winapi is supposed to be a conversion of CPython's code, I wonder what is different.

[gitlab-importer]

In Heptapod by @nulano on Oct 11, 2023, 16:07

The error is caused by sys.executable = '' when running under WSL.

By testing with another program, I have found that WSL is translating the first argument from a path like /mnt/c/path/to/pypy.exe or ./path/to/pypy.exe to just pypy.exe, which fails to be resolved in pypy/module/sys/initpath.py:find_executable (unless the PyPy directory is in the system PATH or the current working directory).

However, the path is resolved correctly in pypy/module/sys/initpath.py:pypy_find_stdlib, perhaps pypy_find_executable should also consider dynamic_location?

@mcc I find that adding sys.executable = r"C:\path\to\pypy.exe" somewhere in pypy\Lib\site.py works as a workaround.

[gitlab-importer]

In Heptapod by @mattip on Oct 16, 2023, 20:25

@nulano can you suggest a way to test a fix for this?

[gitlab-importer]

In Heptapod by @nulano on Oct 16, 2023, 22:27

I can also reproduce the same issue by running the following code from PyPy launched normally:

import os, sys
os.chdir("C:\\")
print(sys.executable)  # C:\Tools\pypy3.10-v7.3.12-win64\pypy.exe
res = os.spawnv(os.P_WAIT, sys.executable, ["pypy.exe", "-c", '"import sys; assert len(sys.executable) > 0"'])
assert res == 0, res  # AssertionError: 1

[gitlab-importer]

In Heptapod by @mattip on Oct 26, 2023, 14:05

There was a problem with trailing new-line characters when reading PYPY_PORTABLE_DEPS.txt. Fixed in bbccbd51b0e9. Thanks for the report.

[gitlab-importer]

In Heptapod by @mattip on Oct 26, 2023, 14:06

Whoops, commented on the wrong issue. Reopened.

[gitlab-importer]

In Heptapod by @mattip on Oct 26, 2023, 19:26

For this snippet

>>> print(12, sys.executable)
>>> os.spawnv(os.P_WAIT, sys.executable, ["asdfasfd", "-c", '"import sys; print(12, sys.executable)"'])

OS	python	spawnv prints 12, <full path to python>?
Linux	CPython	only `12``
Linux	PyPy	only 12
Windows	CPython	yes
Windows	PyPy	only 12

[mattip]

We don't have a fix for this yet, I will mark it for v7.3.15. @nulano do you think you could take a look?

[nulano]

I think I've got a potential fix written.

I've added both an apptest (which passes) and an extra_tests test (not tested, I haven't run a translation yet).

What is the current procedure for testing?
I assume the buildbots are not yet migrated to the Git repo, so I can only test locally?

[mattip]

Buildbots are functional against this repo. You can push a branch here and then kick off a build. I am struggling a bit with the nightly build polling, but will be manually kicking off builds until I get the automation working.

[nulano]

I've pushed the changes to https://github.com/nulano/pypy/tree/sys_executable.

I don't think I have permissions to push to this repo, can the buildbots run off a fork?

[mattip]

I think a possible strategy would be to submit a PR which will trigger a github action to translate, you could then download the artifact. But for windows we will need to add that to the workflow matrix which currently only builds linux x86_64.

I don't think I have permissions to push to this repo

See the invite.

[nulano]

Ok, I've pushed to branch/issue4003 and ran the buildbots - looks OK.
See PR #4828.