pip install selects 32 bit wheels for 64 bit python if vcvarsall.bat amd64_x86 in environment

[RyanThornton]

BPO	38989
Nosy	@vsajip, @merwok, @zooba, @dstufft, @altendky

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-09-28.20:54:35.119>
created_at = <Date 2019-12-06.18:49:45.840>
labels = ['3.8', 'type-bug', 'library']
title = 'pip install selects 32 bit wheels for 64 bit python if vcvarsall.bat amd64_x86 in environment'
updated_at = <Date 2020-09-29.11:13:15.455>
user = 'https://bugs.python.org/RyanThornton'

bugs.python.org fields:

activity = <Date 2020-09-29.11:13:15.455>
actor = 'vinay.sajip'
assignee = 'none'
closed = True
closed_date = <Date 2020-09-28.20:54:35.119>
closer = 'steve.dower'
components = ['Distutils']
creation = <Date 2019-12-06.18:49:45.840>
creator = 'Ryan Thornton'
dependencies = []
files = []
hgrepos = []
issue_num = 38989
keywords = ['3.5regression']
message_count = 6.0
messages = ['357936', '357945', '357946', '358129', '377627', '377662']
nosy_count = 7.0
nosy_names = ['vinay.sajip', 'eric.araujo', 'steve.dower', 'dstufft', 'altendky', 'Ryan Thornton', 'puetzk']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38989'
versions = ['Python 3.8']

[RyanThornton]

## Expected Behavior

pip install should download dependencies matching the architecture of the python executable being used.

## Actual Behavior

When calling pip install from a Visual Studio command prompt configured to cross compile from x64 to x86, pip installs wheels matching the architecture of Visual Studio's cross compile target (i.e. VSCMD_ARG_TGT_ARCH=x86) and not the architecture of python itself (x64).

This results in a broken installation of core libraries.

## Steps to Reproduce

System Details:
Windows 10 x64
Python 3.8 x64
Visual Studio 2017 15.9.14

Environment Details:
vcvarsall.bat amd64_x86

1. "C:\Program Files\Python38\python.exe" -mvenv "test"
2. cd test\Scripts
3. pip install cffi==1.13.2

Results in the following:

Collecting cffi
Using cached https://files.pythonhosted.org/packages/f8/26/5da5cafef77586e4f7a136b8a24bc81fd2cf1ecb71b6ec3998ffe78ea2cf/cffi-1.13.2-cp38-cp38-win32.whl

## Context

I think the regression was introduced here:
62dfd7d

62dfd7d

[puetzk]

Besides the fact the MSVC's target platform isn't really related to the architecture for dependencies being installed, I'm not sure VSCMD_ARG_TGT_ARCH is an appropriate variable to look at in the first place. It doesn't seem to be at all documented, and (from looking at the implemenation) it seems to be meant as how parse_cmd.bat communicates with vcvars.bat, dotnet.bat, winsdk.bat, etc (these are all sub-programms that make up vcvarsall.bat)

I think %PLATFORM% (which generally ends up set equal to %VSCMD_ARG_TGT_ARCH% once vsdevcmd\ext\vcvars.bat has done its thing) is might be a better variable that cross-compiling should actually look at when it wants to know what architecture MSVC will compile to.

[puetzk]

Just to link the various pieces together: I think

https://bitbucket.org/pypa/distlib/src/2d145da7cb42590039fbd56a9ab764d5d4716a98/distlib/wheel.py#lines-53

is the place in distlib that's being breaking pip due to redefining distutils.util.get_platform() as the cross-compile target architecture rather than the build architecture.

https://docs.python.org/3/distutils/apiref.html doesn't show it as a documented public function, but the commit Ryan linked did introduce a new get_host_plaform() that matches the old behavior; if that became public and distlib used itinstead (conditional on the python version?) that would also be a way to fix the symptom Ryan noted (pip now choosing the wrong wheel to install).

I don't know how the pieces here fit together well enough to now which side should change.

@zooba: adding you to the Cc since it was your comments at #11774 (comment) that seem to have led to the decision to re-define get_platform, maybe you have some relevant context?

[zooba]

maybe you have some relevant context

Nothing you haven't figured out yourself. As you say, they're both private functions, and as we investigated adding a "get_target_platform" function, it turned out that nearly all users were expecting the target platform, so rather than updating everything we added one for the less-common case.

get_host_platform was not defined before, so I'd say use it if it's there and use get_platform if it's not.

I doubt we're going to add any new public/supported APIs to distutils ever again, to be honest.

[zooba]

I'm closing this as external, as the canonical source for platform tags is now the packaging library.

Either pip and/or distlib should switch to using that library or the logic included (which I'll note is considerably more complex - but also more well defined and tested - than what is in distutils).

[vsajip]

FYI I'm in the process of updating distlib to add the get_platform() / get_host_platform() distinction. The next release should have it.