Using `--platform manylinux2010` includes pyarrow wheel for `manylinux2014` #1355

Nimayer · 2021-06-04T13:31:13Z

I'm building a pex file for a project with the following command

pex -v --platform manylinux2010-x86_64-cp-38-cp38 -f $PWD/wheels uploader_core -r requirements.txt -e uploader_core.uploader_core -o uploader_core.pex

This is the content of requirements.txt

pandas>=1.2.4
numpy>=1.16.5
pyarrow>=4.0.0

This is the output of the pex command above

pex: Building pex :: Resolving distributions (['uploader_core', 'requirements.txt']) :: Resolving requirements. :: Resolving for:
pex: Hashing pex
pex: Hashing pex: 217.5ms
pex: Isolating pex: 0.2ms
pex: Building pex :: Resolving distributions (['uploader_core', 'requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='uploader_core', processed_text='uploader_core', source='<string>', start_line=1, end_line=1), requirement=Requirement.parse('uploader_core'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/mnt/c/Users/fizzo/Documents/Software/mnand_db_uploader_core/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/mnt/c/Users/fizzo/Documents/Software/mnand_db_uploader_core/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pyarrow>=4.0.0\n', processed_text='pyarrow>=4.0.0', source='/mnt/c/Users/fizzo/Documents/Software/mnand_db_uploader_core/requirements.txt', start_line=3, end_line=3pex: Building pex :: Resolving distributions (['uploader_core', 'requirements.txt']) :: Resolving requirements. :: Installing:
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c898440282606f7a601')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl', fingerprint='59b1b3a0fbea45fd6439ce2472b522d81f974f1a')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl', fingerprint='b97c6966fb8fd273f0bc8533cf0dee4978b9f373')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/uploader_core-0pex: Using cached installation of numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl at /home/fizzo/.pex/installed_wheels/f9497f670630fa14a1d41bd5fee325e076f7cf38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

As you can see from the log, pex includes the pyarrow wheel for manylinux2014, even though a manylinux2010 wheel is available.
Relevant log line:

InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmph0c491j0/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl', fingerprint='59b1b3a0fbea45fd6439ce2472b522d81f974f1a')

The resulting binary gives an error on CentOS 6 which is manylinux2010

I've also tried the --manylinux=manylinux2010 option without success.

The text was updated successfully, but these errors were encountered:

jsirois · 2021-06-10T16:56:42Z

@Nimayer I'm missing two bits of information that would be useful debugging this:

What are the requirements of uploader_core (I assume that is a local project with a setup.py or pyproject.toml).
What is the contents of the $PWD/wheels find links repo?

jsirois · 2021-06-10T17:04:39Z

I get the expected result without the items above:

$ cat requirements.txt 
pandas>=1.2.4
numpy>=1.16.5
pyarrow>=4.0.0

$ pex -v --platform manylinux2010-x86_64-cp-38-cp38 -r requirements.txt -o uploader_core.pex
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Resolving for:
pex: Hashing pex: 29.0ms                                                                                                                                                                                                      
pex: Isolating pex: 0.1ms
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/home/jsirois/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/home/jsirois/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Installing:e/jsirois/requirements.txt', start_line=3, end_line=3), requirement=Requirement.parse('pyarrow>=4.0.0'), editable=False)
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl', fingerprint='b97c6966fb8fd273f0bc8533cf0dee4978b9f373')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl', fingerprint='45b71a81ff5a2f895eb2d0864118ec491d88f320')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c8984402826pex: Using cached installation of six-1.16.0-py2.py3-none-any.whl at /home/jsirois/.pex/installed_wheels/79e6f2e4f9e24898f1896df379871b9c9922f147/six-1.16.0-py2.py3-none-any.whl                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
pex: Using cached installation of python_dateutil-2.8.1-py2.py3-none-any.whl at /home/jsirois/.pex/installed_wheels/3005ff67df93ee276fb8631e17c677df852254ad/python_dateutil-2.8.1-py2.py3-none-any.whl
pex: Using cached installation of pytz-2021.1-py2.py3-none-any.whl at /home/jsirois/.pex/installed_wheels/b97c6966fb8fd273f0bc8533cf0dee4978b9f373/pytz-2021.1-py2.py3-none-any.whl
pex: Using cached installation of pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl at /home/jsirois/.pex/installed_wheels/45b71a81ff5a2f895eb2d0864118ec491d88f320/pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl
pex: Using cached installation of numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl at /home/jsirois/.pex/installed_wheels/f9497f670630fa14a1d41bd5fee325e076f7cf38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
pex: Using cached installation of pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl at /home/jsirois/.pex/installed_wheels/853375618e08d469a2114c898440282606f7a601/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl
pex: Building pex: 1940.2ms
pex:   Resolving distributions (['requirements.txt']): 1939.5ms
pex:     Resolving requirements.: 1685.9ms
pex:       Resolving for:
  DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')): 1357.7ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/home/jsirois/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/home/jsirois/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pyarrow>=4.0.0\n', processed_text='pyarrow>=4.0.0', source='/home/jsirois/requirements.txt', start_line=3, end_line=3), requirement=Requirement.parse('pyarrow>=4.0.0'), editable=False): 0.2ms
pex:       Installing:
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl', fingerprint='b97c6966fb8fd273f0bc8533cf0dee4978b9f373')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl', fingerprint='45b71a81ff5a2f895eb2d0864118ec491d88f320')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpbot1rwln/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c898440282606f7a601'): 257.1ms
Saving PEX file to uploader_core.pex
pex: Zipping PEX file.: 6056.1ms

$ unzip -qc uploader_core.pex PEX-INFO | jq .distributions
{
  "numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl": "dcbc5a8eb9b1f1228b9637ab9f23e0eb28ac32fc",
  "pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl": "ce909f800f6952caa20469f92e3eee5b450e483e",
  "pyarrow-4.0.1-cp38-cp38-manylinux2010_x86_64.whl": "11f2413b8f295315374525dacc1169016e05c6df",
  "python_dateutil-2.8.1-py2.py3-none-any.whl": "77d0214fbad3406ecd6d73f2b776bf03c373e978",
  "pytz-2021.1-py2.py3-none-any.whl": "196c12ed57d65e6f65ac29a310d9ada7dfb1a33f",
  "six-1.16.0-py2.py3-none-any.whl": "035d7c208925c1832def39b592f3477ca36397bf"
}

Nimayer · 2021-06-11T13:15:39Z

Thank you for the reply!

@Nimayer I'm missing two bits of information that would be useful debugging this:

What are the requirements of uploader_core (I assume that is a local project with a setup.py or pyproject.toml).

uploader_core is a local project with the following setup.py

from distutils.core import setup

setup(name='uploader_core',
      version='0.3',
      description='Engine for specification-checking, summary computation and converting CSV to Parquet format.',
      author='Federico Amedeo Izzo',
      author_email='fizzo@micron.com',
      url='https://bitbucket.micron.com/bbdc/projects/MNANDPREQUAL/repos/mnand_db_uploader_core/',
      packages=['uploader_core']
     )

Before launching the pex command I generated a wheel for uploader_core by using the following command

pip3 wheel -w . .

What is the contents of the $PWD/wheels find links repo?

$PWD/wheels contains only the wheel for uploader_core:

uploader_core-0.3-py3-none-any.whl

I can add that I am using pex 2.1.42 installing using pip3 20.0.2 on Ubuntu 20.04

I see in your later reply that you cannot reproduce the problem, since in your case the correct manylinux2010 wheel of pyarrow is downloaded.

Can it be a problem related to the pex/pip3 version I am using?

jsirois · 2021-06-11T14:03:52Z

uploader_core is a local project with the following setup.py

I don't think its relevant, but is it correct to say that that setup.py is technically lying? Presumably that code has an install_requires=["pandas>=1.2.4", "numpy>=1.16.5", "pyarrow>=4.0.0"] in reality and that's why you're forced to use the requirements.txt you use when building the PEX file instead of just doing this all in 1 step with:

pex uploader_core -e uploader_core.uploader_core -o uploader_core.pex

I can add that I am using pex 2.1.42 installing using pip3 20.0.2 on Ubuntu 20.04

I was also using Pex 2.1.42. The local Pip version is not relevant, Pex vendors its own copy of Pip.

So, I notice these paths in your output now that I read more closely: /mnt/c. Are you using Windows? In particular Ubuntu 20.04 under WSL? If so, can you try this using docker and the ubuntu:20.04 image to see if the problem persists? If it does; then I should be able to repro with docker.

Nimayer · 2021-06-18T09:42:57Z

Sorry for the late reply.

Yes, I am using Ubuntu 20.04 under WSL 1, and I am able to reproduce the issue on WSL1/Ubuntu 20.04 and an amd64 VM running Ubuntu 20.04.

I'm also able to reproduce the issue on a ubuntu:20.04 docker image, these are the commands that I used:
(Ignore the --trusted-host and --index options, I'm working under a corporate HTTPS proxy and I can't get a proper CA certificate bundle for pip3 and pex)

docker run --name pex_test -it ubuntu:20.04
echo -e 'pandas>=1.2.4\nnumpy>=1.16.5\npyarrow>=4.0.0' > requirements.txt
apt update; apt install -y python3-pip
pip3 install pex --trusted-host files.pythonhosted.org
pex -v --platform manylinux2010-x86_64-cp-38-cp38 -r requirements.txt -o uploader_core.pex --index http://files.pythonhosted.org

output:

root@a0661c2f1e33:/# pex -v --platform manylinux2010-x86_64-cp-38-cp38 -r requirements.txt -o uploader_core.pex --index http://files.pythonhosted.org
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Resolving for:
pex: Hashing pex
pex: Hashing pex: 50.5ms
pex: Isolating pex: 0.1ms
pex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pyarrow>=4.0.0\n', processed_text='pyarrow>=4.0.0', source='/requirements.txt', start_line=3, end_line=3), requirement=Requirement.parse('pyarrow>=4.0.0'), editablepex: Building pex :: Resolving distributions (['requirements.txt']) :: Resolving requirements. :: Installing:
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c898440282606f7a601')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl', fingerprint='59b1b3a0fbea45fd6439ce2472b522d81f974f1a')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl in /root/.pex/installed_wheels/3005ff67df93ee276fb8631e17c677df852254ad/python_dateutil-2.8.1-py2.py3-none-any.whl                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          

pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl in /root/.pex/installed_wheels/853375618e08d469a2114c898440282606f7a601/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl
pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl in /root/.pex/installed_wheels/59b1b3a0fbea45fd6439ce2472b522d81f974f1a/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl
pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl in /root/.pex/installed_wheels/f9497f670630fa14a1d41bd5fee325e076f7cf38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl in /root/.pex/installed_wheels/79e6f2e4f9e24898f1896df379871b9c9922f147/six-1.16.0-py2.py3-none-any.whl
pex: Installing /tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl in /root/.pex/installed_wheels/b97c6966fb8fd273f0bc8533cf0dee4978b9f373/pytz-2021.1-py2.py3-none-any.whl
pex: Building pex: 8401.7ms
pex:   Resolving distributions (['requirements.txt']): 8401.0ms
pex:     Resolving requirements.: 8054.6ms
pex:       Resolving for:
  DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')): 6223.2ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='pandas>=1.2.4\n', processed_text='pandas>=1.2.4', source='/requirements.txt', start_line=1, end_line=1), requirement=Requirement.parse('pandas>=1.2.4'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='numpy>=1.16.5\n', processed_text='numpy>=1.16.5', source='/requirements.txt', start_line=2, end_line=2), requirement=Requirement.parse('numpy>=1.16.5'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='pyarrow>=4.0.0\n', processed_text='pyarrow>=4.0.0', source='/requirements.txt', start_line=3, end_line=3), requirement=Requirement.parse('pyarrow>=4.0.0'), editable=False): 0.2ms
pex:       Installing:
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/python_dateutil-2.8.1-py2.py3-none-any.whl', fingerprint='3005ff67df93ee276fb8631e17c677df852254ad')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl', fingerprint='853375618e08d469a2114c898440282606f7a601')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl', fingerprint='59b1b3a0fbea45fd6439ce2472b522d81f974f1a')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl', fingerprint='f9497f670630fa14a1d41bd5fee325e076f7cf38')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/six-1.16.0-py2.py3-none-any.whl', fingerprint='79e6f2e4f9e24898f1896df379871b9c9922f147')
  InstallRequest(target=DistributionTarget(platform=Platform(platform='manylinux2010_x86_64', impl='cp', version='38', abi='cp38')), wheel_path='/tmp/tmpi78s1iap/manylinux2010_x86_64-cp-38-cp38/pytz-2021.1-py2.py3-none-any.whl', fingerprint='b97c6966fb8fd273f0bc8533cf0dee4978b9f373'): 1739.0ms
Saving PEX file to uploader_core.pex
pex: Zipping PEX file.: 7305.1ms

Notice how the pyarrow-4.0.1-cp38-cp38-manylinux2014_x86_64.whl wheel is downloaded even though a manylinux2010 target is specified.

Let me know if you are able to reproduce the issue on docker using my commands.

P.S.: I'm using rootless docker but it shouldn't matter

I don't think its relevant, but is it correct to say that that setup.py is technically lying?

Yes, you are right, I will look more into it to avoid having to use requirements.txt.

jsirois · 2021-06-18T22:12:33Z

Thanks @Nimayer that was what I needed to repro and figure this out. The specific circumstance in-play is when the interpreter you're using to run Pex happens to support the platform string you've written. Since that interpreter is a CPython 3.8 interpreter on a manylinux2014 host, it does, in fact, support the full set of CPython 3.8 manylinux2010 tags. When Pex sees that situation, it just does a normal resolve using the current interpreter instead of the strict platform you indicated. That's a bug and it's easy enough to fix.

Previously, if a `--platform` resolve was being executed with an interpreter that matched the platform, a regular full-featured interpreter resolve was performed. This prevented, for example, resolving using a CPython 3.8 interpreter on a manylinux2014 capable host for deployment to a host that is only manylinux2010 capable. The `--resolve-local-platforms` option exists to force this sort of fallback to a full-featured interpreter resolve when that is desired. Fixes pex-tool#1355

Previously, if a `--platform` resolve was being executed with an interpreter that matched the platform, a regular full-featured interpreter resolve was performed. This prevented, for example, resolving using a CPython 3.8 interpreter on a manylinux2014 capable host for deployment to a host that is only manylinux2010 capable. The `--resolve-local-platforms` option exists to force this sort of fallback to a full-featured interpreter resolve when that is desired. Fixes #1355

jsirois self-assigned this Jun 10, 2021

jsirois added the in progress label Jun 10, 2021

jsirois mentioned this issue Jun 18, 2021

Release 2.1.43 #1340

Closed

9 tasks

jsirois mentioned this issue Jun 19, 2021

Allow --platform resolves for current interpreter. #1364

Merged

jsirois closed this as completed in #1364 Jun 19, 2021

jsirois removed the in progress label May 15, 2022

jsirois mentioned this issue Mar 1, 2023

Attempt "cross-builds" of sdists for foreign platforms. #2075

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using `--platform manylinux2010` includes pyarrow wheel for `manylinux2014` #1355

Using `--platform manylinux2010` includes pyarrow wheel for `manylinux2014` #1355

Nimayer commented Jun 4, 2021

jsirois commented Jun 10, 2021

jsirois commented Jun 10, 2021

Nimayer commented Jun 11, 2021

jsirois commented Jun 11, 2021 •

edited

Nimayer commented Jun 18, 2021

jsirois commented Jun 18, 2021

Using --platform manylinux2010 includes pyarrow wheel for manylinux2014 #1355

Using --platform manylinux2010 includes pyarrow wheel for manylinux2014 #1355

Comments

Nimayer commented Jun 4, 2021

jsirois commented Jun 10, 2021

jsirois commented Jun 10, 2021

Nimayer commented Jun 11, 2021

jsirois commented Jun 11, 2021 • edited

Nimayer commented Jun 18, 2021

jsirois commented Jun 18, 2021

Using `--platform manylinux2010` includes pyarrow wheel for `manylinux2014` #1355

Using `--platform manylinux2010` includes pyarrow wheel for `manylinux2014` #1355

jsirois commented Jun 11, 2021 •

edited