Using pip download to fetch package sources seems to trigger building wheels for some packages.

[tvalentyn]

Environment

• pip version: 20.1.1
• Python version: 3.7
• OS: Debian Linux

Description

In numpy/numpy#14053 we observed that

python3 -m pip download --dest /tmp numpy==1.18.0 --no-binary :all: --no-use-pep517

finishes immediately, while:

python3 -m pip download --dest /tmp numpy==1.18.0 --no-binary :all:

takes a while to complete.

We originally noticed the slowdown in https://issues.apache.org/jira/browse/ARROW-8983.
The command python -m pip download --dest /tmp pyarrow==0.17.0 --no-binary :all: also gets stuck for a few minutes with "Installing build dependencies ... ", and increased CPU usage when pip downloads sources of numpy, a dependency of pyarrow. Interestingly, in case of pyarrow, adding --no-use-pep517 does not help.

Is there an inefficiency in pip download, or it's a known behavior and is considered WAI?

Thank you.

[sbidoul]

I made a quick test in verbose mode, and it appears that part of the slowdown comes from --no-binary :all: which extends to build dependencies. Since build dependencies include cython, pip first builds cython from source before attempting to obtain metadata.

python3 -m pip download --dest /tmp numpy==1.18.0 --no-binary numpy (or --no-binary :all: --only-binary cython) is faster because it installs cython from binary wheel.

Nevertheless, that still involves some C compilation which does not occur with --no-use-pep517. I'm not sure why yet.

There is ongoing discussion on how/whether --no-binary should apply to build dependencies. I don't think that has been conclusive yet.

[jorisvandenbossche]

Naive question: why is it needed to build numpy or create its build environment when you only want to download the source code? Is it to get some metadata that is only available from a wheel?

[pfmoore]

Is it to get some metadata that is only available from a wheel?

Precisely that.

[jorisvandenbossche]

And how do you get that metadata in case of --no-use-pep517 ?

[sbidoul]

And how do you get that metadata in case of --no-use-pep517 ?

setup.py egg_info

[sbidoul]

I looked into this a bit further and it appears that setuptools' implementation of PEP 517 prepare_metadata_for_build_wheel() uses setup.py dist_info. In the case of numpy 1.18.0, this involves cython and some compilation (of _configtest.c).

setup.py egg_info runs instantly.

That explains the performance difference, although I can't explain myself why dist_info and egg_info behave differently, the resulting metadata (METADATA vs PKG-INFO) being very similar.

[uranusjr]

Numpy has a highly customised setup.py that triggers said compilation, unless explicitly excluded. egg_info is one of those commands.

See parse_setuppy_commands() here: https://github.com/numpy/numpy/blob/v1.18.0/setup.py

[zeha]

Why does downloading an sdist need metadata from a wheel in the first place?

[zeha]

Appears to be related to (duplicate of?) #7995.

[uranusjr]

Closing this in favour of #7995.