pip download with --no-binary flags for projects with alternative pep 517 build systems breaks under some circumstances

[techalchemy]

Environment

• pip version: 18.0
• Python version: 3.7.0
• OS: Ubuntu 18.0.4

I am in the middle of cutting a pipenv release and I am running the vendoring tooling I stole borrowed from you guys. This tooling is run via pipenv run invoke vendoring.update which essentially wraps the command in a virtualenv.

Description
The failure is happening when downloading licenses, specifically for ptyprocess==0.6.0. The command in question was simply to download the sdist to a target without dependencies. Note that this is functional when disabling build isolation.

This is a ReqTracker related error, which indicates:

LookupError: https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4
c27b0eb6695d846 (from https://pypi.org/simple/requests-download/) is already being built: requests_download from https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24
f83bd614/requests_download-0.1.2.tar.gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4c27b0eb6695d846 (from flit)

Expected behavior
1 sdist in the target directory sans errors

How to Reproduce

    $ pip download --no-binary :all: --only-binary none --no-deps -d ./__tmp__ ptyprocess==0.6.0

Output

(pipenv)  ³ pipenv-MfOPs1lW  ~/g/pipenv   add-vistir     pip download --no-binary :all: --only-binary none --no-deps -d ./__tmp__ ptyprocess==0.6.0
Collecting ptyprocess==0.6.0                                                                                                               
  Using cached https://files.pythonhosted.org/packages/7d/2d/e4b8733cf79b7309d84c9081a4ab558c89d8c89da5961bf4ddb050ca1ce0/ptyprocess-0.6.0.tar.gz                                                                  
  Saved ./__tmp__/ptyprocess-0.6.0.tar.gz                                                                                           
  Missing build requirements in pyproject.toml for ptyprocess==0.6.0 from https://files.pythonhosted.org/packages/7d/2d/e4b8733cf79b7309d84c9081a4ab558c89d8c89da5961bf4ddb050ca1ce0/ptyprocess-0.6.0.tar.gz#sha256
=923f299cc5ad920c68f2bc0bc98b75b9f838b93b599941a6b63ddbc2476394c0.                                                                            
  This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
  Installing build dependencies ... error                                                                                                                    
  Complete output from command /home/hawk/.virtualenvs/pipenv-MfOPs1lW/bin/python3.7 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-gz0kd7at --no-warn-script-location --no-binary :all: -
-only-binary none -i https://pypi.org/simple -- flit:                                            
  Collecting flit                                                                                                                               
    Using cached https://files.pythonhosted.org/packages/1b/bd/be3cbf0b837dea22ae45bf371879e9caebc6ee064e40f7d7fac7a4c79c45/flit-1.0.tar.gz
    Missing build requirements in pyproject.toml for flit from https://files.pythonhosted.org/packages/1b/bd/be3cbf0b837dea22ae45bf371879e9caebc6ee064e40f7d7fac7a4c79c45/flit-1.0.tar.gz#sha256=95b8577b2232da39ee
14ae237575b7a85afeeabc1e87f4a19485fac34f85aa89.                                                                                                                                                                    
    This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
    Installing build dependencies: started                                                                                                                                                                         
    Installing build dependencies: finished with status 'done'                                                                                                                                                     
  Collecting requests (from flit)                                                                                          
    Using cached https://files.pythonhosted.org/packages/54/1f/782a5734931ddf2e1494e4cd615a51ff98e1879cbe9eecbdfeaf09aa75e9/requests-2.19.1.tar.gz                                                                 
  Collecting docutils (from flit)                        
    Using cached https://files.pythonhosted.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-0.14.tar.gz                                                                   
  Collecting requests_download (from flit)                                                                                                   
    Using cached https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.gz                                                         
    Missing build requirements in pyproject.toml for requests_download from https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.
gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4c27b0eb6695d846 (from flit).                                                                                                                            
    This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
    Installing build dependencies: started                                                                                                                                                                         
    Installing build dependencies: finished with status 'error'
    Complete output from command /home/hawk/.virtualenvs/pipenv-MfOPs1lW/bin/python3.7 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qg0tvxfz --no-warn-script-location --no-binary :all:
 --only-binary none -i https://pypi.org/simple -- flit:
    Collecting flit                                                                                                                               
      Using cached https://files.pythonhosted.org/packages/1b/bd/be3cbf0b837dea22ae45bf371879e9caebc6ee064e40f7d7fac7a4c79c45/flit-1.0.tar.gz
      Missing build requirements in pyproject.toml for flit from https://files.pythonhosted.org/packages/1b/bd/be3cbf0b837dea22ae45bf371879e9caebc6ee064e40f7d7fac7a4c79c45/flit-1.0.tar.gz#sha256=95b8577b2232da39
ee14ae237575b7a85afeeabc1e87f4a19485fac34f85aa89.
      This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
    Collecting requests (from flit)
      Using cached https://files.pythonhosted.org/packages/54/1f/782a5734931ddf2e1494e4cd615a51ff98e1879cbe9eecbdfeaf09aa75e9/requests-2.19.1.tar.gz
    Collecting docutils (from flit)
      Using cached https://files.pythonhosted.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-0.14.tar.gz
    Collecting requests_download (from flit)
      Using cached https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.gz
    Exception:
    Traceback (most recent call last):
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/basecommand.py", line 141, in main
        status = self.run(options, args)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 299, in run
        resolver.resolve(requirement_set)                                                                                                                                                                          
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/resolve.py", line 102, in resolve 
        self._resolve_one(requirement_set, req)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
        self.require_hashes
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 297, in prepare_linked_requirement
        with self.req_tracker.track(req):
      File "/home/hawk/.pyenv/versions/3.7.0/lib/python3.7/contextlib.py", line 112, in __enter__
        return next(self.gen)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 74, in track
        self.add(req)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 46, in add
        % (link, fp.read()))
    LookupError: https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4
c27b0eb6695d846 (from https://pypi.org/simple/requests-download/) is already being built: requests_download from https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24
f83bd614/requests_download-0.1.2.tar.gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4c27b0eb6695d846 (from flit)

    ----------------------------------------
  Command "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/bin/python3.7 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qg0tvxfz --no-warn-script-location --no-binary :all: --only-binary none -i
 https://pypi.org/simple -- flit" failed with error code 2 in None

[benoit-pierre]

pip needs to prepare each source requirement (run its setup.py, note the warning about PEP 517 not being supported) to get its install requirements.

But in this case, there's a circular dependencies in the build/install requirements chain: ptyprocess list flit as a build requirement, but flit depends on requests_download, and requests_download list flit as build requirements too!

Note that this works fine if wheel installs are allowed at least for requests_download: pip download -v --no-binary :all: --only-binary requests_download ptyprocess.

[techalchemy]

@benoit-pierre yay, that makes good sense to me; would you consider this working as intended then? I think you did the implementation of the requirement tracker which is a nice idea and a good step toward something I think everyone wants, but the element that was not intuitive to me was the specific cause of the issue (I probably could have guessed, but I just installed flit directly myself and moved on).

I'm sure there are lots of reasons why LookupErrors are actually thrown and in isolation it's kind of difficult to handle this, but it seems like there is some core conceptual breach going on here which permits a build system to depend on something that depends on it -- from a UX perspective it would be ideal if we could minimally flag that somehow (and possibly offer this type of alternative?)

Longer term the ideal case would be to not allow build systems to depend on packages that depend on them and thus avoid circular dependencies in the first place (this seems like a good case for vendoring in flit, but it also feels like enforcement might be possible here as well)

[benoit-pierre]

IMHO it works as expected yes.

from a UX perspective it would be ideal if we could minimally flag that somehow

I guess the exception / error message could be tweaked.

(and possibly offer this type of alternative?)

That's going to be hard.

Longer term the ideal case would be to not allow build systems to depend on packages that depend on them and thus avoid circular dependencies in the first place.

Yeah, any build system that have dependencies relying on it will be trouble...

(this seems like a good case for vendoring in flit, but it also feels like enforcement might be possible here as well)

How would you enforce it?

[techalchemy]

It feels like we are back in setup.py territory having to execute everything in order to determine dependencies -- is there no way to do a pre-build check do say:

1. What build system does this tool require?
2. Download those (assuming they are not installed/we are in an isolated environment)
3. What does building that build system require?
4. Download those,
5. What does it require to build those requirements?
6. Does a requirement depend on the build system ? If so, then we have a circular dependency

Obviously this can be done recursively and it can get pretty complex, but I'm not 100% sure that would cause a problem (I guess it probably would?) I'm not sure that situation (nested cyclic dependencies for build systems) is ever occurring currently but it may in the future. The simplest bet would just be to handle the exceptional cases.

(and possibly offer this type of alternative?)

That's going to be hard.

For cases like mine where the user is specifying --no-binary :all: the simple answer is to just suggest using a binary alternative such as wheel, if possible.

[benoit-pierre]

There's no way around it... Even with a proper source distribution, there's not enough metadata to determine dependencies. Not that it matters anyway, since pip support things like VCS installs. So it needs to run setup.py egg_info to generate enough metadata (or whatever the PEP 517 equivalent is). And to do that, build dependencies must be installed. Which yields to...

Obviously this can be done recursively and it can get pretty complex, but I'm not 100% sure that would cause a problem (I guess it probably would?) I'm not sure that situation (nested cyclic dependencies for build systems) is ever occurring currently but it may in the future. The simplest bet would just be to handle the exceptional cases.

The possibility of nested cyclic dependencies! That's exactly what is happening in this issue and what the requirement tracker is for.

For cases like mine where the user is specifying --no-binary :all: the simple answer is to just suggest using a binary alternative such as wheel, if possible.

How do you know the issue is with not allowing wheels? How do you know if wheels are even available? I find this kind of things (suggesting a "magic" command that will fix the situation) is a bad idea: not only is it impossible to find one that works in all cases, but the end results is invariably users blindly executing it (look at the mess with the "you should upgrade pip" message), so you're only moving the goal post further...

[techalchemy]

Hah, ok. I guess I am persuaded, I didn't consider that wheels are built for VCS requirements as well now; we are working on a library to handle a lot of that during dependency resolution and I believe we simply raise a resolution error if we encounter cyclic dependencies (we built it on an acyclic graph)

I don't really know how else this might come up or where the error needs to be raised, but once we have a better understanding of the scope of how this error might happen, from my POV an ideal message might be: BuildError: Cycilc build system dependencies found between requirements: requests_download (from flit) and flit

I realize it's likely not quite that simple, given the variables, but anything that provides that kind of info would be super helpful

That said this is likely low priority, I'm sure it's not that common

[takluyver]

I just came across this - FWIW, I plan to drop requests_download as a dependency from flit once pip supports PEP 517 enough that you can pip install a Github repo of a project built with flit. It's only there at the moment to provide the flit installfrom command to make up for this. I'm looking forward to dropping that.

I've tried to make sure that all the other dependencies of flit use setup.py packaging, to avoid precisely this kind of circular dependency issue.

[adamtheturtle]

This is a problem for Homebrew recipes of Python projects which use virtualenv_install_with_resources - this specifies --no-binary :all:.

[takluyver]

As of Flit 1.2.1, requests_download is now an optional dependency (pip install flit[installfrom]). That should avoid this problem, because installing Flit as a build dependency will not attempt to install requests_download. AFAIK all of Flit's real dependencies are built by setup.py.

[bcroq]

Same problem with tomlkit/poetry:

$ pip download -d . --no-binary :all: tomlkit
...
LookupError: https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from https://pypi.org/simple/tomlkit/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: tomlkit from https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2

$ pip download -d . --no-binary :all: poetry
...
LookupError: https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from https://pypi.org/simple/tomlkit/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: tomlkit<0.6.0,>=0.5.1 from https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from poetry)

[reversefold]

Should --no-binary apply to build-time dependencies? Since pip only needs the build-time dependencies during the run of pip in order to look up dependencies in the source package it seems like it would be fine for it to install the build-time dependencies any way that works.

[reversefold]

After further thinking on this I also realized that when I was having this error I was actually running pip in an environment where the build-time dependencies were already installed. Why would pip need to re-download and re-manage the dependencies of poetry, for built-time-only use, when they're already installed in the same environment pip is being run from?

The more I think about this the more it makes sense that pip should be able to use already-installed packages when it comes to build-time requirements.

[takluyver]

There's a distinction here between reusing build dependencies that are already built and those that are already installed. Reusing dependencies that are already installed means building in an existing environment, which is what we're trying to get away from. But if the dependencies are already cached as wheels locally, setting up a new environment with them should be fast - it doesn't need to download or build them again.

[reversefold]

I can understand that but there is an inherent flaw, as evidenced by this ticket. If --no-binary :all: is used then build-time dependencies cannot be resolved if the required builder depends in any way, either directly or transitively, on its own build system. It's a chicken and egg problem.

It seems like there needs to be some way of relaxing the --no-binary :all: restriction so that it doesn't apply to build-time dependencies so that this doesn't happen. I believe that usage of --no-binary :all: is generally meant for dependencies of package being built, not the build-time dependencies which won't be packaged or installed in the same was as the package asked for.

Perhaps --no-binary :all: could be relaxed to not apply to build dependencies and another flag could be introduced to revert that behavior if it's really wanted and the caller knows what they're doing? Or maybe another value for --no-binary which means "everything but build dependencies"?

[JBKahn]

I'm trying to better understand the cause of the issue, and what a resolution might be for pip/its users. Is the idea that the current implementation means build systems shouldn't have external dependancies that are un-vendored?

I think that relying on dependancies to use a different build system than those that depend on them would be a bit of a nightmare in terms of management of dependancies further into the tree.

I'm not involved in the core python area but if that is the direction we're going, should that be part of the PEP for alternative build systems? I assume there is a close relationship/overlap with the core team and the pypa which is why I'm asking this here. If that's not true, I can post that question in a more appropriate place.

[pfmoore]

The root cause here is that if you insist on not using binaries (--no-binary :all:) then you need to be able to build your build system. Which is, let's say, an "interesting" recursive problem ;-)

The support added to PEP 517 for self-hosting backends was designed to address this situation, but I don't know if flit has adopted it yet (I can't actually recall offhand if pip has added that support yet).

A better solution in this case might be to allow binaries for flit, so you don't get into the loop of using flit to build one of its dependencies in the first place. But I guess whether that works for you depends on why you are using --no-binary :all: in the first place.

[takluyver]

Flit doesn't support it yet (I've got a substantial refactor of Flit in the works, trying to find time to complete it).

I think support made it into a released pep517 package, so hopefully all pip needs is to vendor the latest version of that.

[chrahunt]

I don't see any specific actions targeting pip from the above discussion - any issues with closing this?

[JBKahn]

@chrahunt you added "waiting response", did you mean from someone here?

I'm not sure if this is a PIP problem, a PEP problem or none of the above. Just a stance from this library would help. Should a build system used by pip have 0 dependancies? If not, then it can cause circular issues if the build system relies on X which uses the build system for its own build. If it can have dependancies, is there a reasonable way to do that? Is vendoring the only way today?

Depending on the answer to that question, there may or may not be actions targeting pip.

[chrahunt]

Anyone really - just getting confirmation that no one sees a specific action that I missed, then if @techalchemy doesn't respond in some time or another maintainer doesn't remove the label then the bot will close the issue.

I would put those questions to the packaging category on Discourse (and/or the distutils-sig mailing list/pypa/packaging-problems repo) and drop a link here so interested parties can provide feedback. They are a bit higher-level than pip itself, and those resources would help expose the discussion to a wider audience who can help define best practices which could make their way into the Packaging User Guide for example.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.