New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`--download` fails when using local `--find-links` #1111

Closed
qwcode opened this Issue Aug 6, 2013 · 6 comments

Comments

Projects
None yet
3 participants
@qwcode
Contributor

qwcode commented Aug 6, 2013

As discovered in #1107, --download fails to work correctly with local --find-links

you get no error, but you get the contents of the archive dumped into the download dir, and not the archive.

pip.download.unpack_file_url has no download_dir logic like pip.download.unpack_http_url does.

Maybe we shouldn't support this, and just raise a command error saying the options are incompatible. It's odd in the first place to want to download from a local store.

@dhandy2013

This comment has been minimized.

Show comment
Hide comment
@dhandy2013

dhandy2013 Sep 26, 2013

+1 for supporting --download with local --find-links.

Here is my actual use case: I want to download all of the requirements of a Python application using a pip command and the requirements.txt for that application. Some of the requirements are available on PyPI, but others are available locally, in a sub-directory named "local".

Contents of my requirements.txt:

python-novaclient>=2.13
python-glanceclient>=0.9
acpython>=1.3

Contents of my "local" sub-directory:

$ ls local
acpython-1.3.tar.gz

(acpython-1.3.tar.gz is a source tarball created using "python setup.py sdist".)

pip command:

$ pip install -q --download=./download --find-links=file://`pwd`/local -r requirements.txt
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_adaptive/acpython
Storing complete log in /home/adaptive/.pip/pip.log

Afterwards, the download directory contains the downloaded python-glanceclient and python-novaclient .tar.gz files, but also the extracted contents of acpython, which is not helpful.

dhandy2013 commented Sep 26, 2013

+1 for supporting --download with local --find-links.

Here is my actual use case: I want to download all of the requirements of a Python application using a pip command and the requirements.txt for that application. Some of the requirements are available on PyPI, but others are available locally, in a sub-directory named "local".

Contents of my requirements.txt:

python-novaclient>=2.13
python-glanceclient>=0.9
acpython>=1.3

Contents of my "local" sub-directory:

$ ls local
acpython-1.3.tar.gz

(acpython-1.3.tar.gz is a source tarball created using "python setup.py sdist".)

pip command:

$ pip install -q --download=./download --find-links=file://`pwd`/local -r requirements.txt
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_adaptive/acpython
Storing complete log in /home/adaptive/.pip/pip.log

Afterwards, the download directory contains the downloaded python-glanceclient and python-novaclient .tar.gz files, but also the extracted contents of acpython, which is not helpful.

@dhandy2013

This comment has been minimized.

Show comment
Hide comment
@dhandy2013

dhandy2013 Sep 26, 2013

Here is the result of my investigation. I hope this is helpful.

Pip unpacks every package and looks at its egg info, even when you are just trying to download. When you are only downloading, pip creates a temporary directory for each package, into which it tries to unpack the downloaded package, so that it can look at its egg info. So far, so good.

The problem is that the unpack method behaves differently depending on whether the package was downloaded from a file url or from an http url. See the unpack_url() method:

Pip 1.4.1
pip/req.py line 1225

    def unpack_url(self, link, location, only_download=False):
        if only_download:
            loc = self.download_dir
        else:
            loc = location
        if is_vcs_url(link):
            return unpack_vcs_link(link, loc, only_download)
        # a local file:// index could have links with hashes
        elif not link.hash and is_file_url(link):
            return unpack_file_url(link, loc)
        else:
            if self.download_cache:
                self.download_cache = os.path.expanduser(self.download_cache)
            retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
            if only_download:
                write_delete_marker_file(location)
            return retval

In my case, this method is called with:

  • link = a Link object pointing to the file URL of a source package found in the find-links directory
  • location = the path to the download directory
  • only_download = True

For some reason, if only_download is true, unpack_url() chooses to ignore the location parameter, and the file is unpacked into the download directory instead of into the temp directory that pip just created.

pip fails after it returns from this method, because the later code that gets the egg info expects the unpacked contents to be in the temp directory, not in the download directory.

dhandy2013 commented Sep 26, 2013

Here is the result of my investigation. I hope this is helpful.

Pip unpacks every package and looks at its egg info, even when you are just trying to download. When you are only downloading, pip creates a temporary directory for each package, into which it tries to unpack the downloaded package, so that it can look at its egg info. So far, so good.

The problem is that the unpack method behaves differently depending on whether the package was downloaded from a file url or from an http url. See the unpack_url() method:

Pip 1.4.1
pip/req.py line 1225

    def unpack_url(self, link, location, only_download=False):
        if only_download:
            loc = self.download_dir
        else:
            loc = location
        if is_vcs_url(link):
            return unpack_vcs_link(link, loc, only_download)
        # a local file:// index could have links with hashes
        elif not link.hash and is_file_url(link):
            return unpack_file_url(link, loc)
        else:
            if self.download_cache:
                self.download_cache = os.path.expanduser(self.download_cache)
            retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
            if only_download:
                write_delete_marker_file(location)
            return retval

In my case, this method is called with:

  • link = a Link object pointing to the file URL of a source package found in the find-links directory
  • location = the path to the download directory
  • only_download = True

For some reason, if only_download is true, unpack_url() chooses to ignore the location parameter, and the file is unpacked into the download directory instead of into the temp directory that pip just created.

pip fails after it returns from this method, because the later code that gets the egg info expects the unpacked contents to be in the temp directory, not in the download directory.

@wolever

This comment has been minimized.

Show comment
Hide comment
@wolever

wolever Dec 15, 2013

Bump this, as it's also causing issues with pip2pi, which recently switched to using pip install --download instead of pip bundle to download packages. pip bundle behaved correctly, but it appears that pip install --download does not.

wolever commented Dec 15, 2013

Bump this, as it's also causing issues with pip2pi, which recently switched to using pip install --download instead of pip bundle to download packages. pip bundle behaved correctly, but it appears that pip install --download does not.

@wolever

This comment has been minimized.

Show comment
Hide comment
@wolever

wolever Dec 15, 2013

Also, to address the question of use-cases: this is important for pip2pi because a local tarball may have dependencies, and pip install --download output/ mypkg.tar.gz should put those dependencies as well as the mypkg.tar.gz file, into the output/ directory. Additionally, it's useful to be able to run pip2pi pypi.example.com: mypkg.tar.gz.

wolever commented Dec 15, 2013

Also, to address the question of use-cases: this is important for pip2pi because a local tarball may have dependencies, and pip install --download output/ mypkg.tar.gz should put those dependencies as well as the mypkg.tar.gz file, into the output/ directory. Additionally, it's useful to be able to run pip2pi pypi.example.com: mypkg.tar.gz.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Feb 2, 2014

Contributor

closing due to merge of #1524

@wolever regarding your comment (#1111 (comment)), that case should work (in develop branch) for sdists (*.tar.gz). wheels currently have a bug though (#1112)

Contributor

qwcode commented Feb 2, 2014

closing due to merge of #1524

@wolever regarding your comment (#1111 (comment)), that case should work (in develop branch) for sdists (*.tar.gz). wheels currently have a bug though (#1112)

@qwcode qwcode closed this Feb 2, 2014

@wolever

This comment has been minimized.

Show comment
Hide comment
@wolever

wolever Feb 3, 2014

Awesome, this does look like it addresses the issue. Thanks!

wolever commented Feb 3, 2014

Awesome, this does look like it addresses the issue. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment