Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Upload packages to PyPI directly, for pip 1.5 #2715

Closed
rgbkrk opened this Issue Jan 10, 2014 · 26 comments

Comments

Projects
None yet
6 participants

rgbkrk commented Jan 10, 2014

As of pip 1.5 (released January 1, 2014), pip will no longer install externally hosted files by default.

If a user tries to pip install matplotlib it will default to the most recent package hosted on PyPI (1.1.1).

Pinning the dependency to install matplotlib 1.3.1 results in a helpful error message

$ pip install matplotlib==1.3.1
Downloading/unpacking matplotlib==1.3.1
  Could not find a version that satisfies the requirement matplotlib==1.3.1 (from versions: 0.86.1, 0.86.2, 0.86, 0.91.0, 0.91.1, 1.0.1, 1.1.0, 1.1.1)
  Some externally hosted files were ignored (use --allow-external to allow).
Cleaning up...
No distributions matching the version for matplotlib==1.3.1
Storing debug log for failure in /home/vagrant/.pip/pip.log

Users can work around this by either specifying to allow external and unverified,

$ pip install matplotlib==1.3.1 --allow-external matplotlib --allow-unverified matplotlib
Downloading/unpacking matplotlib==1.3.1
  matplotlib an externally hosted file and may be unreliable
  matplotlib is potentially insecure and unverifiable.
  Downloading matplotlib-1.3.1.tar.gz (42.2MB): 42.2MB downloaded
...

or by installing straight from the package for their respective system (tarball for *nix shown below)

$ pip install https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.3.1/matplotlib-1.3.1.tar.gz

It would be really nice to have this package available for pip. It seems that in the past this was a filesize limit restriction workaround on PyPI, which can now handle bigger packages.

If the filesize is still an issue, perhaps we can work with the PyPI folks to either increase the limit or put matplotlib on a whitelist.

Owner

mdboom commented Jan 13, 2014

It seems matplotlib is still too big. Uploading through the web interface gives:

Error processing form

distribution file too large

python setup.py sdist upload gives:

Submitting dist/matplotlib-1.3.1.tar.gz to http://pypi.python.org/pypi
Upload failed (400): distribution file too large

Any help you can provide to make this work would be great.

Owner

tacaswell commented Jan 13, 2014

Does the tarball strip out the testing images?

dstufft commented Jan 13, 2014

So the (obvious) two solutions are strip things out of the tarball to make it smaller, or bump up the file size. Can you tell me what the file size you tried to upload?

Owner

mdboom commented Jan 13, 2014

No, the tarball does not strip out the testing images. I'm not sure we want to do that except as a last resort.

The source tarball is ~42MB.

dstufft commented Jan 13, 2014

Ok, that's about 2MB over the current limit. The limit is pretty arbitrary so we can bump that up a bit. I'll comment on this ticket once I get time to do that (should be sometime today).

rgbkrk commented Jan 13, 2014

It's primarily the lib/matplotlib/tests/baseline_images I assume that are making this so big?

Is the purpose here so people can test their installation (or development setup)? If people use matplotlib in production settings where they repeatedly deploy it, would it make sense to make a version that doesn't bundle all this in?

Owner

mdboom commented Jan 15, 2014

If I wanted to provide packages both with and without the test data, how would you recommend doing that on pypi? Have two separate package entries, i.e. use the existing matplotlib for without tests, and matplotlib-tests for the tests (which installs into a namespace package, I guess...?)

I don't want to change this for a bugfix release, but rather wait for our next major release (which is scheduled for "pretty soon now").

Owner

mdboom commented Jan 15, 2014

And in the interest of not changing things -- I should probably go and upload to PyPI the last few years worth of releases exactly as they were on Sourceforge -- it's actually pretty typical with matplotlib for users to specify old versions, so I'd like that to work without the --allow-unverified flags, too. This implies bumping up the file size limit, as I don't see another way around that.

rgbkrk commented Jan 21, 2014

Installing matplotlib-tests into a namespace package sounds good.

I've never used it, but I imagine the syntax in setup.py would look something like:

 entry_points={
        "matplotlib": [
            "tests = matplotlib-tests"
        ]
 }

I summon @ivanov for more expertise than myself (we discussed this in person).

cbrnr commented Feb 7, 2014

I +1 the idea to bump up the file limit (e.g. 50MB should be sufficient). I just recently ran into this problem where only an old version was downloaded. I can't imagine I'm the only one trying to "pip install matplotlib", so it's definitely not good that this doesn't work (especially since I'm using Python 3, where >= 1.2.0 is required).

Owner

mdboom commented Feb 7, 2014

@dstufft: What's the status on the file size limit? I think I'd prefer that solution (for now), so as to not change our installation procedure in a significant way in a bug fix release. Is there a PyPI bug tracker entry (or some such) I could follow?

cbrnr commented Feb 7, 2014

I didn't find a dedicated issue there, but I reported this problem at https://sourceforge.net/p/pypi/support-requests/342/ and pointed out the file size limitation issue.

dstufft commented Feb 7, 2014

I'm a terrible person and forgot all about this, let me do it right now.

dstufft commented Feb 7, 2014

Can you try to upload to TestPyPI and see if it works there? If it does I'll deploy it out to PyPI itself.

Owner

mdboom commented Feb 7, 2014

I get:

Creating tar archive
removing 'matplotlib-1.3.1' (and everything under it)
running upload
Submitting dist/matplotlib-1.3.1.tar.gz to http://testpypi.python.org/pypi
error: None

How would I further debug this?

dstufft commented Feb 7, 2014

o.O

Uhhh, can you try using Twine to upload instead? https://pypi.python.org/pypi/twine/

Owner

mdboom commented Feb 7, 2014

How do I tell twine to go to testpypi?

› twine upload -r http://testpypi.python.org/pypi dist/*                                                                                                                 1 « git · cfb6ac64578c · (unknown) »
Traceback (most recent call last):
  File "/home/mdboom/python/bin/twine", line 11, in <module>
    sys.exit(main())
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/__main__.py", line 24, in main
    return app.cli(sys.argv[1:])
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/application.py", line 97, in cli
    (k, v) for k, v in args._get_kwargs()
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/commands/upload.py", line 69, in __call__
    parsed = urlparse(config["repository"])
KeyError: 'repository'

dstufft commented Feb 7, 2014

Make sure testpypi is in your ~/.pypirc and use -r testpypi like:

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
username:dstufft
password:whatever

[testpypi]
repository:https://testpypi.python.org/pypi
username:dstufft
password:whatever

dstufft commented Feb 7, 2014

FWIW i'm dstufft on freenode if you want to do this in a more immediate response format :)

Owner

mdboom commented Feb 7, 2014

Uploading distributions to https://testpypi.python.org/pypi
Uploading matplotlib-1.3.1.tar.gz
No handlers could be found for logger "requests.packages.urllib3.connectionpool"
Traceback (most recent call last):
  File "/home/mdboom/python/bin/twine", line 11, in <module>
    sys.exit(main())
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/__main__.py", line 24, in main
    return app.cli(sys.argv[1:])
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/application.py", line 97, in cli
    (k, v) for k, v in args._get_kwargs()
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/commands/upload.py", line 177, in __call__
    resp.raise_for_status()
  File "/home/mdboom/python/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway

dstufft commented Feb 7, 2014

Ah ha, Hmm

Let me go dig through the logs. Did you generate it jsut with python setup.py sdist from this repo?

dstufft commented Feb 8, 2014

I believe this should be fixed now on PyPI and TestPyPI

Owner

mdboom commented Feb 8, 2014

Success (on TestPyPI).

Owner

mdboom commented Feb 8, 2014

On PyPI proper, it seems it still has the file size limit:

requests.exceptions.HTTPError: 400 Client Error: distribution file too large

Don't know if that's still to be expected. Once that's resolved, I'll upload this, and this bug can be closed...

Thanks for all of your help.

ewdurbin commented Feb 8, 2014

@mdboom just verified the updated limit is in place on PyPI, would you mind trying again?

Owner

mdboom commented Feb 9, 2014

Success! Thanks for everyone's help. Closing.

@mdboom mdboom closed this Feb 9, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment