Upload packages to PyPI directly, for pip 1.5

[rgbkrk]

As of pip 1.5 (released January 1, 2014), pip will no longer install externally hosted files by default.

If a user tries to pip install matplotlib it will default to the most recent package hosted on PyPI (1.1.1).

Pinning the dependency to install matplotlib 1.3.1 results in a helpful error message

$ pip install matplotlib==1.3.1
Downloading/unpacking matplotlib==1.3.1
  Could not find a version that satisfies the requirement matplotlib==1.3.1 (from versions: 0.86.1, 0.86.2, 0.86, 0.91.0, 0.91.1, 1.0.1, 1.1.0, 1.1.1)
  Some externally hosted files were ignored (use --allow-external to allow).
Cleaning up...
No distributions matching the version for matplotlib==1.3.1
Storing debug log for failure in /home/vagrant/.pip/pip.log

Users can work around this by either specifying to allow external and unverified,

$ pip install matplotlib==1.3.1 --allow-external matplotlib --allow-unverified matplotlib
Downloading/unpacking matplotlib==1.3.1
  matplotlib an externally hosted file and may be unreliable
  matplotlib is potentially insecure and unverifiable.
  Downloading matplotlib-1.3.1.tar.gz (42.2MB): 42.2MB downloaded
...

or by installing straight from the package for their respective system (tarball for *nix shown below)

$ pip install https://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.3.1/matplotlib-1.3.1.tar.gz

It would be really nice to have this package available for pip. It seems that in the past this was a filesize limit restriction workaround on PyPI, which can now handle bigger packages.

If the filesize is still an issue, perhaps we can work with the PyPI folks to either increase the limit or put matplotlib on a whitelist.

[mdboom]

It seems matplotlib is still too big. Uploading through the web interface gives:

Error processing form

distribution file too large

python setup.py sdist upload gives:

Submitting dist/matplotlib-1.3.1.tar.gz to http://pypi.python.org/pypi
Upload failed (400): distribution file too large

Any help you can provide to make this work would be great.

[tacaswell]

Does the tarball strip out the testing images?

[dstufft]

So the (obvious) two solutions are strip things out of the tarball to make it smaller, or bump up the file size. Can you tell me what the file size you tried to upload?

[mdboom]

No, the tarball does not strip out the testing images. I'm not sure we want to do that except as a last resort.

The source tarball is ~42MB.

[dstufft]

Ok, that's about 2MB over the current limit. The limit is pretty arbitrary so we can bump that up a bit. I'll comment on this ticket once I get time to do that (should be sometime today).

[rgbkrk]

It's primarily the lib/matplotlib/tests/baseline_images I assume that are making this so big?

Is the purpose here so people can test their installation (or development setup)? If people use matplotlib in production settings where they repeatedly deploy it, would it make sense to make a version that doesn't bundle all this in?

[mdboom]

If I wanted to provide packages both with and without the test data, how would you recommend doing that on pypi? Have two separate package entries, i.e. use the existing matplotlib for without tests, and matplotlib-tests for the tests (which installs into a namespace package, I guess...?)

I don't want to change this for a bugfix release, but rather wait for our next major release (which is scheduled for "pretty soon now").

[mdboom]

And in the interest of not changing things -- I should probably go and upload to PyPI the last few years worth of releases exactly as they were on Sourceforge -- it's actually pretty typical with matplotlib for users to specify old versions, so I'd like that to work without the --allow-unverified flags, too. This implies bumping up the file size limit, as I don't see another way around that.

[rgbkrk]

Installing matplotlib-tests into a namespace package sounds good.

I've never used it, but I imagine the syntax in setup.py would look something like:

 entry_points={
        "matplotlib": [
            "tests = matplotlib-tests"
        ]
 }

I summon @ivanov for more expertise than myself (we discussed this in person).

[cbrnr]

I +1 the idea to bump up the file limit (e.g. 50MB should be sufficient). I just recently ran into this problem where only an old version was downloaded. I can't imagine I'm the only one trying to "pip install matplotlib", so it's definitely not good that this doesn't work (especially since I'm using Python 3, where >= 1.2.0 is required).

[mdboom]

@dstufft: What's the status on the file size limit? I think I'd prefer that solution (for now), so as to not change our installation procedure in a significant way in a bug fix release. Is there a PyPI bug tracker entry (or some such) I could follow?

[cbrnr]

I didn't find a dedicated issue there, but I reported this problem at https://sourceforge.net/p/pypi/support-requests/342/ and pointed out the file size limitation issue.

[dstufft]

I'm a terrible person and forgot all about this, let me do it right now.

[dstufft]

Can you try to upload to TestPyPI and see if it works there? If it does I'll deploy it out to PyPI itself.

[mdboom]

I get:

Creating tar archive
removing 'matplotlib-1.3.1' (and everything under it)
running upload
Submitting dist/matplotlib-1.3.1.tar.gz to http://testpypi.python.org/pypi
error: None

How would I further debug this?

[dstufft]

o.O

Uhhh, can you try using Twine to upload instead? https://pypi.python.org/pypi/twine/

[mdboom]

How do I tell twine to go to testpypi?

› twine upload -r http://testpypi.python.org/pypi dist/*                                                                                                                 1 « git · cfb6ac64578c · (unknown) »
Traceback (most recent call last):
  File "/home/mdboom/python/bin/twine", line 11, in <module>
    sys.exit(main())
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/__main__.py", line 24, in main
    return app.cli(sys.argv[1:])
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/application.py", line 97, in cli
    (k, v) for k, v in args._get_kwargs()
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/commands/upload.py", line 69, in __call__
    parsed = urlparse(config["repository"])
KeyError: 'repository'

[dstufft]

Make sure testpypi is in your ~/.pypirc and use -r testpypi like:

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
username:dstufft
password:whatever

[testpypi]
repository:https://testpypi.python.org/pypi
username:dstufft
password:whatever

[dstufft]

FWIW i'm dstufft on freenode if you want to do this in a more immediate response format :)

[mdboom]

Uploading distributions to https://testpypi.python.org/pypi
Uploading matplotlib-1.3.1.tar.gz
No handlers could be found for logger "requests.packages.urllib3.connectionpool"
Traceback (most recent call last):
  File "/home/mdboom/python/bin/twine", line 11, in <module>
    sys.exit(main())
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/__main__.py", line 24, in main
    return app.cli(sys.argv[1:])
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/application.py", line 97, in cli
    (k, v) for k, v in args._get_kwargs()
  File "/home/mdboom/python/lib/python2.7/site-packages/twine/commands/upload.py", line 177, in __call__
    resp.raise_for_status()
  File "/home/mdboom/python/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway

[dstufft]

Ah ha, Hmm

Let me go dig through the logs. Did you generate it jsut with python setup.py sdist from this repo?

[dstufft]

I believe this should be fixed now on PyPI and TestPyPI

[mdboom]

Success (on TestPyPI).

[mdboom]

On PyPI proper, it seems it still has the file size limit:

requests.exceptions.HTTPError: 400 Client Error: distribution file too large

Don't know if that's still to be expected. Once that's resolved, I'll upload this, and this bug can be closed...

Thanks for all of your help.

[ewdurbin]

@mdboom just verified the updated limit is in place on PyPI, would you mind trying again?

[mdboom]

Success! Thanks for everyone's help. Closing.