Wheel packages not listed in fallback cache mode

[maroux]

Hi,

With the following settings:

pypi.fallback = cache
pypi.always_show_upstream = true

I expected pypicloud to mirror all fallback urls including wheels, but that doesn't happen.

For some packages, that means installation fails (e.g. cryptography on Mac OS X, see pyca/cryptography#2692 and https://pypi.python.org/simple/cryptography/).

I didn't find any config option to enable or disable this behavior.

[stevearc]

I recommend looking at the simple page of your pypicloud: http://your-pypicloud-server.com/pypi/cryptography/ to see what packages are available. If there are no wheels, then there's definitely a bug. I suspect that you'll see the wheels, though.

Try doing a pip install with -v. It'll print out what versions of the cryptography package it has found, and why it is or is not selecting certain packages.

[maroux]

@stevearc Yep, I verified that /simple/cryptography/ doesn't list wheel packages.

Here's the response from pip install -v :

  Getting page https://<pypi-url>/pypi/cryptography/
  Looking up "https://<pypi-url>/pypi/cryptography/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): <pypi-url>
  "GET /pypi/cryptography/ HTTP/1.1" 200 6387
  Updating cache with response from "https://<pypi-url>/pypi/cryptography/"
  Analyzing links from page https://<pypi-url>/pypi/cryptography/
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.2.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.2.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.2.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.2.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.3.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.3
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.4.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.4
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.5.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.5.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.5.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.5.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.5.3.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.5.3
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.5.4.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.5.4
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.5.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.5
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.6.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.6.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.6.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.6
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.7.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.7.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.7.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.7.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.7.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.7
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.8.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.8.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.8.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.8.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.8.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.8
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.9.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.9.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.9.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.9.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.9.3.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.9.3
    Found link https://<pypi-url>/api/package/cryptography/cryptography-0.9.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 0.9
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.0.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.0.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.0.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.0.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.0.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.0
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.1.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.1.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.1.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.1.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.2.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.2.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.2.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.2.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.2.3.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.2.3
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.3.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.3.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.3.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.3.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.3.3.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.3.3
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.3.4.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.3.4
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.3.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.3
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.4.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.4
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.5.1.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.5.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.5.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.5.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.5.3.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.5.3
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.5.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.5
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.6.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.6
    Found link https://s3.amazonaws.com/<s3-bucket>/cryptography/cryptography-1.7.1.tar.gz?Signature=%2FEqFRWi0In0pTitp6WNdCKe4mW8%3D&Expires=1489779456&AWSAccessKeyId=AKIAJUWVESDMNBXFX7MA (from https://<pypi-url>/pypi/cryptography/), version: 1.7.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.7.2.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.7.2
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.7.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.7
    Found link https://s3.amazonaws.com/<s3-bucket>/cryptography/cryptography-1.8.1.tar.gz?Signature=4PHHJdTbvbWOUW5h92wYzHnsbBc%3D&Expires=1489779456&AWSAccessKeyId=AKIAJUWVESDMNBXFX7MA (from https://<pypi-url>/pypi/cryptography/), version: 1.8.1
    Found link https://<pypi-url>/api/package/cryptography/cryptography-1.8.tar.gz (from https://<pypi-url>/pypi/cryptography/), version: 1.8
  Using version 1.8.1 (newest of versions: 0.1, 0.2, 0.2.1, 0.2.2, 0.3, 0.4, 0.5, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6, 0.6.1, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.9, 0.9.1, 0.9.2, 0.9.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.1.1, 1.1.2, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.4, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.6, 1.7, 1.7.1, 1.7.2, 1.8, 1.8.1)

[stevearc]

Oh, weird! Okay, could I get a little more information? Set pypi.always_show_upstream = false and tell me what appears on /simple/cryptograpy (I want to see what packages you have stored locally). Also what is the output of a pip freeze?

[maroux]

Hey @stevearc

It looks like I was using https://github.com/joshbenner/docker-pypicloud which has pinned the pypicloud version to a pretty old release.

Retried with your docker repo and it works just fine. I did have a few issues with the docker file particularly with the lack of support for env vars, but I'll open an issue/PR in that repo.

Thanks and sorry for the trouble!

[stevearc]

That's good to hear. I've recently been doing some work on the docker side of things, so definitely let me know if there's something you think should be added!