Various Python packages build failures when contacting pypi

[Cynerd]

Maintainer: @neheb
Environment:

Description:
Some python packages now fail to download their dependencies from pypi. I suspect that culprit is following commit: 3b93506

Packages I noticed:

• bcrypt (@cshoredaniel )
• python-twisted (@jefferyto)
• python-dateutils (@kissg1988 )
• python-pycparser (@jefferyto )

twisted:

Download error on https://pypi.org/simple/incremental/: invalid protocol version -- Some packages may not be found!
Couldn't find index page for 'incremental' (maybe misspelled?)
Download error on https://pypi.org/simple/: invalid protocol version -- Some packages may not be found!

cffi

Collecting cffi==1.12.1
Exception:
Traceback (most recent call last):
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 318, in run
    resolver.resolve(requirement_set)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/resolve.py", line 102, in resolve
    self._resolve_one(requirement_set, req)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 218, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 164, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/index.py", line 572, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/index.py", line 530, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/index.py", line 675, in _get_pages
    page = self._get_page(location)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/index.py", line 793, in _get_page
    return _get_html_page(link, session=self.session)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/index.py", line 144, in _get_html_page
    "Cache-Control": "max-age=0",
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_internal/download.py", line 396, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/urllib3/connection.py", line 344, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 277, in create_urllib3_context
    context = SSLContext(ssl_version or ssl.PROTOCOL_SSLv23)
  File "/home/beast/beast/workspace/turris-os-packages-dragons-mox/build/staging_dir/hostpkg/lib/python3.7/ssl.py", line 388, in __new__
    self = _SSLContext.__new__(cls, protocol)
ValueError: invalid protocol version
You are using pip version 18.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[neheb]

That doesn't look right. None of those projects use pypi.org. All use https://files.pythonhosted.org

I tested it now and it seems to work.

edit: Never mind, I see the problem. In the twisted case, python-incremental is already a dependency. It should not be trying to download it.

In the case of cffi, it looks like it's using some deprecated API. Meaning it needs to be patched.

[neheb]

Maybe I see what's happening. urllib3 is calling SSLv23_method, which is not only deprecated but not defined in Python 3 with the patch. Can you try this with urllib3 and see if it helps?

diff --git a/src/urllib3/util/ssl_.py b/src/urllib3/util/ssl_.py
index 64ea192..1344f94 100644
--- a/src/urllib3/util/ssl_.py
+++ b/src/urllib3/util/ssl_.py
@@ -261,7 +261,7 @@ def create_urllib3_context(ssl_version=None, cert_reqs=None,
         Constructed SSLContext object with specified options
     :rtype: SSLContext
     """
-    context = SSLContext(ssl_version or ssl.PROTOCOL_SSLv23)
+    context = SSLContext(ssl_version or ssl.PROTOCOL_TLS)
 
     context.set_ciphers(ciphers or DEFAULT_CIPHERS)
 

[Cynerd]

I tested it and no vail. I checked it and in build_dir it is applied but error is still the same.

I am going to test it on my own pc not on compilation server to see if there is no outside influence.

Edit: Thinking about it. This does not make sense that it would help. Package python-urllib3 does not even have host build. That patch is not used nor that specific version of urllib when you invoke pip in build system. I don't see deep enough to know where that dependency is build for host but it is not from that package for sure I would say. This is problem of host build pip or some of its dependency build for host not package.

[neheb]

Out of curiosity, is Turris OS using OpenSSL 1.1.2 or 1.0.2?

[Cynerd]

I tried it on my pc and I was able to reproduce it. What is interesting is that I was not able to reproduce it with plain OpenWRT make defconfig. That means that that cause is either something in our patch set or in configuration. I looked in to our patches and the only one that mind be influencing this is this one: https://gitlab.labs.nic.cz/turris/turris-build/blob/master/patches/packages/hack/0001-python-Fix-build.patch

I just dropped it and trying again but that patch I would say should not have change anything. If host packages are already used correctly then it introduces nothing and if they are not then it should have helped not break stuff I think.
Edit: Well I was right. It did not help. Trying Turris OS config in clean OpenWRT now, we will see.

Also I tried to revert 3b93506 and build passed so it definitely has something to do with that patch. Question is what causes that regression, our patch, configuration or something else.

We are using 1.1.1. We have no patch on openssl so we use openwrt master version at the moment.

[Cynerd]

I managed to reproduce it even without our patches so it has nothing to do with that. I copied configuration for Turris Omnia from Turris OS and tried to build it on plain OpenWRT. I ended up with same error. I suspect that it is configuration based. I am going to compare differences between our config and make defconfig and we will see.

[Cynerd]

Faster way to reproduce with clean openwrt master and latest feeds: make defconfig diffconfig toolchain/compile target/compile package/python-cffi/compile.

[neheb]

Can't duplicate. The closest thing to an error is this:

rmdir: failed to remove '/home/mangix/devstuff/openwrt/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/python-cffi-1.12.1/ipkg-arm_cortex-a9_vfpv3/python-cffi/usr': Directory not empty

Note that I also have #8369 in my tree.

[neheb]

Never mind. Duplicated. Will try to fix.

[neheb]

Fixed. Please test #8412

[commodo]

For now, I'll choose to revert these 2 patches and wait for upstream Python to address this.
The OpenSSL deprecated API test-matrix is too big to account for other potential issues, and I don't really want to do firefighting all that much these days.