Macos certif #447

Czaki · 2020-10-08T21:48:31Z

This PR is for fix #443
Because try of using /Applications/Python 2.7/Install Certificates.command fail I decide to vendor in script base on one present on my machine.

I'm not sure what to do with python 3.5 so I skip this step on this version. @mayeut any suggestion?

The test for this case could be hard. We could download something from github.com, but any break in this functionality may produce error long time after change happen.

henryiii · 2020-10-08T22:03:45Z

Another solution I've seen mentioned that might include Python 3.5 is:

pip install certifi
CERT_PATH=$(python -m certifi)
export SSL_CERT_FILE=${CERT_PATH}
export REQUESTS_CA_BUNDLE=${CERT_PATH}

But I think the problem is that Python 3.6+ have their own copy of OpenSSL, while 3.5 does not and uses the system copy, so does this even show up at all for 3.5?

joerick · 2020-10-09T12:36:10Z

Thanks for looking at this @Czaki. But due to the existing test_ssl I'm still unsure exactly what we're fixing here. As I noted in #443, I'd like to see a failing test that recreates the SSL error before we attempt any fixes using Install Certificates.command or similar.

cibuildwheel/macos.py

mayeut · 2020-10-09T19:55:11Z

But I think the problem is that Python 3.6+ have their own copy of OpenSSL, while 3.5 does not and uses the system copy, so does this even show up at all for 3.5?

The system copy was so old on python 3.5 that it did not support TLS 1.2 which caused some issue. Its openssl version has been patched in #71 not to use the system one.

I'm not sure what to do with python 3.5 so I skip this step on this version. @mayeut any suggestion?

It shall be patched like the other versions.

djhoese · 2020-10-09T20:13:47Z

I just tried forcing the TLS protocol version in my branch for vispy and it worked! I made sure not to use the dev version of cibuildwheel and I don't think I was skipping the test or anything. I think this is the main cause. Can someone summarize all of the issues related to why cibuildwheel started doing that and why it works?

mayeut · 2020-10-09T20:54:52Z

@joerick, I can reproduce the issue: https://github.com/mayeut/cibuildwheel/runs/1233247122?check_suite_focus=true#step:6:158

Czaki · 2020-10-09T21:04:32Z

my test run does not finish yet, but basic on @mayeut I expand the test set.

But I think that we should test more ssl context cases.

EDIT also got error

https://dev.azure.com/PartSeg/Open%20Source%20contrib/_build/results?buildId=1764&view=results

Czaki · 2020-10-09T21:14:36Z

@mayeut I got this for python 3.5

FileNotFoundError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl'

Did you have any idea? Or omit python 3.5 as there is plan to drop its support?

This reverts commit 0980d5d.

mayeut · 2020-10-09T21:34:45Z

@mayeut I got this for python 3.5
FileNotFoundError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl'
Did you have any idea? Or omit python 3.5 as there is plan to drop its support?

That's because the folder /Library/Frameworks/Python.framework/Versions/3.5/etc/openssl does not exist yet. It should be created before symlink is created (it does not exist because neither the official installer nor the patch create this folder)

Czaki · 2020-10-10T00:16:53Z

@mayeut When omit test_ssl for python 3.5 then everything pass. But for python 3.5 its fail. Could You review my changes? Have You any idea why python 3.5 does not use certificates from certifi?

mayeut · 2020-10-10T11:21:14Z

@mayeut When omit test_ssl for python 3.5 then everything pass. But for python 3.5 its fail. Could You review my changes? Have You any idea why python 3.5 does not use certificates from certifi?

I just tested "manually" the patch on my machine (i.e., just following the logic):
before:

Matt$ /Library/Frameworks/Python.framework/Versions/3.5/bin/python3
Python 3.5.4 (v3.5.4:3f56838976, Aug  7 2017, 12:56:33) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2u  20 Dec 2019'
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/certs')
>>> from urllib.request import urlopen
>>> data = urlopen("https://raw.githubusercontent.com/joerick/cibuildwheel/master/README.md")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1261, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 385, in wrap_socket
    _context=self)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 760, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 996, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)

patch logic:

Matt$ mkdir -p /Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/
Matt$ cp /Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/cert.pem /Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/

after:

Matt$ /Library/Frameworks/Python.framework/Versions/3.5/bin/python3
Python 3.5.4 (v3.5.4:3f56838976, Aug  7 2017, 12:56:33) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/cert.pem', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/certs')
>>> from urllib.request import urlopen
>>> data = urlopen("https://raw.githubusercontent.com/joerick/cibuildwheel/master/README.md")
>>> data.read()
b'cibuildwheel\n============\n\n[![PyPI](https://img.shie................

So it should be working. Now investigating why it's failing in CI.

mayeut · 2020-10-10T11:27:32Z

With an updated test case to print ssl.get_default_verify_paths(), I'm still getting the following

DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/certs')

cafile should not be None here.

cibuildwheel/resources/install_certifi.py

mayeut

With mode changed with octal 0o775, tests are passing as expected. c.f. https://github.com/mayeut/cibuildwheel/runs/1235348121?check_suite_focus=true

Co-authored-by: Matthieu Darbois <mayeut@users.noreply.github.com>

YannickJadoul · 2020-10-10T12:53:43Z

With mode changed with octal 0o775, tests are passing as expected.

Great! Thanks you so much for checking and finding that, @mayeut! :-)

@Czaki, I've taken the liberty to apply @mayeut's suggestion, and (hopefully) get a green build.

YannickJadoul

I haven't really followed the whole discussion or problem, so these are mostly minor style-related comments.

Apart from those, it seems like @djhoese, @Czaki, and @mayeut together figured out what went wrong and how to best fix this, so not a lot of comments there.

cibuildwheel/resources/install_certifi.py

YannickJadoul · 2020-10-10T14:14:37Z

cibuildwheel/resources/install_certifi.py

+import stat
+import subprocess
+import sys
+STAT_0o775 = ( stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR


New line, before?

cibuildwheel/resources/install_certifi.py

test/test_1_ssl.py

YannickJadoul · 2020-10-10T14:18:11Z

test/test_1_ssl.py

-            data = urlopen("https://www.nist.gov", context=context)
+        context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
+        data = urlopen("https://www.nist.gov", context=context)
+        data = urlopen("https://raw.githubusercontent.com/joerick/cibuildwheel/master/CI.md", context=context)


Do we need to assert something for the received data? E.g. just that it's not empty, or so?

maybe we should, but then we should have some file with some fixed content.

Not per se. You could just test if something non-empty was downloaded? Or take away the data =, maybe? It just feels weird, all these useless assignments.

cibuildwheel/macos.py

@YannickJadoul

change for python 3.5 now ok, letting @YannickJadoul follow-up with his review

This reverts commit aaa1109.

Czaki · 2020-10-10T15:04:37Z

Before merge I have one question.
If we should test multiple SSL version ssl.PROTOCOL_* or only this two tests which we already have.

mayeut · 2020-10-10T15:32:34Z

The test with ssl.PROTOCOL_TLSv1_2 was only there to check support was indeed there for this protocol a couple years back. I don't think we should have any specific tests other than that until web servers start to heavily shift to newer protocols and disable TLS v1.2 (that was the case for PyPI where the minimum supported protocol shifted to TLS v1.2, hence this test).

c.f. https://www.ssllabs.com/ssl-pulse/ for an idea of currently deployed protocols.

YannickJadoul

Thanks, @Czaki! Looks good to me, now!

mayeut · 2020-10-10T16:24:40Z

Thanks @Czaki !

Czaki · 2020-10-10T16:46:22Z

@joerick 1.6.3?

Czaki added 3 commits October 8, 2020 23:12

update certif on macos

edebcae

vendor in certifi script

d089766

exception rename for python 2

6f6b21a

Czaki mentioned this pull request Oct 8, 2020

SSL issues with OSX on Azure #443

Closed

fix formating

021a093

djhoese reviewed Oct 9, 2020

View reviewed changes

cibuildwheel/macos.py Outdated Show resolved Hide resolved

fix typo

9afd456

Czaki added 2 commits October 9, 2020 22:54

patch also python 3.5

0980d5d

add test

8a69686

Revert "patch also python 3.5"

5be9658

This reverts commit 0980d5d.

enable for python 3.5, create missed dir

e68db65

Czaki force-pushed the macos_certif branch from a0d8733 to e68db65 Compare October 9, 2020 22:24

Czaki added 5 commits October 10, 2020 00:32

create dir only on python 3.5

48cfa74

reorder tests

aaa1109

test if fail only on cp35

3ad94eb

fix env modify

f1be10b

check permissions

2c16f23

alfaro96 mentioned this pull request Oct 10, 2020

MNT Use cibuildwheel to build the wheels scikit-learn/scikit-learn#17921

Merged

mayeut reviewed Oct 10, 2020

View reviewed changes

cibuildwheel/resources/install_certifi.py Outdated Show resolved Hide resolved

mayeut requested changes Oct 10, 2020

View reviewed changes

Update cibuildwheel/resources/install_certifi.py

93b75d0

Co-authored-by: Matthieu Darbois <mayeut@users.noreply.github.com>

YannickJadoul reviewed Oct 10, 2020

View reviewed changes

mayeut previously approved these changes Oct 10, 2020

View reviewed changes

Czaki added 2 commits October 10, 2020 16:54

reformat install_certifi

5d79069

Revert "reorder tests"

3c864f9

This reverts commit aaa1109.

YannickJadoul approved these changes Oct 10, 2020

View reviewed changes

mayeut merged commit 8094b3f into pypa:master Oct 10, 2020

Czaki mentioned this pull request Oct 27, 2020

Windows: urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed #452

Closed

Czaki deleted the macos_certif branch October 27, 2020 17:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Macos certif #447

Macos certif #447

Czaki commented Oct 8, 2020

henryiii commented Oct 8, 2020

joerick commented Oct 9, 2020

mayeut commented Oct 9, 2020

djhoese commented Oct 9, 2020

mayeut commented Oct 9, 2020

Czaki commented Oct 9, 2020 •

edited

Loading

Czaki commented Oct 9, 2020

mayeut commented Oct 9, 2020

Czaki commented Oct 10, 2020

mayeut commented Oct 10, 2020

mayeut commented Oct 10, 2020

mayeut left a comment

YannickJadoul commented Oct 10, 2020

YannickJadoul left a comment

YannickJadoul Oct 10, 2020

YannickJadoul Oct 10, 2020

Czaki Oct 10, 2020

YannickJadoul Oct 10, 2020

Czaki commented Oct 10, 2020

mayeut commented Oct 10, 2020 •

edited

Loading

YannickJadoul left a comment

mayeut commented Oct 10, 2020

Czaki commented Oct 10, 2020

Macos certif #447

Macos certif #447

Conversation

Czaki commented Oct 8, 2020

henryiii commented Oct 8, 2020

joerick commented Oct 9, 2020

mayeut commented Oct 9, 2020

djhoese commented Oct 9, 2020

mayeut commented Oct 9, 2020

Czaki commented Oct 9, 2020 • edited Loading

Czaki commented Oct 9, 2020

mayeut commented Oct 9, 2020

Czaki commented Oct 10, 2020

mayeut commented Oct 10, 2020

mayeut commented Oct 10, 2020

mayeut left a comment

Choose a reason for hiding this comment

YannickJadoul commented Oct 10, 2020

YannickJadoul left a comment

Choose a reason for hiding this comment

YannickJadoul Oct 10, 2020

Choose a reason for hiding this comment

YannickJadoul Oct 10, 2020

Choose a reason for hiding this comment

Czaki Oct 10, 2020

Choose a reason for hiding this comment

YannickJadoul Oct 10, 2020

Choose a reason for hiding this comment

Czaki commented Oct 10, 2020

mayeut commented Oct 10, 2020 • edited Loading

YannickJadoul left a comment

Choose a reason for hiding this comment

mayeut commented Oct 10, 2020

Czaki commented Oct 10, 2020

Czaki commented Oct 9, 2020 •

edited

Loading

mayeut commented Oct 10, 2020 •

edited

Loading