🐛 [BUG]: CERTIFICATE_VERIFY_FAILED error with canarie-api monitoring #198
Assignees
Labels
bug
Something isn't working
Comments
|
Thanks for submitting an issue. |
|
Root cause is the |
tlvu
added a commit
that referenced
this issue
Aug 14, 2022
LetsEncrypt older root certificate "DST Root CA X3" expired on September 30, 2021, see https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ All the major browsers and OS platform has previously added the new root certificate "ISRG Root X1" ahead of time so the transition to the new root certificate is seemless for all clients. Python `requests` package bundle their own copy of known root certificates and is late to add this new root cert "ISRG Root X1". Had it automatically fallback to the OS copy of the root cert bundle, this would have been seemless. The fix is to force `requests` to use the OS copy of the root cert bundle. Fix for this error: ``` $ docker exec proxy python -c "import requests; requests.request('GET', 'https://lvupavicsmaster.ouranos.ca/geoserver' > )" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in request response = session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 433, in send raise SSLError(e, request=request) requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661) ``` Default SSL root cert bundle of `requests`: ``` $ docker exec proxy python -c "import requests; print requests.certs.where()" /usr/local/lib/python2.7/dist-packages/requests/cacert.pem ``` Confirm the fix works: ``` $ docker exec -it proxy bash root@37ed3a2a03ae:/opt/local/src/CanarieAPI/canarieapi# REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt python -c "import requests; requests.request('GET', 'https://lvupavicsmaster.ouranos.ca/geoserver')" root@37ed3a2a03ae:/opt/local/src/CanarieAPI/canarieapi# $ docker exec proxy env |grep REQ REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt ``` Fixes #198
tlvu
added a commit
that referenced
this issue
Aug 18, 2022
…-to-verify-LetsEncrypt-ssl-cert canarie-api: fix unable to verify LetsEncrypt SSL certs LetsEncrypt older root certificate "DST Root CA X3" expired on September 30, 2021, see https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ All the major browsers and OS platform has previously added the new root certificate "ISRG Root X1" ahead of time so the transition to the new root certificate is seemless for all clients. Python `requests` package bundle their own copy of known root certificates and is late to add this new root cert "ISRG Root X1". Had it automatically fallback to the OS copy of the root cert bundle, this would have been seemless. The fix is to force `requests` to use the OS copy of the root cert bundle. Fix for this error: ``` $ docker exec proxy python -c "import requests; requests.request('GET', 'https://lvupavicsmaster.ouranos.ca/geoserver' > )" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in request response = session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 433, in send raise SSLError(e, request=request) requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661) ``` Default SSL root cert bundle of `requests`: ``` $ docker exec proxy python -c "import requests; print requests.certs.where()" /usr/local/lib/python2.7/dist-packages/requests/cacert.pem ``` Confirm the fix works: ``` $ docker exec -it proxy bash root@37ed3a2a03ae:/opt/local/src/CanarieAPI/canarieapi# REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt python -c "import requests; requests.request('GET', 'https://lvupavicsmaster.ouranos.ca/geoserver')" root@37ed3a2a03ae:/opt/local/src/CanarieAPI/canarieapi# $ docker exec proxy env |grep REQ REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt ``` Fixes #198
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
Using LetsEncrypt SSL certificate, the canarie-api monitoring page https://lvupavicsdev.ouranos.ca/canarie/node/service/status fail with a bunch of
CERTIFICATE_VERIFY_FAILEDbut all the services are working properly.This is most likely due to https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ meaning the openssl library in the proxy container is too old.
This bug only impact PAVICS deployment using LetsEncrypt certificate and only if the canarie-api monitoring matter. Otherwise all other services are still working fine (Jenkins run all pass http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/master/1253/console).
Ouranos production has been switched to use another SSL certificate provider as a work-around.
@moulab88 FYI
Details
How to reproduce
The text was updated successfully, but these errors were encountered: