[bug] ClearML SSL certificate verification

[leej3]

Describe the bug

I am reporting this here because it impacts QHub users who have the ClearML integration enabled. I have encountered this on a deployment that is using self-signed TLS certs, I don't know if it occurs for default deployments. The problem appeared some weeks back when debugging some other ClearML details and I just discovered a hacky solution for this particular issue. I don't know the exact source of the bug though... it may not actually be in QHub but if we can help avoid other users encountering it in an easy manner that would be good. And a less hacky solution would be good too!

When running a pipeline that uses ClearML a failure occurs when the ClearML task is created. It gives the error:

Retrying (Retry(total=239, connect=240, read=240, redirect=240, status=240)) 
after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] 
certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /auth.login

In this call frame the environment variable CURL_CA_BUNDLE is '/home/conda/environments/cba_pinned/lib/python3.8/site-packages/certifi/cacert.pem' (it is unset in the user environment). If this is instead set to an empty string the Task creation succeeds albeit with warnings that TLS verification failed for the various hosts accessed. For example :

/home/conda/environments/cba_pinned/lib/python3.8/site-packages/urllib3/connectionpool.py:1013: 
InsecureRequestWarning: Unverified HTTPS request is being made to host 'bigquery.googleapis.com'.
 Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

Are the TLS certs mounted into the container and - if that is the case - would automatically setting REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE to this path be useful?

How to reproduce

This is on a deployment on GCP, with a custom TLS cert, and an internal load balancer. The deployment is off the optional_render branch (roughly v0.3.12).

[leej3]

Using conda installed curl could conceivably be making things worse... see here

[leej3]

I'm still tapping away at this. As far as I can tell I can reproduce it more minimally by running the user server from a docker container locally (I've forwarded the host network stack and mounted in the root, intermediate, and website TLS certs).

docker run -u root --rm -ti --net=host  -v $PWD:/mnt  quansight/qhub-jupyterlab:v0.3.14 

Currently, I am attempting to trigger the cert error with:

conda activate
python -c 'import requests;requests.get("https://qhub.deployment...")

I am copying the certs into /usr/local/share/ca-certificates and using update-ca-certificates (roughly according to this. I'm not sure it's working but I am learning stuff as I go...

[iameskild]

Superceeded by #1217