[CT-392] [Bug] SSL: CERTIFICATE_VERIFY_FAILED occurs non-deterministicly

[jaklan]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Hi, I have a problem with running dbt deps command behind corporate VPN. I have noticed that sometimes it fails due to:

requests.exceptions.SSLError: HTTPSConnectionPool(host='hub.getdbt.com', port=443):
Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

but sometimes it just works when I run the command again.

I prepared a small script to investigate it:

import requests

try:
    with requests.get("https://hub.getdbt.com", stream=True) as rsp:
        ip, port = rsp.raw._connection.sock.getpeername()
        print(ip, port)
except:
    print("except")
    with requests.get("https://hub.getdbt.com", stream=True, verify=False) as rsp:
        ip, port = rsp.raw._connection.sock.getpeername()
        print(ip, port)

and run multiple times - here's the output:

35.198.187.166 443
---
35.234.85.218 443
---
except
3.65.48.84 443
---
except
3.127.196.128 443
---
except
18.158.19.118 443

It looks like the error depends on the target IP address, but it's just my guess, I would really love to hear your ideas.

Expected Behavior

I would expect the command always fails or always passes.

Steps To Reproduce

Run dbt deps behind some VPN with self-signed certificates in the chain.

Relevant log output

No response

Environment

- OS: macOS
- Python: 3.9
- dbt: 1.0.3

What database are you using dbt with?

No response

Additional Context

No response

[nathaniel-may]

Hi @jaklan, thanks for the details in this report and I'm sorry to hear you've been having issues with connecting to hub.getdbt.com.

I did some preliminary investigation of the ssl/tls certificates on the ip addresses you mentioned and everything is up-to-date on our end as far as I can tell. You may want to contact your network administrator and see if there's something they can do to help.

Since the certs at all the above mentioned ip addresses were signed by the same certificate authority, the only immediate workaround I can think of is if you create a local dns entry on your machine that forces "hub.getdbt.com" to forward to the ip addresses that seem to be working. This isn't ideal because the entry would have to be manually updated if we make any dns changes in the future. Detailing how to set that up is beyond the scope of the assistance I can provide though.

I'm going to close this ticket as out of scope, but please feel free to re-open or open a new issue if there's something else you think we can do.

[jaklan]

@nathaniel-may Hah, I just wanted to update and close the issue, good timing 😉 From what I discovered - you use Netlify for hub.getdbt.com, and Netlify uses both GCP and AWS infrastructure. When being inside corporate network, there's an internal proxy CA in the certificate chain. But what is interesting - it's only there when connecting to AWS servers, but not when connecting to GCP servers and that's why dbt deps sometimes worked, and sometimes not. Quite surprising, but definitely it's not a dbt-related issue.

Btw, we have finally resolved the issue by creating custom CA bundle including both certifi certificates used by requests and our internal ones, but maybe it's worth to add some envar which which would disable SSL verification in requets.get() call to hub.getdbt.com? It could make a life easier for some people working behind VPN, especially for local setup. There was some dirty workaround to achieve that before, but it was recently fixed: psf/requests#6071

[nathaniel-may]

Thanks for the update, @jaklan! Glad to hear you got to the root of the issue. I completely understand the desire for a simpler solution here, but we will likely not provide new ways to disable security features. In this case, certificate validation protects your projects from falling victim to a man in the middle attack.

Thanks for reaching out, and happy dbt-ing!

[jaklan]

@nathaniel-may fully understand the reasoning. I just mentioned that because I know it was quite problematic for our Data Scientists working with dbt, because they weren't able to install deps inside VPN (which ofc affected the whole dbt setup) until someone more into networking resolved the root cause of the issue. Disabling https verification would be an easy workaround for them in the meantime, but I agree it's mostly about proper knowledge management inside company about dealing with certs, and could end up with a risky setup if not used with the proper caution.