[Generic] tls verify not being honored at the httprequest level when internal_ssl is enabled

[sstarcher]

I don't know if this is a bug in tornado or how it's supposed to function, but tls_verify is not being honored at the HTTPRequest level and is only functional when overriding it at the AsyncHTTPClient level.

[sstarcher]

It's also confusing that the code has an OAUTH_TLS_VERIFY variable also that is only false when it's set to 0. Neither OAUTH_TLS_VERIFY or OAUTH2_TLS_VERIFY set to False, false, or 0 affect the TLS verify for the generic handler.

[minrk]

You're right that it's confusing that there's OAUTH_TLS_VERIFY and OAUTH2_TLS_VERIFY and they behave differently. The GenericOAuthenticator implementation doesn't conform to standards used elsewhere in the code (e.g. OAUTH2 prefix instead of OAUTH), so the OAUTH_TLS_VERIFY is ignored, only OAUTH2_TLS_VERIFY is used. I'll work on unifying those.

It is strange that verify_cert is not honored at the HTTPRequest level, since that's explicitly part of the tornado API. We may need to look into what's going on there.

[sstarcher]

@minrk How shall we fix this? my PR does, but if you have a better idea I would be up for it.

[minrk]

Hm, when I set out to test this the tls verify settings worked as intended, so there was nothing to fix. Could you share some configuration and environment details?

My env:

Package            Version
------------------ ----------
alembic            1.4.0
async-generator    1.10
attrs              19.3.0
certifi            2019.11.28
certipy            0.1.3
cffi               1.14.0
chardet            3.0.4
cryptography       2.8
decorator          4.4.1
entrypoints        0.3
idna               2.8
importlib-metadata 1.5.0
ipython-genutils   0.2.0
Jinja2             2.11.1
jsonschema         3.2.0
jupyter-telemetry  0.0.5
jupyterhub         1.1.0
Mako               1.1.1
MarkupSafe         1.1.1
oauthenticator     0.11.0
oauthlib           3.1.0
pamela             1.0.0
pip                19.0.3
prometheus-client  0.7.1
pycparser          2.19
pyOpenSSL          19.1.0
pyrsistent         0.15.7
python-dateutil    2.8.1
python-editor      1.0.4
python-json-logger 0.1.11
requests           2.22.0
ruamel.yaml        0.16.7
ruamel.yaml.clib   0.2.0
setuptools         40.8.0
six                1.14.0
SQLAlchemy         1.3.13
tornado            6.0.3
traitlets          4.3.3
urllib3            1.25.8
zipp               2.2.0

and configuration (I used JupyterHub for both the oauth and provider):

# oauth consumer
c.JupyterHub.authenticator_class = 'generic-oauth'
c.JupyterHub.spawner_class = 'simple'
c.JupyterHub.ssl_key = 'tls.key'
c.JupyterHub.ssl_cert = 'tls.crt'
c.JupyterHub.port = 9000
c.ConfigurableHTTPProxy.api_url = 'http://127.0.0.1:9010'
c.ConfigurableHTTPProxy.pid_file = 'oauth-chp.pid'
c.JupyterHub.hub_port = 9090
c.JupyterHub.log_level = 10
c.JupyterHub.db_url = 'sqlite://'
c.GenericOAuthenticator.client_id = 'jupyterhub-oauth-client-service'
c.GenericOAuthenticator.client_secret = 'abc1234567'
c.GenericOAuthenticator.basic_auth = False
c.GenericOAuthenticator.userdata_url = 'https://127.0.0.1:8000/hub/api/user'
c.GenericOAuthenticator.authorize_url = 'https://127.0.0.1:8000/hub/api/oauth2/authorize'
c.GenericOAuthenticator.token_url = 'https://127.0.0.1:8000/hub/api/oauth2/token'
c.GenericOAuthenticator.oauth_callback_url = 'https://oauth.local.minrk.net:9000/hub/oauth_callback'
c.GenericOAuthenticator.tls_verify = False
c.GenericOAuthenticator.username_key = 'name'

c.GenericOAuthenticator.extra_params = {
    'client_id': c.GenericOAuthenticator.client_id,
    'client_secret': c.GenericOAuthenticator.client_secret,
}

# oauth provider
c.JupyterHub.authenticator_class = 'dummy'
c.JupyterHub.spawner_class = 'simple'
c.JupyterHub.ssl_key = 'tls.key'
c.JupyterHub.ssl_cert = 'tls.crt'

c.JupyterHub.services = [
    {
        'name': 'oauth',
        'api_token': 'abc1234567',
        'oauth_client_id': "jupyterhub-oauth-client-service",
        'oauth_redirect_uri': 'https://oauth.local.minrk.net:9000/hub/oauth_callback',
    }
]

[sstarcher]

For debugging I took master of this repo and just made a single change

hub-7655dddccf-hcfxh:hub tls verify is
hub-7655dddccf-hcfxh:hub False
hub-7655dddccf-hcfxh:hub [W 2020-02-11 16:43:06.920 JupyterHub iostream:1407] SSL Error on 13 ('1.1.1.1', 443): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

--- a/oauthenticator/generic.py
+++ b/oauthenticator/generic.py
@@ -128,6 +126,8 @@ class GenericOAuthenticator(OAuthenticator):
             )
             headers.update({"Authorization": "Basic {}".format(b64key.decode("utf8"))})

+        print('tls verify is')
+        print(self.tls_verify)
         req = HTTPRequest(url,
                           method="POST",
                           headers=headers,

      c.GenericOAuthenticator.tls_verify = False

alembic (1.4.0)
async-generator (1.10)
bcrypt (3.1.7)
cachetools (4.0.0)
certifi (2019.11.28)
certipy (0.1.3)
cffi (1.14.0)
chardet (3.0.4)
cryptography (2.8)
decorator (4.4.1)
entrypoints (0.3)
escapism (1.0.0)
globus-sdk (1.8.0)
google-auth (1.11.0)
idna (2.8)
ipython-genutils (0.2.0)
Jinja2 (2.11.1)
jupyterhub (1.0.0)
jupyterhub-dummyauthenticator (0.3.1)
jupyterhub-firstuseauthenticator (0.12)
jupyterhub-hmacauthenticator (0.1)
jupyterhub-kubespawner (0.11.2.dev0)
jupyterhub-ldapauthenticator (1.2.2)
jupyterhub-ltiauthenticator (0.4.0)
jupyterhub-tmpauthenticator (0.6)
kubernetes (10.0.1)
ldap3 (2.6.1)
Mako (1.1.1)
MarkupSafe (1.1.1)
mwoauth (0.3.7)
nullauthenticator (1.0.0)
oauthenticator (0.10.1.dev0)
oauthlib (3.1.0)
pamela (1.0.0)
pip (9.0.1)
prometheus-client (0.7.1)
psycopg2-binary (2.8.4)
py-spy (0.3.2)
pyasn1 (0.4.8)
pyasn1-modules (0.2.8)
pycparser (2.19)
pycurl (7.43.0.5)
PyJWT (1.7.1)
PyMySQL (0.9.3)
pyOpenSSL (19.1.0)
python-dateutil (2.8.1)
python-editor (1.0.4)
PyYAML (5.3)
requests (2.22.0)
requests-oauthlib (1.3.0)
rsa (4.0)
setuptools (39.0.1)
six (1.14.0)
SQLAlchemy (1.3.13)
statsd (3.3.0)
tornado (6.0.3)
traitlets (4.3.3)
urllib3 (1.25.8)
websocket-client (0.57.0)
wheel (0.30.0)

[sstarcher]

I'm using https://github.com/jupyterhub/zero-to-jupyterhub-k8s

[minrk]

Aha! This only happens with the curl httpclient, not the default simple httpclient. That narrows it down and I can reproduce it. I'll try to see what the root of it is.

[minrk]

Wait, no, nevermind, I cannot reproduce it with pycurl, I had accidentally re-enabled tls_verify when I was testing. tls_verify does indeed behave as intended with pycurl in my tests.

Is it possible to share some of your z2jh config? Do you perhaps have internal ssl enabled?

Since you have inserted some debugging, can you check after creating the request, the values of:

req.validate_cert
req.ssl_options
http_client.__class__
http_client.defaults

[sstarcher]

Since I'm using zero-to-jupyter they do alot of dynamic config.

I do have internal ssl enabled.

      c.Spawner.cmd = ['jupyter-labhub']

      c.JupyterHub.internal_ssl = True
      c.JupyterHub.bind_url = "https://0.0.0.0:8081"
      c.JupyterHub.hub_bind_url = "https://0.0.0.0:8081"
      c.ConfigurableHTTPProxy.api_url = 'https://proxy-api.nb.svc.cluster.local:8001'
      c.JupyterHub.trusted_alt_names = ["IP:127.0.0.1", "DNS:jupyter.{{ .Environment.Values.domain }}", "DNS:*.nb.svc.cluster.local"]
      c.JupyterHub.hub_connect_url = "https://hub.nb.svc.cluster.local:8081"

      c.KubeSpawner.working_dir = '/home/jovyan/'

[sstarcher]

So is the default client overriding the req... that certainly seems like a bug.

hub-69f655f797-pltfq:hub False
hub-69f655f797-pltfq:hub None
hub-69f655f797-pltfq:hub <class 'tornado.simple_httpclient.SimpleAsyncHTTPClient'>
hub-69f655f797-pltfq:hub {'connect_timeout': 20.0, 'request_timeout': 20.0, 'follow_redirects': True, 'max_redirects': 5, 'decompress_response': True, 'proxy_password': '', 'allow_nonstandard_methods': False, 'validate_cert': True, 'ssl_options': <ssl.SSLContext object at 0x7fbc6b492a08>}

[minrk]

I think the reason there's an issue is that there are two "levels" of ssl config, the single params and the ssl options, and only one can be used at a time.

So the issue is that the ssl_options being set from defaults configured when internal_ssl is enabled has priority over single flags, even at the request level. I'm not sure this is a bug other than that there's no warning that setting ssl options in two incompatible ways isn't going to work.

I think we probably shouldn't be setting global defaults when internal_ssl is enabled, but barring that, overriding defaults when creating self.client is the quickest workaround.

[sstarcher]

@minrk I removed the force_instance and it failed.

[minrk]

Thanks for your patience and testing!