aws-eks: addHelmChart not accepting SSL certificate from public JupyterHub helm chart

[youngjeong46]

Describe the bug

We are using EKS Blueprints to provision an EKS cluster with a JupyterHub addon. JupyterHub addon uses its public helm chart and adds to the EKS cluster using addHelmChart (jupyterhub v2.0 from https://hub.jupyer.org/helm-chart/). It has been working until recently, when it started producing an error with unknown authority, and rolls back.

The certificate was recently renewed per index.yaml, but the chart deploys successfully when using Helm CLI. We have explored the option of bypassing SSL using insecure-skip-tls-verify flag, but it is not an option available under addHelmChart nor is security best practice.

Expected Behavior

Deploys the helm chart successfully.

Current Behavior

When the CDK deploys with added JupyterHub addon, we get the following error:

Message returned: Error: b'Release "blueprints-addon-jupyterhub" does not exist. Installing it now.\nError: looks like "https://hub.jupyter.org/helm-chart/" is not a valid chart repository or cannot be reached: Get "https://hub.jupyter.org/helm-chart/index.yaml": x509: certificate signed by unknown authority\n'

Reproduction Steps

1. git clone https://github.com/aws-quickstart/cdk-eks-blueprints.git
2. In the directory, run npm i.
3. Then deploy an example stack: npx cdk deploy blueprint-construct-dev

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.78.0

Framework Version

No response

Node.js Version

16.16.0

OS

13.3.1

Language

Typescript

Language Version

No response

Other information

No response

[melodyyangaws]

+1
have the same error when install the jupyerhub chart via the eks_cluster.add_helm_chart() in CDKv2 (Python). Manually helm install works though. How to workaround it in CDK?

[pahud]

How did you manually install the helm chart? Can you share command with full arguments? Are you suggesting to have an option in aws-eks to turn on insecure-skip-tls-verify for addHelmChart() ?

[youngjeong46]

How did you manually install the helm chart? Can you share command with full arguments?

Yes, using helm cli: helm install blueprints-addon-jupyterhub jupyterhub/jupyterhub --version 2.0.0 -n jupyterhub --create-namespace

Are you suggesting to have an option in aws-eks to turn on insecure-skip-tls-verify for addHelmChart() ?

This would probably be a good immediate solution, although I'm not certain that would align with best practices.

[nikhil-sharma31]

Is this resolved? I am also facing the same issue
Even after using insecure-skip-tls-verify, it does not resolve

[nikhil-sharma31]

I was able to resolve this by updating certificates on my server
sudo yum update ca-certificates
sudo yum reinstall ca-certificates

[melodyyangaws]

any update on the issue team? there are multiple CDK projects are impacted, including some workshop content for our public events.

[melodyyangaws]

How did you manually install the helm chart? Can you share command with full arguments?

just the usual helm install commandhelm install jhub jupyterhub/jupyterhub --values chart/jupyter-values.yaml --version 1.2.0 -n jupyter, it works. But the manual workaround doesn't fit in our case, since we have to automate the installation via CDK. A series of setups are depending on the jupyterhub installation outcome, such as ALB and CloudFront CDK templates. Is there any workaround in CDK so far?

[javydekoning]

I think add_helm_chart() uses a Custom Resource handler (node14.x).

Trusted CA's are hardcoded in NodeJS.

hub.jupyter.org uses GTS:

openssl s_client -servername hub.jupyter.org -connect hub.jupyter.org:443 -showcerts
CONNECTED(00000006)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R4
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 2P2
verify return:1
depth=0 CN = jupyter.org
verify return:1
write W BLOCK
---
Certificate chain
 0 s:/CN=jupyter.org
   i:/C=US/O=Google Trust Services LLC/CN=GTS CA 2P2
-----BEGIN CERTIFICATE-----
MIID2zCCA4KgAwIBAgIQdEwksZMlZA4RcP9cTeZAhTAKBggqhkjOPQQDAjBGMQsw
CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzET
MBEGA1UEAxMKR1RTIENBIDJQMjAeFw0yMzA1MTcwMzMwMDlaFw0yMzA4MTUwMzMw
MDhaMBYxFDASBgNVBAMTC2p1cHl0ZXIub3JnMFkwEwYHKoZIzj0CAQYIKoZIzj0D
AQcDQgAErDLj8FR4nnkG7oOeTI0FbLSyabnuWOPibTK8UcWFeYlmyWB1wvC+vTl6
Um8jBQNe4vN1nqdsIqsx1OqSGxQW2KOCAoAwggJ8MA4GA1UdDwEB/wQEAwIHgDAT
BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTW68dg
uvjVZE8p1uEmGL7I+eU21TAfBgNVHSMEGDAWgBSHI6lQSA4HiVQKcTD2M9IKR/ad
rDB4BggrBgEFBQcBAQRsMGowNQYIKwYBBQUHMAGGKWh0dHA6Ly9vY3NwLnBraS5n
b29nL3MvZ3RzMnAyL1llRUdDNlhEd3ZRMDEGCCsGAQUFBzAChiVodHRwOi8vcGtp
Lmdvb2cvcmVwby9jZXJ0cy9ndHMycDIuZGVyMCUGA1UdEQQeMByCC2p1cHl0ZXIu
b3Jngg0qLmp1cHl0ZXIub3JnMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGCisGAQQB
1nkCBQMwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb2cvZ3Rz
MnAyL2d5OEJRYlFvZVlJLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AK33
vvp8/xDIi509nB4+GGq0Zyldz7EMJMqFhjTr3IKKAAABiCf5ARgAAAQDAEYwRAIg
Q64DtHE7eZ9oA4xvoQBvuyumw+0XwcxafeGxCXXynL0CIGHZqjK8ELGp/Hijg0B+
eNi2uwnkqn/Ej4oTxSRV2cEnAHYAs3N3B+GEUPhjhtYFqdwRCUp5LbFnDAuH3PAD
Dnk2pZoAAAGIJ/kA9wAABAMARzBFAiEA4GFRJPujUAS0Yf4t17h4zMQuR3fx0nlO
E7d7XPxhfCsCIGOmQhhPRPHQWQolPxDmAwB/kN90uS/qRSLtR3aoSqq2MAoGCCqG
SM49BAMCA0cAMEQCIAtM1ANAM+R8W9kguvl4Nsj2B4WSVzO9FqrC0t09YZrwAiAP
YYsiVFCZL5pdGYVE3Jc57WJ1HsqbFqT+ZmEGTi0FDA==
-----END CERTIFICATE-----
 1 s:/C=US/O=Google Trust Services LLC/CN=GTS CA 2P2
   i:/C=US/O=Google Trust Services LLC/CN=GTS Root R4
-----BEGIN CERTIFICATE-----
MIIDITCCAqagAwIBAgINAhZoJeFwBEBhJJH1QDAKBggqhkjOPQQDAzBHMQswCQYD
VQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIG
A1UEAxMLR1RTIFJvb3QgUjQwHhcNMjIxMDA1MDAwMDQyWhcNMjcwOTMwMDAwMDQy
WjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2Vz
IExMQzETMBEGA1UEAxMKR1RTIENBIDJQMjBZMBMGByqGSM49AgEGCCqGSM49AwEH
A0IABKdQkzjAHqOUsb/TkH7cz5lRtD374tNZ8rYrCUb1mxypE+VmCb1Jgzq+93tR
dE78GRzPI4+q6raha1TEyWgoniOjggF2MIIBcjAOBgNVHQ8BAf8EBAMCAYYwHQYD
VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMBIGA1UdEwEB/wQIMAYBAf8CAQAw
HQYDVR0OBBYEFIcjqVBIDgeJVApxMPYz0gpH9p2sMB8GA1UdIwQYMBaAFIBM1ut0
/0k2o9XY/LU+xWrwlB2MMGgGCCsGAQUFBwEBBFwwWjAmBggrBgEFBQcwAYYaaHR0
cDovL29jc3AucGtpLmdvb2cvZ3RzcjQwMAYIKwYBBQUHMAKGJGh0dHA6Ly9wa2ku
Z29vZy9yZXBvL2NlcnRzL2d0c3I0LmRlcjA0BgNVHR8ELTArMCmgJ6AlhiNodHRw
Oi8vY3JsLnBraS5nb29nL2d0c3I0L2d0c3I0LmNybDBNBgNVHSAERjBEMAgGBmeB
DAECATA4BgorBgEEAdZ5AgUDMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vcGtpLmdv
b2cvcmVwb3NpdG9yeS8wCgYIKoZIzj0EAwMDaQAwZgIxAMnbIiQb5fsdexUuVGoB
MVwsDPGd7VC13Y0OBezt7FqFHDwqm8nnVdV/FkNyXNv9/AIxAN51NGqMcbexMOYK
pLC0zXfjNwvqBsZhmzCCQIM6MVyBID0rjjxPu7laIaHqAu6T5Q==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=jupyter.org
issuer=/C=US/O=Google Trust Services LLC/CN=GTS CA 2P2
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 2094 bytes and written 375 bytes
---
New, TLSv1/SSLv3, Cipher is AEAD-CHACHA20-POLY1305-SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : AEAD-CHACHA20-POLY1305-SHA256
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Start Time: 1686656154
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

The hardcoded root cert here: https://github.com/nodejs/node/blob/v14.x/src/node_root_certs.h#L3267-L3279

[dgraeber]

Thanks for more info! Our build (in CodeBuild) is leveraging Node16. I'd hate to modify our CICD to use charts that are embedded (rather than fetching from a repo). Any thoughts as to how to mitigate this via CDK constructs?

[pepito-j]

Doesn't look like it's actually being caused by NodeJS. Checking the handler, it uses Python 3.7 as it's runtime for handling HelmChart:

• https://github.com/aws/aws-cdk/blob/v2.86.0/packages/aws-cdk-lib/aws-eks/lib/helm-chart.ts#L112-L161

• aws-cdk/packages/aws-cdk-lib/aws-eks/lib/kubectl-provider.ts

  Lines 136 to 150 in 1130fab

  	const handler = new lambda.Function(this, 'Handler', {
  	code: lambda.Code.fromAsset(path.join(__dirname, 'kubectl-handler')),
  	runtime: lambda.Runtime.PYTHON_3_7,
  	handler: 'index.handler',
  	timeout: Duration.minutes(15),
  	description: 'onEvent handler for EKS kubectl resource provider',
  	memorySize,
  	environment: cluster.kubectlEnvironment,
  	role: cluster.kubectlLambdaRole ? cluster.kubectlLambdaRole : undefined,
  	// defined only when using private access
  	vpc: cluster.kubectlPrivateSubnets ? cluster.vpc : undefined,
  	securityGroups: cluster.kubectlSecurityGroup ? [cluster.kubectlSecurityGroup] : undefined,
  	vpcSubnets: cluster.kubectlPrivateSubnets ? { subnets: cluster.kubectlPrivateSubnets } : undefined,
  	});

The associated code that runs the helm installation is here:

• aws-cdk/packages/aws-cdk-lib/aws-eks/lib/kubectl-handler/helm/__init__.py

  Lines 160 to 200 in ff21c69

  	def helm(verb, release, chart = None, repo = None, file = None, namespace = None, version = None, wait = False, timeout = None, create_namespace = None, skip_crds = False):
  	import subprocess
  	cmnd = ['helm', verb, release]
  	if not chart is None:
  	cmnd.append(chart)
  	if verb == 'upgrade':
  	cmnd.append('--install')
  	if create_namespace:
  	cmnd.append('--create-namespace')
  	if not repo is None:
  	cmnd.extend(['--repo', repo])
  	if not file is None:
  	cmnd.extend(['--values', file])
  	if not version is None:
  	cmnd.extend(['--version', version])
  	if not namespace is None:
  	cmnd.extend(['--namespace', namespace])
  	if wait:
  	cmnd.append('--wait')
  	if skip_crds:
  	cmnd.append('--skip-crds')
  	if not timeout is None:
  	cmnd.extend(['--timeout', timeout])
  	cmnd.extend(['--kubeconfig', kubeconfig])
  	maxAttempts = 3
  	retry = maxAttempts
  	while retry > 0:
  	try:
  	output = subprocess.check_output(cmnd, stderr=subprocess.STDOUT, cwd=outdir)
  	logger.info(output)
  	return
  	except subprocess.CalledProcessError as exc:
  	output = exc.output
  	if b'Broken pipe' in output:
  	retry = retry - 1
  	logger.info("Broken pipe, retries left: %s" % retry)
  	else:
  	raise Exception(output)
  	raise Exception(f'Operation failed after {maxAttempts} attempts: {output}')

So, it seems like an issue with Python 3.7 certs instead. I updated the onEvent handler function to 3.8 out-of-band and was able to get a successful deployment.

[melodyyangaws]

great finding, thanks @pepito-j ! The python3.7 is already end of life support since last month (Jun 2023), do you think it's caused by the EOL? To unblock us quickly, could you please PR the code change and get the team reviewed? highly appreciate it.

[melodyyangaws]

CDK team, when can we have the fixed rollout? Our upcoming events are impacted, especially the one at the end of the year. Highly appreciate your attention and urgency in addressing the issue.