QCL add cluster yaml, config, staging hub #2295

pnasrat · 2023-03-02T16:18:36Z

Add cluster file for QCL hub
Generate QCL cluster config and support secrets
See [QCL] Tracking: New Hub Deployment #2287

consideRatio

Some notes about domain names

.github/workflows/deploy-hubs.yaml

config/clusters/qcl/cluster.yaml

config/clusters/qcl/support.values.yaml

pnasrat · 2023-03-02T16:35:03Z

Getting API rate limited but local validate fails. Will fix up after lunch

deployer validate qcl
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://jupyterhub.github.io/helm-chart/" chart repository
Saving 1 charts
Downloading jupyterhub from repo https://jupyterhub.github.io/helm-chart/
Deleting outdated charts
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://helm.dask.org/" chart repository
Saving 2 charts
Downloading dask-gateway from repo https://helm.dask.org/
Deleting outdated charts
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://helm.dask.org/" chart repository
...Successfully got an update from the "https://jupyterhub.github.io/helm-chart/" chart repository
Saving 2 charts
Downloading binderhub from repo https://jupyterhub.github.io/helm-chart/
Downloading dask-gateway from repo https://helm.dask.org/
Deleting outdated charts
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://yuvipanda.github.io/cryptnono/" chart repository
...Successfully got an update from the "https://kubernetes.github.io/autoscaler" chart repository
...Successfully got an update from the "https://kubernetes.github.io/ingress-nginx" chart repository
...Successfully got an update from the "https://grafana.github.io/helm-charts" chart repository
...Successfully got an update from the "https://prometheus-community.github.io/helm-charts" chart repository
Saving 5 charts
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Downloading ingress-nginx from repo https://kubernetes.github.io/ingress-nginx
Downloading cluster-autoscaler from repo https://kubernetes.github.io/autoscaler
Downloading cryptnono from repo https://yuvipanda.github.io/cryptnono/
Deleting outdated charts
Validating non-encrypted support values files for qcl...
1 / 1: Validating non-encrypted hub values files for staging...
Error: failed to parse /home/pnasrat/workspace/src/github.com/2i2c-org/infrastructure/config/clusters/qcl/staging.values.yaml: error converting YAML to JSON: yaml: line 4: mapping values are not allowed in this context

pnasrat · 2023-03-02T16:41:08Z

Note to self validate runs

helm template /home/pnasrat/workspace/src/github.com/2i2c-org/infrastructure/helm-charts/basehub --values=/home/pnasrat/workspace/src/github.com/2i2c-org/infrastructure/config/clusters/qcl/common.values.yaml --values=/home/pnasrat/workspace/src/github.com/2i2c-org/infrastructure/config/clusters/qcl/staging.values.yaml

pnasrat · 2023-03-02T16:49:47Z

Sigh obvious typo now validate fails with

Error: values don't meet the specifications of the schema(s) in the following chart(s):
basehub:
- (root): Additional property basehub is not allowed

consideRatio · 2023-03-02T16:53:09Z

When using basehub, your values should not be indented under basehub.

The daskhub chart dependa on basehub, so basehub chart config is nested under such key for a daskhub chart

pnasrat · 2023-03-03T14:39:32Z

Tested TLS working correctly via

curl -I https://grafana.qcl.2i2c.cloud
HTTP/2 302 
date: Fri, 03 Mar 2023 14:35:30 GMT
content-type: text/html; charset=utf-8
cache-control: no-cache
expires: -1
location: /login
pragma: no-cache
x-content-type-options: nosniff
x-frame-options: deny
x-xss-protection: 1; mode=block
strict-transport-security: max-age=15724800; includeSubDomains

curl -I https://grafana.quantifiedcarbon.com
HTTP/2 302 
date: Fri, 03 Mar 2023 14:35:45 GMT
content-type: text/html; charset=utf-8
cache-control: no-cache
expires: -1
location: /login
pragma: no-cache
x-content-type-options: nosniff
x-frame-options: deny
x-xss-protection: 1; mode=block
strict-transport-security: max-age=15724800; includeSubDomains

for more information, see https://pre-commit.ci

Typo Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>

Domain name fix Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>

Domain name fixes Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>

See #2287

for more information, see https://pre-commit.ci

Based off practices in utoronto hub and that the hubs reference table only uses the 2i2c.cloud urls so should support both Note as we use grafana hub ingress rule that is handled differently to the support helm-chart redirect template for hub urls

for more information, see https://pre-commit.ci

pnasrat · 2023-03-03T14:58:21Z

Actually need to add enc-staging values file before review

pnasrat · 2023-03-03T15:59:35Z

Deployed staging hub here https://staging.quantifiedcarbon.com/hub/login?next=%2Fhub%2F I may need to get the auth wired in

for more information, see https://pre-commit.ci

pnasrat · 2023-03-03T16:36:57Z

Ready for review PTAL. Staging hub is up from a manual deploy. Then will do further configuration under main tracking issue.

consideRatio

Nice work this looks great!!

Things I figure remains in our out of this PR are:

Redirects for prod / grafana
Prod hub
admin_users to include listed accounts from QCL
singleuser.profileList configuration

consideRatio · 2023-03-05T12:04:06Z

config/clusters/qcl/support.values.yaml

+grafana:
+  grafana.ini:
+    server:
+      root_url: https://grafana.quantifiedcarbon.com/
+  auth.github:
+    enabled: true
+    allowed_organizations: 2i2c-org
+  ingress:
+    hosts:
+      - grafana.qcl.2i2c.cloud
+      - grafana.quantifiedcarbon.com
+    tls:
+      - secretName: grafana-tls
+        hosts:
+          - grafana.qcl.2i2c.cloud
+          - grafana.quantifiedcarbon.com


I considered this config quite a bit, thinking that perhaps it was important to list grafana.quantifiedcarbon.com as the topmost host. I think it may not be important based on investigation in #2304 though.

What do you think about going for grafana.quantifiedcarbon.com as the first entry systematically in lists to reduce a risk of issues for now?

I'll read through your investigation on #2304 and this can be updated if need be before we enable the prod hub and handover to the community

consideRatio · 2023-03-05T12:06:37Z

config/clusters/qcl/support.values.yaml

+redirects:
+  rules:
+    - from: staging.qcl.2i2c.cloud
+      to: staging.quantifiedcarbon.com


Nice work figuring this out! I wasn't aware of this system to create Ingress resources was available in the basehub helm chart, and suspect it may not have been documented either yet.

We should have redirect rules for grafana and the production hub here as well. Maybe the production hub is out of scope for this PR and thats fine.

There is some documentation here https://infrastructure.2i2c.org/en/latest/howto/manage-domains/index.html

I think when we start work on the deployer improvments making the generator take optional args for custom domains might help

Yes I think this is good to go and I'll split out the prod hub.

github-actions · 2023-03-06T13:32:50Z

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/4343972840

pnasrat self-assigned this Mar 2, 2023

pnasrat force-pushed the add-qcl-hub branch from 300e86a to e15ef6a Compare March 2, 2023 16:26

consideRatio reviewed Mar 2, 2023

View reviewed changes

.github/workflows/deploy-hubs.yaml Outdated Show resolved Hide resolved

config/clusters/qcl/cluster.yaml Outdated Show resolved Hide resolved

config/clusters/qcl/support.values.yaml Outdated Show resolved Hide resolved

pnasrat and others added 13 commits March 3, 2023 09:42

Add cluster file for QCL hub

8cd8c63

Generate QCL cluster config and support secrets

89a822a

[pre-commit.ci] auto fixes from pre-commit.com hooks

32e6c0c

for more information, see https://pre-commit.ci

Update .github/workflows/deploy-hubs.yaml

d32994a

Typo Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>

Update config/clusters/qcl/cluster.yaml

2b61464

Domain name fix Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>

Update config/clusters/qcl/support.values.yaml

ecb3080

Domain name fixes Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>

Fix copy and paste error in staging values

6ea9d26

QCL Fix validation failures on yaml

ef3918c

See #2287

[pre-commit.ci] auto fixes from pre-commit.com hooks

08c51e4

for more information, see https://pre-commit.ci

Use 2i2c.cloud for now

0ae85c0

QCL add grafana token

3bd7572

[pre-commit.ci] auto fixes from pre-commit.com hooks

3c936c9

for more information, see https://pre-commit.ci

pnasrat force-pushed the add-qcl-hub branch from ad61147 to 3c936c9 Compare March 3, 2023 14:42

pnasrat marked this pull request as ready for review March 3, 2023 14:54

Add staging hub secrets

4fb4731

pnasrat marked this pull request as draft March 3, 2023 15:45

Add NFS homedirs created by terraform

168aa91

pnasrat marked this pull request as ready for review March 3, 2023 15:58

Add qcl cluster hub oauth for 2i2c staff

2cf3728

pnasrat added 2 commits March 3, 2023 11:19

Wire in GitHubOAuthenticator for QCL staging

35960a2

Fix indentation level for qcl hub auth config

504cc4b

pnasrat force-pushed the add-qcl-hub branch from 98a5935 to 504cc4b Compare March 3, 2023 16:31

[pre-commit.ci] auto fixes from pre-commit.com hooks

6c1dc72

for more information, see https://pre-commit.ci

consideRatio approved these changes Mar 5, 2023

View reviewed changes

pnasrat merged commit 6546c28 into master Mar 6, 2023

pnasrat deleted the add-qcl-hub branch March 6, 2023 13:32

pnasrat mentioned this pull request Mar 6, 2023

[QCL] Tracking: New Hub Deployment #2287

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QCL add cluster yaml, config, staging hub #2295

QCL add cluster yaml, config, staging hub #2295

pnasrat commented Mar 2, 2023

consideRatio left a comment

pnasrat commented Mar 2, 2023

pnasrat commented Mar 2, 2023

pnasrat commented Mar 2, 2023

consideRatio commented Mar 2, 2023

pnasrat commented Mar 3, 2023

pnasrat commented Mar 3, 2023

pnasrat commented Mar 3, 2023

pnasrat commented Mar 3, 2023

consideRatio left a comment •

edited

Loading

consideRatio Mar 5, 2023

pnasrat Mar 6, 2023

consideRatio Mar 5, 2023

pnasrat Mar 6, 2023

github-actions bot commented Mar 6, 2023

QCL add cluster yaml, config, staging hub #2295

QCL add cluster yaml, config, staging hub #2295

Conversation

pnasrat commented Mar 2, 2023

consideRatio left a comment

Choose a reason for hiding this comment

pnasrat commented Mar 2, 2023

pnasrat commented Mar 2, 2023

pnasrat commented Mar 2, 2023

consideRatio commented Mar 2, 2023

pnasrat commented Mar 3, 2023

pnasrat commented Mar 3, 2023

pnasrat commented Mar 3, 2023

pnasrat commented Mar 3, 2023

consideRatio left a comment • edited Loading

Choose a reason for hiding this comment

consideRatio Mar 5, 2023

Choose a reason for hiding this comment

pnasrat Mar 6, 2023

Choose a reason for hiding this comment

consideRatio Mar 5, 2023

Choose a reason for hiding this comment

pnasrat Mar 6, 2023

Choose a reason for hiding this comment

github-actions bot commented Mar 6, 2023

consideRatio left a comment •

edited

Loading