Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QCL add cluster yaml, config, staging hub #2295

Merged
merged 19 commits into from Mar 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/deploy-hubs.yaml
Expand Up @@ -167,6 +167,7 @@ jobs:
failure_2i2c-aws-us: "${{ env.failure_2i2c-aws-us }}"
failure_ubc-eoas: "${{ env.failure_ubc-eoas }}"
failure_nasa-veda: "${{ env.failure_nasa-veda }}"
failure_qcl: "${{ env.failure_qcl }}"

# Only run this job on pushes to the default branch and when the job output is not
# an empty list
Expand Down
24 changes: 24 additions & 0 deletions config/clusters/qcl/cluster.yaml
@@ -0,0 +1,24 @@
name: qcl
provider: gcp # https://console.cloud.google.com/kubernetes/clusters/details/europe-west1/qcl-cluster/observability?project=qcl-hub
gcp:
key: enc-deployer-credentials.secret.json
project: qcl-hub
cluster: qcl-cluster
# We default to a regional cluster
zone: europe-west1
support:
helm_chart_values_files:
- support.values.yaml
- enc-support.secret.values.yaml
hubs:
- name: staging
# Tip: consider changing this to something more human friendly
display_name: "qcl - staging"
domain: staging.quantifiedcarbon.com
helm_chart: basehub
auth0:
enabled: false
helm_chart_values_files:
- common.values.yaml
- staging.values.yaml
- enc-staging.secret.values.yaml
53 changes: 53 additions & 0 deletions config/clusters/qcl/common.values.yaml
@@ -0,0 +1,53 @@
nfs:
enabled: true
pv:
mountOptions:
- soft
- noatime
# Google FileStore IP
serverIP: 10.155.184.90
# Name of Google Filestore share
baseShareName: /homes/
jupyterhub:
prePuller:
continuous:
enabled: true
hook:
enabled: true
custom:
2i2c:
add_staff_user_ids_to_admin_users: true
add_staff_user_ids_of_type: "github"
homepage:
templateVars:
org:
name: "QuantifiedCarbon"
logo_url: https://avatars.githubusercontent.com/u/124042132?s=400&u=b84f1c7dfd1f9699b2adec7c8eb9ca7b9b2b0a6e&v=4
url: https://quantifiedcarbon.com
designed_by:
name: "2i2c"
url: https://2i2c.org
operated_by:
name: "2i2c"
url: https://2i2c.org
funded_by:
name: ""
url: ""
hub:
allowNamedServers: true
config:
Authenticator:
enable_auth_state: true
# This hub uses GitHub Teams auth and so we don't set
# allowed_users in order to not deny access to valid members of
# the listed teams. These people should have admin access though.
admin_users:
- pnasrat
JupyterHub:
authenticator_class: github
GitHubOAuthenticator:
populate_teams_in_auth_state: true
allowed_organizations:
- 2i2c-org:hub-access-for-2i2c-staff
scope:
- read:org
30 changes: 30 additions & 0 deletions config/clusters/qcl/enc-deployer-credentials.secret.json
@@ -0,0 +1,30 @@
{
"type": "ENC[AES256_GCM,data:J8cYorP4TDrRsyLNO0m/,iv:gNEDWXB5uFWrUcJEvlXBjeh0xOnBcoJDDW4fcn33rf0=,tag:VNZmjp1E0khU/csXb3sYTQ==,type:str]",
"project_id": "ENC[AES256_GCM,data:Rp14YyOc6g==,iv:Vd6O/O+pHiIwGVFIyslC3CrQwhGP3OPKn9UbLmbwQlQ=,tag:MC3wx/oPE1vJp3rqmb1CCg==,type:str]",
"private_key_id": "ENC[AES256_GCM,data:kcX02wdypbmRt4dpMuC0ady6nHMPuPF8OO3codoRWwM8MQ5jGoaMMw==,iv:8XeZ+hDsqKlLBNrDbTqToxJp5nLM06eUEiztdt3CvBc=,tag:j56VjJIGp1lhk3pZZ1M9Qw==,type:str]",
"private_key": "ENC[AES256_GCM,data:WHmnf1PlnbyUBcp2G3SPa9AgKMATisd9N9MIYxNSaMlRQ6u/ySzAJJj12HcSEst7Jeaaia8s1dqsyJfjTNg8yLzJ27onj8kWEYMI4yPpQoFl9cohmcSNXWMEY5W5EvJ+lAZBACFSQ1R5oFe3Kws9MwkGPJmBJBNyyUfrxqBTb/mW3Ro2+GmTcPmDy7ocsupd7PwMtsxuzkGjdgWMd4U0JGBxq/asWhz3nfa01sI+tC9cAszlCkm8gAo5FJox+Ip2iS9TEn/fxV6NZiiq7ge+6f2+0awMZlQTv4gdQcCbyudPARvcSkM7qLhPrbsGadTc3KNsHOepdel6D/LBNDCii+YuOM7lApU9jQvuQMtd+IzRKiJ7xcORE5qI8XYSrwH4Ck7k3vFYlghriAg3yEXnkuWfX1YonSYzaHU7cOn3QCQG98QrNkAo545W55c9BjLb4JPNWQkZKfPnWkbwqE4zpJmJ5P+gDBM53Ws0GwMKoAbSb9OmwzxWJTlOJlAxy6ImoKpF88Xx6US/js41isbo64Tlyr2SmsKiSPlXQiFvj+lKi7a/rXgm0OInkrqvJuBIBexHPV4DgCrCQGgLxBD5W7jyLYlLCnYk5ZiKGHU+TWLAq/Dw8EeV3wSq1BRIQ301gJewowGGUKztyOgNqKb2WacnaIq4wrwOeDUCUhmQoOSfYkRhIE9EyCd5dv3SR8ULRHLnqE8FZUKV11YwaIr8t7V+Xgesr8Fy3wKLUQFF0aeVg3WnALEG9fYSAf29uF1qTA2W1Q9mNnsR9wdnq4ud62gM62NBOmOGbLWnA7uWDdHBoiuj++gI7MxjUXXkiR2430vbX06S6H5wGUwN954SpQ+ZJWOo2GUbfyJhaJ1Sw6Dvh3fhf19zLG1FLQ3iP4SxGhZPZDd9qACLuU832u4/cc7CcEOLuYd4iFzaUIPezWCT3F0OMJb7hsKRPOgzavXr1lcGr8PbdHVoo5yXnHic+g18vjaxxjMuA/Wov29QAtMixJH/JNJPjtv3DHcXUpv92B00p2rQ/wzMgHTO7+7Vd/wHiUCr26m+fyVmGcMthTYKLan8q5HqSUGJKynP0UD/W9s6CCfz8ErxNM0sUGzpBa/W2keU+gS3BZXn82v6FvBdmvl1g0C9a0uOxNaOdxtElVyzFWoG5P32WAbAT0EYcx+sbWukqqecKqG07BnP3K/KBUdt6HNXllhXn+owznvW97b/5ogtWt1/FzXaVNKBWHwhZ9OVIkLUCDtINIUeU05FE0Ao9P/8iEHdHwazhFO2qfGmH8jp6cQOhElLLaKGFy4JQbel/sXMIRn+xnWmiZL1d3OUGOqPFJW4XTPQckSyzghigjl6M9Rz/rErLQqprhZNmOi7haN0EhusFYhKUcXxp0KoXPtqKRYV3K/rYMQO/7qPEp8WIMYXxBPGhbyeFgKi6a0e3hufdXDqwxjoppbo/wtfjyatyJtCKViieRr1KqwZLejPZZFpRZU+TPgw3ZM/DA6B40uOxK5dcwc3ho+JLUHFejrbaEhDhmP6w5LB7AZatIgnWr0ZG4dVnA4aVRpsNf5EMHN/ocTF5szagdryEwnPPEyHTIsA43B69FnKHnaztu8c6q1kB0g4B8L/YK5RridB/eRj6LaFKD+u5LjCJbaPF1N4BUR2pFdIk7V25EA+zHcZcZmqJEJTZM9W7RQ1nyGzwfbRuSnCt7fHXaCNF9UcOLgGFg6FGcEFuwL2kgDcnHEPZEnjPCPqxYAvlHoMFVkZ35wR99a710oMu7EO+a/ryQUvz9FnF0eiffq/Jboh7ZDyHV3WKaxqAUcmJCyllZ0RJZthCAlJOy2fBy1+2TrIgSk0gjXwuzfJ0nAE3AAWlUzGpLxqUnph9uPoxD6uBifBd282gBL2a4O7yH6tCE5Lbg2x5zT/+lG58On2OV0MupY9HerUS0Z1oAvTmY+Vws2iWoX1JBTClLa59NU1/q+58nFNOJaTbHIZquZveRyZXfFbJsPHy3ke5mAvs77VFfibBDSXjHoiW/6LD71YhZbfeYK3EBR6ayal4G73QR+LYzN58waViRah49y6NondaJxB3I83J33mR5j6wrqzG1MYlZ286nSPLbW94pXKymI3uUbhowUasF4CH+FOg3aqWub5YrGJXnT3jV51GkMQklLABUMlXeFl+Awh9zzk4UE3Lsx4ikUWL4IqRaiYtR79D3OSpW7e3zaq7UH/5wC/+lCSPxeAM3+x8YEfFCzQj56MzO7ON15tYxZ8jSLCTUDNZ53vrdS2,iv:laM7J0Yo5UxEQdsmQLOV+QaB7G9TIclTxfERcgzJZuY=,tag:fmb6RDfZjACtbH2bKIXMOw==,type:str]",
"client_email": "ENC[AES256_GCM,data:ea4cRpdyOZAHYXa9ZShwFkxoYZz048eT291DbAariM5XGxI2i7gUz5Y=,iv:kDHKKH8+6TNsQOn1I18DG/lSfazjdoKQjDn3vfki4ZI=,tag:+vl3YsKgdOTuo0TpmWOQ1Q==,type:str]",
"client_id": "ENC[AES256_GCM,data:Symv7H8sdPVMvt1vLHl6WBJ2SAuz,iv:bML8uimAGSb9K0xOaHfDpqrOkuinDHRNrWF27c7Pg0U=,tag:XIyhKL0JEiPBCqCwmRJcJg==,type:str]",
"auth_uri": "ENC[AES256_GCM,data:Sxd/Wdbu1kabpS932/92hxlhBVESwXUiVFo/nzoLMKbd3MmRD6UHs84=,iv:VpNFHVtWqktF9Wn5X3PdZ8DYBN7C1LogljBQJhEMOAs=,tag:98FsqO1BFx+Rgfluuq6j5A==,type:str]",
"token_uri": "ENC[AES256_GCM,data:ZcEiNMOm/kU/kR5zQR4t5vhKn90il3V7nkU5Gj9/7jfYSHo=,iv:RqrqUX7p1rMnBBjgu6+V9hyqU3D3Z7FcBNCyB8jQIvg=,tag:D2HFCerjtfRsiXslU9OVwg==,type:str]",
"auth_provider_x509_cert_url": "ENC[AES256_GCM,data:qBTjTpOOX5WEMX+TXm8h13thtvNagBZcFOslyzyagYtCXf7VBnx1T35s,iv:tD6x5GXTiv9253HoimCDZqUtevqHur7i0m1FSv0cW+0=,tag:8zwBUuPMfZzGnFtS+6Vuqg==,type:str]",
"client_x509_cert_url": "ENC[AES256_GCM,data:P7glJkOYKl7Zvd2gV+pr7ZQVl0mQ9a2IHFegE5hy93Ue+Qwb5/aibySIzX0Rq7lZeeij7nMQzdUMxzyaQHQSXZEU0kaHV7Lm4zQdXLP4eHU1UI6/UqKKwq1QgrGm,iv:tp4cM47+jnEAQuW9gy4JHwYCY/yICxewQeRyYJsMovw=,tag:TfSDQ8sERv5Ow2EN/Zm8DQ==,type:str]",
"sops": {
"kms": null,
"gcp_kms": [
{
"resource_id": "projects/two-eye-two-see/locations/global/keyRings/sops-keys/cryptoKeys/similar-hubs",
"created_at": "2023-03-02T15:10:24Z",
"enc": "CiUA4OM7eOXXC4OaTp56wuQzUscq1hWR5rRUfsRsPI6gP+n8BEQTEkkALQgViLNm3Kbl+x8eWBfrxDeEZrVMNC5gg33R9Nbj6EwtEycIJZu1JrbOTXgSoL5KoAZrsAypTIW7ziJO1V6HU8iMww0gDk6H"
}
],
"azure_kv": null,
"hc_vault": null,
"age": null,
"lastmodified": "2023-03-02T15:10:24Z",
"mac": "ENC[AES256_GCM,data:RXOGJ78vexUV3oJFUSr19X4qIavisXw8SwXp3yjtdVZlUvlK2ADe6x0iF+WOgci0ckqsxVqtKgzFNxdyi4J5wuVx8eScNNGfvz3sMniJi7lVllMS9ljd8zPAocy5yfrTku3CE839/DQ+SAAGENSg0At6mGbdt1WnSvyHsKA+hqE=,iv:ajHRhpTcSXWYEuVh+2KhwAv6tL1/pKyovnHwivYncGo=,tag:PrMqCUbScYAUdCq2Bufb7Q==,type:str]",
"pgp": null,
"unencrypted_suffix": "_unencrypted",
"version": "3.7.3"
}
}
15 changes: 15 additions & 0 deletions config/clusters/qcl/enc-grafana-token.secret.yaml
@@ -0,0 +1,15 @@
grafana_token: ENC[AES256_GCM,data:BJwMKSEZ1/DRbERIYW1uqRE8xMNXbKST7oXnO65p36hZaqmkMyXChxC1HLg17w==,iv:vGnpzE1LuOGQuAtaOiSsCeMVjWm+o+ZI7j5L9IKXa/M=,tag:38R8Ax+f88LKjuCbK2xKFw==,type:str]
sops:
kms: []
gcp_kms:
- resource_id: projects/two-eye-two-see/locations/global/keyRings/sops-keys/cryptoKeys/similar-hubs
created_at: "2023-03-02T19:57:18Z"
enc: CiUA4OM7eFpLfzWMdNgB/fUH1LgVEXJSWjAxQd0roGYd25JYcwI+EkkALQgViMRQd5fIxK/d4ZqZU3PAbOB2Ndb6EZNVI7glaamwpgrTr3iaYxrhygb/Im+qhawxfmJgEdqf4EpNL8llopf+FPHO4W+D
azure_kv: []
hc_vault: []
age: []
lastmodified: "2023-03-02T19:57:18Z"
mac: ENC[AES256_GCM,data:CGlISHcwXcLrtJHDHg7GnOhF5gALlZZbnGPm7Ey1n7FP+ENXqJzjCCXGqbYyJg7VG424qIoe1DqGyBZE7s8RdPMuM2e+Lyu35Pb85C7VfNyvRGH5UhCnfHR74Ml3YZRfb7iO1YsDsmFS6QC9Yj9QhdwoEgm6oRGYp6fdg40sdIE=,iv:kbvz5XeDuq3B31IUVZZ4qL0PEXTsqbsO5y11FT0l7+0=,tag:zrBb8QFu/XQbAtBTx6VpTQ==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.7.3
20 changes: 20 additions & 0 deletions config/clusters/qcl/enc-staging.secret.values.yaml
@@ -0,0 +1,20 @@
jupyterhub:
hub:
config:
GitHubOAuthenticator:
client_id: ENC[AES256_GCM,data:nY+6W1QS/fC4+jT/W+hOchhJItU=,iv:1+JjCJsFcpTqfv2crZ6KzvedUECOdr93nDMnCZ73rIg=,tag:B8adK6ZNkAhUu7CSg66gpg==,type:str]
client_secret: ENC[AES256_GCM,data:AM+G9HSP/AooL2zdDWXyw3A2pAR2Pa5umk5MrkyV1mKUEWC9zOyl1w==,iv:crxe2CbTRNoWuKQLq5aR5h/0YQHH2iqn0KTiRGhf6k8=,tag:sTKAJ6duz3rl1igkjHkdpw==,type:str]
sops:
kms: []
gcp_kms:
- resource_id: projects/two-eye-two-see/locations/global/keyRings/sops-keys/cryptoKeys/similar-hubs
created_at: "2023-03-03T15:23:26Z"
enc: CiUA4OM7eCVreNyL/Kf5UimO1jVFFE6xqoF/BH4DPfTRE7JsKolSEkkALQgViCtecMw1qTbG0Erw/WUl2+/E/ZdOtLftVnh8f77BPNp7vx4GTdlRq0Je9Iq0jPJxXoNIQGkIcimuCN/V8vbcan1flujI
azure_kv: []
hc_vault: []
age: []
lastmodified: "2023-03-03T15:23:27Z"
mac: ENC[AES256_GCM,data:7C3JVgmMLDydJu4O89B72sDqZMHMIH3x/zZ/TpZNF5W287U1Kc3Mk0VXZmrfev1+xx5Eck8OUwLM6tALWoXlWQww2oVVflnOlTn6aWJRGLOMfJKXo6BtLnTvEt1JpqhBKmIejU/Mg1DJFBY88Y1lB5L97tarwX9XmPBsElGe2dI=,iv:yWGBcL3+1II/yHJP8kGKNYBGJeCyZggKkSUugD0hphA=,tag:xR7W95mt12g13O5JOI697A==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.7.3
17 changes: 17 additions & 0 deletions config/clusters/qcl/enc-support.secret.values.yaml
@@ -0,0 +1,17 @@
prometheusIngressAuthSecret:
username: ENC[AES256_GCM,data:iB5bF1oqHG3zAMTf6flHGy82OtBEdH1mJej3nPQNwhpUAYnDAt39GCjUyeL6sNZHplxj7wpjLivlXGajqYxEDQ==,iv:xoVgueVKvSgclTSG46o0p+gImFzp0cu1M4arBwUPtTY=,tag:mysqYMEwFWnDYV/k+VIhCQ==,type:str]
password: ENC[AES256_GCM,data:ZELmo3NoENJ+ilLbSuFZzaZpqHteZFZM60TS9bGvVNVufsBB38AomEEAXI/MpEXeUWMzs/3L+edcYP3Xuq3ygw==,iv:hkT4M+6KfEBUNt2z8JdKtensjq+ExclbtkXGeZn8ib4=,tag:NN2HowJJrJjSY01+FP641w==,type:str]
sops:
kms: []
gcp_kms:
- resource_id: projects/two-eye-two-see/locations/global/keyRings/sops-keys/cryptoKeys/similar-hubs
created_at: "2023-03-02T16:01:07Z"
enc: CiUA4OM7eAzqquM2C5z56TmwD7fjzxa6jfmtcBNBVI2Vd8aiv8VzEkkALQgViCy3VHMyhsi3nLCT15bbr0ZnrAJ74n2ac+GyF8BWUqJbyj+k6YCknA5QS+QL9caLnCeECxq4iBMZwpfgC4V09DLGg3S1
azure_kv: []
hc_vault: []
age: []
lastmodified: "2023-03-02T16:01:07Z"
mac: ENC[AES256_GCM,data:4pf9l0nXSswiwf7W8sOREinFqgX0AvxREpUJkEf5VwdelMz1M3+qYDT20nK4entSe7iUjsu23j0GXsOoNiP5q/SYE6CaVQyp08JSx3c6ajKakTu5bQykYH22SB1F2ZFagma1aTFe+0Isil64hm+Cqq3wF8pEhp2KF1MrYOkn+XQ=,iv:+8JFA3Z4FfEjVdu7jvxFHam25YFBtlJJvlO3T4BvPDU=,tag:l1Bxs6Qb9RQWhXqmYjY4mg==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.7.3
8 changes: 8 additions & 0 deletions config/clusters/qcl/staging.values.yaml
@@ -0,0 +1,8 @@
userServiceAccount:
annotations:
iam.gke.io/gcp-service-account: qcl-staging@qcl-hub.iam.gserviceaccount.com
jupyterhub:
hub:
config:
GitHubOAuthenticator:
oauth_callback_url: https://staging.quantifiedcarbon.com/hub/oauth_callback
35 changes: 35 additions & 0 deletions config/clusters/qcl/support.values.yaml
@@ -0,0 +1,35 @@
prometheusIngressAuthSecret:
enabled: true

redirects:
rules:
- from: staging.qcl.2i2c.cloud
to: staging.quantifiedcarbon.com
Comment on lines +4 to +7
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work figuring this out! I wasn't aware of this system to create Ingress resources was available in the basehub helm chart, and suspect it may not have been documented either yet.

We should have redirect rules for grafana and the production hub here as well. Maybe the production hub is out of scope for this PR and thats fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some documentation here https://infrastructure.2i2c.org/en/latest/howto/manage-domains/index.html

I think when we start work on the deployer improvments making the generator take optional args for custom domains might help

Yes I think this is good to go and I'll split out the prod hub.


prometheus:
server:
ingress:
enabled: true
hosts:
- prometheus.qcl.2i2c.cloud
tls:
- secretName: prometheus-tls
hosts:
- prometheus.qcl.2i2c.cloud

grafana:
grafana.ini:
server:
root_url: https://grafana.quantifiedcarbon.com/
auth.github:
enabled: true
allowed_organizations: 2i2c-org
ingress:
hosts:
- grafana.qcl.2i2c.cloud
- grafana.quantifiedcarbon.com
tls:
- secretName: grafana-tls
hosts:
- grafana.qcl.2i2c.cloud
- grafana.quantifiedcarbon.com
Comment on lines +20 to +35
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered this config quite a bit, thinking that perhaps it was important to list grafana.quantifiedcarbon.com as the topmost host. I think it may not be important based on investigation in #2304 though.

What do you think about going for grafana.quantifiedcarbon.com as the first entry systematically in lists to reduce a risk of issues for now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll read through your investigation on #2304 and this can be updated if need be before we enable the prod hub and handover to the community