[lambda-JupyterHub]$ kubectl get all -n jupyter-hub NAME READY STATUS RESTARTS AGE pod/continuous-image-puller-j5zrq 1/1 Running 0 15m pod/continuous-image-puller-qzk5w 1/1 Running 0 15m pod/hub-5bd88fcc58-cpjhg 0/1 Pending 0 15m pod/nfs-subdir-external-provisioner-c9c66d897-7kq2c 1/1 Running 0 27m pod/proxy-6f67bc4876-nmn5f 1/1 Running 0 15m pod/user-scheduler-c697d4bd5-pwcs8 1/1 Running 0 15m pod/user-scheduler-c697d4bd5-vl2g5 1/1 Running 0 15m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/hub ClusterIP 10.99.219.62 8081/TCP 15m service/proxy-api ClusterIP 10.111.38.96 8001/TCP 15m service/proxy-public NodePort 10.97.192.58 80:31471/TCP 15m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/continuous-image-puller 2 2 2 2 2 15m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/hub 0/1 1 0 15m deployment.apps/nfs-subdir-external-provisioner 1/1 1 1 27m deployment.apps/proxy 1/1 1 1 15m deployment.apps/user-scheduler 2/2 2 2 15m NAME DESIRED CURRENT READY AGE replicaset.apps/hub-5bd88fcc58 1 1 0 15m replicaset.apps/nfs-subdir-external-provisioner-c9c66d897 1 1 1 27m replicaset.apps/proxy-6f67bc4876 1 1 1 15m replicaset.apps/user-scheduler-c697d4bd5 2 2 2 15m NAME READY AGE statefulset.apps/user-placeholder 0/0 15m [lambda-JupyterHub]$ [lambda-JupyterHub]$ [lambda-JupyterHub]$ kubectl describe deployment -n jupyter-hub Name: hub Namespace: jupyter-hub CreationTimestamp: Fri, 28 May 2021 16:17:02 -0700 Labels: app=jupyterhub app.kubernetes.io/managed-by=Helm chart=jupyterhub-0.11.1 component=hub heritage=Helm release=jupyterhub Annotations: deployment.kubernetes.io/revision: 1 meta.helm.sh/release-name: jupyterhub meta.helm.sh/release-namespace: jupyter-hub Selector: app=jupyterhub,component=hub,release=jupyterhub Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable StrategyType: Recreate MinReadySeconds: 0 Pod Template: Labels: app=jupyterhub component=hub hub.jupyter.org/network-access-proxy-api=true hub.jupyter.org/network-access-proxy-http=true hub.jupyter.org/network-access-singleuser=true release=jupyterhub Annotations: checksum/config-map: b4b431e946d3d5314644938daac16c1cdda64abd870706d325264a1c215bdcd2 checksum/secret: ca4f035220c45f32e184e9b49e8316723a4097813b1f302e9d87094b9ee8cfca Service Account: hub Containers: hub: Image: jupyterhub/k8s-hub:0.11.1 Port: 8081/TCP Host Port: 0/TCP Args: jupyterhub --config /etc/jupyterhub/jupyterhub_config.py --upgrade-db Requests: cpu: 200m memory: 512Mi Liveness: http-get http://:http/hub/health delay=300s timeout=3s period=10s #success=1 #failure=30 Readiness: http-get http://:http/hub/health delay=0s timeout=1s period=2s #success=1 #failure=1000 Environment: PYTHONUNBUFFERED: 1 HELM_RELEASE_NAME: jupyterhub POD_NAMESPACE: (v1:metadata.namespace) CONFIGPROXY_AUTH_TOKEN: Optional: false Mounts: /etc/jupyterhub/config/ from config (rw) /etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py") /etc/jupyterhub/secret/ from secret (rw) /etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py") /srv/jupyterhub from hub-db-dir (rw) Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: hub-config Optional: false secret: Type: Secret (a volume populated by a Secret) SecretName: hub-secret Optional: false hub-db-dir: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: hub-db-dir ReadOnly: false Conditions: Type Status Reason ---- ------ ------ Available False MinimumReplicasUnavailable Progressing False ProgressDeadlineExceeded OldReplicaSets: hub-5bd88fcc58 (1/1 replicas created) NewReplicaSet: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 15m deployment-controller Scaled up replica set hub-5bd88fcc58 to 1 Name: nfs-subdir-external-provisioner Namespace: jupyter-hub CreationTimestamp: Fri, 28 May 2021 16:05:27 -0700 Labels: app=nfs-subdir-external-provisioner app.kubernetes.io/managed-by=Helm chart=nfs-subdir-external-provisioner-4.0.10 heritage=Helm release=nfs-subdir-external-provisioner Annotations: deployment.kubernetes.io/revision: 1 meta.helm.sh/release-name: nfs-subdir-external-provisioner meta.helm.sh/release-namespace: jupyter-hub Selector: app=nfs-subdir-external-provisioner,release=nfs-subdir-external-provisioner Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: Recreate MinReadySeconds: 0 Pod Template: Labels: app=nfs-subdir-external-provisioner release=nfs-subdir-external-provisioner Service Account: nfs-subdir-external-provisioner Containers: nfs-subdir-external-provisioner: Image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2 Port: Host Port: Environment: PROVISIONER_NAME: cluster.local/nfs-subdir-external-provisioner NFS_SERVER: 10.26.250.16 NFS_PATH: /BOS_NAS/Kubernetes_Shared/JupyterHub Mounts: /persistentvolumes from nfs-subdir-external-provisioner-root (rw) Volumes: nfs-subdir-external-provisioner-root: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 10.26.250.16 Path: /BOS_NAS/Kubernetes_Shared/JupyterHub ReadOnly: false Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: NewReplicaSet: nfs-subdir-external-provisioner-c9c66d897 (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 27m deployment-controller Scaled up replica set nfs-subdir-external-provisioner-c9c66d897 to 1 Name: proxy Namespace: jupyter-hub CreationTimestamp: Fri, 28 May 2021 16:17:02 -0700 Labels: app=jupyterhub app.kubernetes.io/managed-by=Helm chart=jupyterhub-0.11.1 component=proxy heritage=Helm release=jupyterhub Annotations: deployment.kubernetes.io/revision: 1 meta.helm.sh/release-name: jupyterhub meta.helm.sh/release-namespace: jupyter-hub Selector: app=jupyterhub,component=proxy,release=jupyterhub Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: Recreate MinReadySeconds: 0 Pod Template: Labels: app=jupyterhub component=proxy hub.jupyter.org/network-access-hub=true hub.jupyter.org/network-access-singleuser=true release=jupyterhub Annotations: checksum/hub-secret: 11092faca330272acee39d6a22aea7246f70b727cb5cd85154c9288ddedd31f0 checksum/proxy-secret: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b Containers: chp: Image: jupyterhub/configurable-http-proxy:4.2.2 Ports: 8000/TCP, 8001/TCP Host Ports: 0/TCP, 0/TCP Command: configurable-http-proxy --ip=:: --api-ip=:: --api-port=8001 --default-target=http://hub:$(HUB_SERVICE_PORT) --error-target=http://hub:$(HUB_SERVICE_PORT)/hub/error --port=8000 Requests: cpu: 200m memory: 512Mi Liveness: http-get http://:http/_chp_healthz delay=60s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:http/_chp_healthz delay=0s timeout=1s period=2s #success=1 #failure=3 Environment: CONFIGPROXY_AUTH_TOKEN: Optional: false Mounts: Volumes: Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: NewReplicaSet: proxy-6f67bc4876 (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 15m deployment-controller Scaled up replica set proxy-6f67bc4876 to 1 Name: user-scheduler Namespace: jupyter-hub CreationTimestamp: Fri, 28 May 2021 16:17:02 -0700 Labels: app=jupyterhub app.kubernetes.io/managed-by=Helm chart=jupyterhub-0.11.1 component=user-scheduler heritage=Helm release=jupyterhub Annotations: deployment.kubernetes.io/revision: 1 meta.helm.sh/release-name: jupyterhub meta.helm.sh/release-namespace: jupyter-hub Selector: app=jupyterhub,component=user-scheduler,release=jupyterhub Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=jupyterhub component=user-scheduler release=jupyterhub Annotations: checksum/config-map: 9b6ba50260d0ef3a6196a64dd0033580d072bc71c4b83bc729f9f7350f4413d4 Service Account: user-scheduler Containers: user-scheduler: Image: k8s.gcr.io/kube-scheduler:v1.19.7 Port: Host Port: Command: /usr/local/bin/kube-scheduler --config=/etc/user-scheduler/config.yaml --authentication-skip-lookup=true --v=4 Requests: cpu: 50m memory: 256Mi Liveness: http-get http://:10251/healthz delay=15s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:10251/healthz delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: Mounts: /etc/user-scheduler from config (rw) Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: user-scheduler Optional: false Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: NewReplicaSet: user-scheduler-c697d4bd5 (2/2 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 15m deployment-controller Scaled up replica set user-scheduler-c697d4bd5 to 2 [lambda-JupyterHub]$ [lambda-JupyterHub]$ cat jupyterhub.yaml # custom can contain anything you want to pass to the hub pod, as all passed # Helm template values will be made available there. custom: {} # imagePullSecret is configuration to create a k8s Secret that Helm chart's pods # can get credentials from to pull their images. imagePullSecret: create: false automaticReferenceInjection: true registry: '' username: '' email: '' password: '' # imagePullSecrets is configuration to reference the k8s Secret resources the # Helm chart's pods can get credentials from to pull their images. imagePullSecrets: [] # hub relates to the hub pod, responsible for running JupyterHub, its configured # Authenticator class KubeSpawner, and its configured Proxy class # ConfigurableHTTPProxy. KubeSpawner creates the user pods, and # ConfigurableHTTPProxy speaks with the actual ConfigurableHTTPProxy server in # the proxy pod. hub: config: JupyterHub: admin_access: true authenticator_class: dummy service: type: ClusterIP annotations: {} ports: nodePort: loadBalancerIP: baseUrl: / cookieSecret: publicURL: initContainers: [] fsGid: 1000 nodeSelector: {} tolerations: [] concurrentSpawnLimit: 64 consecutiveFailureLimit: 5 activeServerLimit: deploymentStrategy: ## type: Recreate ## - sqlite-pvc backed hubs require the Recreate deployment strategy as a ## typical PVC storage can only be bound to one pod at the time. ## - JupyterHub isn't designed to support being run in parallell. More work ## needs to be done in JupyterHub itself for a fully highly available (HA) ## deployment of JupyterHub on k8s is to be possible. type: Recreate db: type: sqlite-pvc upgrade: pvc: annotations: {} selector: {} accessModes: - ReadWriteOnce storage: 1Gi subPath: storageClassName: url: password: labels: {} annotations: {} command: [] args: [] extraConfig: {} extraConfigMap: {} extraEnv: {} extraContainers: [] extraVolumes: [] extraVolumeMounts: [] image: name: jupyterhub/k8s-hub tag: '0.11.1' pullPolicy: '' pullSecrets: [] resources: requests: cpu: 200m memory: 512Mi containerSecurityContext: runAsUser: 1000 runAsGroup: 1000 allowPrivilegeEscalation: false services: {} pdb: enabled: false minAvailable: 1 networkPolicy: enabled: true ingress: [] ## egress for JupyterHub already includes Kubernetes internal DNS and ## access to the proxy, but can be restricted further, but ensure to allow ## access to the Kubernetes API server that couldn't be pinned ahead of ## time. ## ## ref: https://stackoverflow.com/a/59016417/2220152 egress: - to: - ipBlock: cidr: 0.0.0.0/0 interNamespaceAccessLabels: ignore allowedIngressPorts: [] allowNamedServers: false namedServerLimitPerUser: authenticatePrometheus: redirectToServer: shutdownOnLogout: templatePaths: [] templateVars: {} livenessProbe: # The livenessProbe's aim to give JupyterHub sufficient time to startup but # be able to restart if it becomes unresponsive for ~5 min. enabled: true initialDelaySeconds: 300 periodSeconds: 10 failureThreshold: 30 timeoutSeconds: 3 readinessProbe: # The readinessProbe's aim is to provide a successful startup indication, # but following that never become unready before its livenessProbe fail and # restarts it if needed. To become unready following startup serves no # purpose as there are no other pod to fallback to in our non-HA deployment. enabled: true initialDelaySeconds: 0 periodSeconds: 2 failureThreshold: 1000 timeoutSeconds: 1 existingSecret: rbac: enabled: true # proxy relates to the proxy pod, the proxy-public service, and the autohttps # pod and proxy-http service. proxy: secretToken: '96dcb17345ddde6a41d200b0e96379a25bbb96f36b438a88879adab0285ec52d' annotations: {} deploymentStrategy: ## type: Recreate ## - JupyterHub's interaction with the CHP proxy becomes a lot more robust ## with this configuration. To understand this, consider that JupyterHub ## during startup will interact a lot with the k8s service to reach a ## ready proxy pod. If the hub pod during a helm upgrade is restarting ## directly while the proxy pod is making a rolling upgrade, the hub pod ## could end up running a sequence of interactions with the old proxy pod ## and finishing up the sequence of interactions with the new proxy pod. ## As CHP proxy pods carry individual state this is very error prone. One ## outcome when not using Recreate as a strategy has been that user pods ## have been deleted by the hub pod because it considered them unreachable ## as it only configured the old proxy pod but not the new before trying ## to reach them. type: Recreate ## rollingUpdate: ## - WARNING: ## This is required to be set explicitly blank! Without it being ## explicitly blank, k8s will let eventual old values under rollingUpdate ## remain and then the Deployment becomes invalid and a helm upgrade would ## fail with an error like this: ## ## UPGRADE FAILED ## Error: Deployment.apps "proxy" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate' ## Error: UPGRADE FAILED: Deployment.apps "proxy" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate' rollingUpdate: # service relates to the proxy-public service service: type: NodePort ports: - port: 80 targetPort: 80 nodePort: 30082 # labels: {} # annotations: {} # nodePorts: # http: # https: # extraPorts: [] # loadBalancerIP: # loadBalancerSourceRanges: [] # chp relates to the proxy pod, which is responsible for routing traffic based # on dynamic configuration sent from JupyterHub to CHP's REST API. chp: containerSecurityContext: runAsUser: 65534 # nobody user runAsGroup: 65534 # nobody group allowPrivilegeEscalation: false image: name: jupyterhub/configurable-http-proxy tag: 4.2.2 pullPolicy: '' pullSecrets: [] extraCommandLineFlags: [] livenessProbe: enabled: true initialDelaySeconds: 60 periodSeconds: 10 readinessProbe: enabled: true initialDelaySeconds: 0 periodSeconds: 2 failureThreshold: 1000 resources: requests: cpu: 200m memory: 512Mi extraEnv: {} nodeSelector: {} tolerations: [] networkPolicy: enabled: true ingress: [] egress: - to: - ipBlock: cidr: 0.0.0.0/0 interNamespaceAccessLabels: ignore allowedIngressPorts: [http, https] pdb: enabled: false minAvailable: 1 # traefik relates to the autohttps pod, which is responsible for TLS # termination when proxy.https.type=letsencrypt. traefik: containerSecurityContext: runAsUser: 65534 # nobody user runAsGroup: 65534 # nobody group allowPrivilegeEscalation: false image: name: traefik # If you bump the tag for Traefik, please help out by keeping an eye out # for intermittency failures in the test suite on acquiring HTTPS # certificates as there has been a lot of struggles with in this in the # past. For example: # https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/1857. tag: v2.3.7 # ref: https://hub.docker.com/_/traefik?tab=tags pullPolicy: '' pullSecrets: [] hsts: includeSubdomains: false preload: false maxAge: 15724800 # About 6 months resources: {} extraEnv: {} extraVolumes: [] extraVolumeMounts: [] extraStaticConfig: {} extraDynamicConfig: {} nodeSelector: {} tolerations: [] extraPorts: [] networkPolicy: enabled: true ingress: [] egress: - to: - ipBlock: cidr: 0.0.0.0/0 interNamespaceAccessLabels: ignore allowedIngressPorts: [http, https] pdb: enabled: false minAvailable: 1 secretSync: containerSecurityContext: runAsUser: 65534 # nobody user runAsGroup: 65534 # nobody group allowPrivilegeEscalation: false image: name: jupyterhub/k8s-secret-sync tag: '0.11.1' pullPolicy: '' pullSecrets: [] resources: {} labels: {} https: enabled: false type: letsencrypt #type: letsencrypt, manual, offload, secret letsencrypt: contactEmail: '' # Specify custom server here (https://acme-staging-v02.api.letsencrypt.org/directory) to hit staging LE acmeServer: https://acme-v02.api.letsencrypt.org/directory manual: key: cert: secret: name: '' key: tls.key crt: tls.crt hosts: [] # singleuser relates to the configuration of KubeSpawner which runs in the hub # pod, and its spawning of user pods such as jupyter-myusername. singleuser: podNameTemplate: extraTolerations: [] nodeSelector: {} extraNodeAffinity: required: [] preferred: [] extraPodAffinity: required: [] preferred: [] extraPodAntiAffinity: required: [] preferred: [] networkTools: image: name: jupyterhub/k8s-network-tools tag: '0.11.1' pullPolicy: '' pullSecrets: [] cloudMetadata: # block set to true will append a privileged initContainer using the # iptables to block the sensitive metadata server at the provided ip. blockWithIptables: true ip: 169.254.169.254 networkPolicy: enabled: true ingress: [] egress: # Required egress to communicate with the hub and DNS servers will be # augmented to these egress rules. # # This default rule explicitly allows all outbound traffic from singleuser # pods, except to a typical IP used to return metadata that can be used by # someone with malicious intent. - to: - ipBlock: cidr: 0.0.0.0/0 except: - 169.254.169.254/32 interNamespaceAccessLabels: ignore allowedIngressPorts: [] events: true extraAnnotations: {} extraLabels: hub.jupyter.org/network-access-hub: 'true' extraEnv: {} lifecycleHooks: {} initContainers: [] extraContainers: [] uid: 1000 fsGid: 100 serviceAccountName: storage: type: dynamic extraLabels: {} extraVolumes: [] extraVolumeMounts: [] static: pvcName: subPath: '{username}' capacity: 10Gi homeMountPath: /home/jovyan dynamic: storageClass: nfs-client pvcNameTemplate: claim-{username}{servername} volumeNameTemplate: volume-{username}{servername} storageAccessModes: [ReadWriteOnce] image: name: jupyterhub/k8s-singleuser-sample tag: '0.11.1' pullPolicy: '' pullSecrets: [] startTimeout: 300 cpu: limit: guarantee: memory: limit: guarantee: 1G extraResource: limits: {} guarantees: {} cmd: jupyterhub-singleuser defaultUrl: extraPodConfig: {} profileList: [] # scheduling relates to the user-scheduler pods and user-placeholder pods. scheduling: userScheduler: enabled: true replicas: 2 logLevel: 4 # plugins ref: https://kubernetes.io/docs/reference/scheduling/config/#scheduling-plugins-1 plugins: score: disabled: - name: SelectorSpread - name: TaintToleration - name: PodTopologySpread - name: NodeResourcesBalancedAllocation - name: NodeResourcesLeastAllocated # Disable plugins to be allowed to enable them again with a different # weight and avoid an error. - name: NodePreferAvoidPods - name: NodeAffinity - name: InterPodAffinity - name: ImageLocality enabled: - name: NodePreferAvoidPods weight: 161051 - name: NodeAffinity weight: 14631 - name: InterPodAffinity weight: 1331 - name: NodeResourcesMostAllocated weight: 121 - name: ImageLocality weight: 11 containerSecurityContext: runAsUser: 65534 # nobody user runAsGroup: 65534 # nobody group allowPrivilegeEscalation: false image: # IMPORTANT: Bumping the minor version of this binary should go hand in # hand with an inspection of the user-scheduelrs RBAC resources # that we have forked. name: k8s.gcr.io/kube-scheduler tag: v1.19.7 pullPolicy: '' pullSecrets: [] nodeSelector: {} tolerations: [] pdb: enabled: true minAvailable: 1 resources: requests: cpu: 50m memory: 256Mi podPriority: enabled: false globalDefault: false defaultPriority: 0 userPlaceholderPriority: -10 userPlaceholder: enabled: true replicas: 0 containerSecurityContext: runAsUser: 65534 # nobody user runAsGroup: 65534 # nobody group allowPrivilegeEscalation: false corePods: nodeAffinity: matchNodePurpose: prefer userPods: nodeAffinity: matchNodePurpose: prefer # prePuller relates to the hook|continuous-image-puller DaemonsSets prePuller: annotations: {} resources: requests: cpu: 0 memory: 0 containerSecurityContext: runAsUser: 65534 # nobody user runAsGroup: 65534 # nobody group allowPrivilegeEscalation: false extraTolerations: [] # hook relates to the hook-image-awaiter Job and hook-image-puller DaemonSet hook: enabled: true # image and the configuration below relates to the hook-image-awaiter Job image: name: jupyterhub/k8s-image-awaiter tag: '0.11.1' pullPolicy: '' pullSecrets: [] containerSecurityContext: runAsUser: 65534 # nobody user runAsGroup: 65534 # nobody group allowPrivilegeEscalation: false podSchedulingWaitDuration: 10 nodeSelector: {} tolerations: [] resources: requests: cpu: 0 memory: 0 continuous: enabled: true pullProfileListImages: true extraImages: {} pause: containerSecurityContext: runAsUser: 65534 # nobody user runAsGroup: 65534 # nobody group allowPrivilegeEscalation: false image: name: k8s.gcr.io/pause tag: '3.2' # https://console.cloud.google.com/gcr/images/google-containers/GLOBAL/pause?gcrImageListsize=30 pullPolicy: '' pullSecrets: [] ingress: enabled: false annotations: {} hosts: [] pathSuffix: '' tls: [] cull: enabled: true users: false removeNamedServers: false timeout: 3600 every: 600 concurrency: 10 maxAge: 0 debug: enabled: false global: {} [lambda-JupyterHub]$