Instructions for GCP are not working for machine-type = f1-micro #1375
Comments
|
Could it be due to insufficient RAM or some other resource? What does |
|
I create the a new cluster, and then when I run I then tried to understand why they're not ready by running |
Almost all the memory is used, and f1-micro is limited to an average of 0.2 CPU which has been exceeded just by the k8s system pods: https://cloud.google.com/compute/docs/machine-types#sharedcore |
|
Thanks @manics, just so I understand, are you suggesting it might not be possible to run kubernetes on f1-micro? Tonight when I tried, the nodes were momentarily ready, but then running nscozzaro1@cloudshell:~ (studyhub-jupyterlab)$ gcloud container clusters create --machine-type f1-micro --num-nodes 3 --zone us-central1-b --cluster-version 1.13.7-gke.24 studyhub-micro
Creating cluster studyhub-micro in us-central1-b... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1/projects/studyhub-jupyterlab/zones/us-central1-b/clusters/studyhub-micro].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-central1-b/studyhub-micro?project=studyhub-jupyterlab
kubeconfig entry generated for studyhub-micro.
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
studyhub-micro us-central1-b 1.13.7-gke.24 34.66.217.57 f1-micro 1.13.7-gke.24 3 RUNNING
nscozzaro1@cloudshell:~ (studyhub-jupyterlab)$ kubectl get node
NAME STATUS ROLES AGE VERSION
gke-studyhub-micro-default-pool-d1bf9b21-2x12 Ready <none> 21s v1.13.7-gke.24
gke-studyhub-micro-default-pool-d1bf9b21-pg4c Ready <none> 21s v1.13.7-gke.24
gke-studyhub-micro-default-pool-d1bf9b21-s3wr Ready <none> 21s v1.13.7-gke.24
nscozzaro1@cloudshell:~ (studyhub-jupyterlab)$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=nscozzaro1@gmail.com
clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created
nscozzaro1@cloudshell:~ (studyhub-jupyterlab)$ kubectl --namespace kube-system create serviceaccount tiller
serviceaccount/tiller created
nscozzaro1@cloudshell:~ (studyhub-jupyterlab)$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller created
nscozzaro1@cloudshell:~ (studyhub-jupyterlab)$ kubectl get node
NAME STATUS ROLES AGE VERSION
gke-studyhub-micro-default-pool-d1bf9b21-2x12 Ready <none> 68s v1.13.7-gke.24
gke-studyhub-micro-default-pool-d1bf9b21-pg4c Ready <none> 68s v1.13.7-gke.24
gke-studyhub-micro-default-pool-d1bf9b21-s3wr Ready <none> 68s v1.13.7-gke.24
nscozzaro1@cloudshell:~ (studyhub-jupyterlab)$ helm init --service-account tiller --wait
$HELM_HOME has been configured at /home/nscozzaro1/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Error: tiller was not found. polling deadline exceeded
nscozzaro1@cloudshell:~ (studyhub-jupyterlab)$ kubectl get node
NAME STATUS ROLES AGE VERSION
gke-studyhub-micro-default-pool-d1bf9b21-2x12 Ready <none> 7m51s v1.13.7-gke.24
gke-studyhub-micro-default-pool-d1bf9b21-pg4c NotReady <none> 7m51s v1.13.7-gke.24
gke-studyhub-micro-default-pool-d1bf9b21-s3wr NotReady <none> 7m51s v1.13.7-gke.24 |
|
I tried the same once, i concluded that it was a bit of a stretch to go for the micronode. I settled for a n1-standard-1 for basic jh stuff and then allowing things to autoscale. Making that single node preemtible could be a good way to reduce cost further i think but that would also allow the service to crash now and then. When i got the 1 core setup functional, i removed some resource requests i think. On mobile omw to work, so its hard to provide an example for me now on setting this but i think there are notes about this in the z2jh.jupyter.org guide. Note you want to reduce resource requets for the hub, proxy, and singleuser pod, not only one of these for example. Also, you may want to trim the requests of system pods like kube-dns etc by going |
That's what the resource usage and limits suggest. f1-micro is burstable which means you can use more than 0.2 CPU for short periods but it'll then be throttled again, this may explain why things may temporarily work. |
|
@consideRatio thank you for those insights... I looked into preemptible nodes, but it seems they only last 24 hours? Since it seems you have some good experience with this, I would be grateful if you would mind sharing the exact steps (commands) to set up the cheapest JupyterHub on GCP/kubernetes that can be permanently running? For reference, below is a summary of the instructions from the the z2jh docs that work for me for launching the current site that I have at studyhub.co (I've left out customizing my config.yaml). How would these change to implement your suggestions? |
|
@nscozzaro sorry for now following up at the time, I'd say its fully possible to get a 1 CPU deployment. The procedure is pretty much to shut down all not needed containers, and minimize the resource requests on all containers. While working to do so, its relevant to know that there is sometimes a k8s resource that influence the default requests pods make: the LimitRange resource. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When I follow the Zero to JupyterHub instructions for Google Cloud, they work perfectly for me. However, when I change the machine-type from
n1-standard-2to the smallest machine typef1-micro(https://cloud.google.com/compute/docs/machine-types) with num-nodes=8, the hub is not created and so the setup fails. My motivation, which I think may be shared by others, is to set up the most cost-efficient JupyterHub possible, at least for an initial setup.The section in the instructions that I'm referring to is (https://zero-to-jupyterhub.readthedocs.io/en/latest/google/step-zero-gcp.html):
If someone knows why this may fail, or confirm what I'm experiencing I'd appreciate it, thank you!
The text was updated successfully, but these errors were encountered: