There are two main ways to configure the connection to the kubernetes cluster:
One option is to specify the path to a kubernetes config YAML file, similar to one used by kubectl. Example:
:compute-clusters [{:factory-fn cook.kubernetes.compute-cluster/factory-fn
:config {:compute-cluster-name "minikube"
;; Location of the kubernetes config file, e.g. $HOME/.kube/config
:config-file "/home/myuser/.kube/config.yml"}}]
For GKE clusters, OAuth is used to authenticate with the cluster. To connect to a GKE cluster, you need two things:
- Google JSON credential file: https://cloud.google.com/iam/docs/creating-managing-service-account-keys
- Cluster master url (this will be the api client base URL). You can get this from 'gcloud container clusters list' or 'kubectl cluster-info' Example:
:compute-clusters [{:factory-fn cook.kubernetes.compute-cluster/factory-fn
:config {:compute-cluster-name "gke"
:base-url "http://<IP ADDRESS>:8000"
;; Location of credential file
:google-credentials "/home/myuser/creds.json"}}]
Cook has two ways of configuring the namespace kubernetes pods are launched in:
- Single static namespace: Cook can be configured to launch all kubernetes pods in a single static namespace. Example:
:namespace {:kind :static
:namespace "cook"}
- Per-user namespaces: Cook can also be configured to launch each pod in a namespace corresponding to the username of the user who submitted the job. Example:
:namespace {:kind :per-user}
- Use config-k8s.edn instead of config.edn
- You need to set a path to the kube config with COOK_K8S_CONFIG_FILE
These are done for you in bin/run-kubernetes-local.sh
In order to run things, you'll need to setup several python virtual environments and pip install their requirements into them:
- One for building the CLI.
- One for running run-kubernetes-local.sh
- One for integration tests
- Set up minikube -- See https://kubernetes.io/docs/tasks/tools/install-minikube/
- Start datomic:
scheduler/bin/start-datomic.sh
- Seed pools -- Only needed if you use pools in config.edn. (Unnecessary with the config.edn used in this example.)
lein exec -p scheduler/datomic/data/seed_pools.clj datomic:free://localhost:4334/jobs
- Make namespace -- Needed the first time you use a kubernetes environment.
kubectl create -f docs/make-kubernetes-namespace.json
- Run local. It's OK to kill and restart.
COOK_DATOMIC_URI=datomic:free://localhost:4334/jobs scheduler/bin/run-kubernetes-local.sh
-
Get a kubernetes dashboard: minikube dashboard
-
Submit a test job
cs submit -i alpine:latest -c .01 -m 16 "echo 200"
COOK_DEFAULT_JOB_CPUS=0.1 COOK_TEST_DOCKER_IMAGE=python:3.5 pytest -m 'not cli'