This is an example demonstrating how to create a custom Kubernetes scheduler. The custom scheduler modifies the Kubernetes default scheduler so that a GPU related job can be provisioned in Nimbix cloud if the local cluster is unable to satisfy the GPU resource requests.
nimbix-app: Base image to be used for building application docker images scheduler: Custom K8s scheduler files for Kubernetes version - 1.8.3 deploy: Sample deployment files
git clone https://github.com/IBM/k8s-custom-scheduler.git
cd k8s-custom-scheduler/scheduler
docker build -t nimbix-sched .While these instructions are specific for ICP, it should apply for any Kubernetes setup with minor modifications
1.Create a secret for the certificate files to access apiserver over https
In ICP, certificates file reside on /etc/cfc/conf in master node. Use the following command to create the secret
kubectl create secret generic certs --from-file=kube-scheduler-config.yaml --from-file=kube-scheduler.crt --from-file=kube-scheduler.key2.Build the images for custom scheduler and sample nimbix job
Use the Dockerfiles in scheduler and nimbix-app directory
3.Deploy the scheduler
Example deployment yaml is available at deploy/k8s-custom-sched.yaml. Update the yaml file with the MASTER_IP
kubectl create -f k8s-custom-sched.yaml4.Create appropriate role binding so that custom scheduler from system:kube-scheduler can modify pods from default namespace
kubectl create rolebinding someRole --clusterrole=admin --user=system:kube-scheduler --namespace=default5.Deploy a sample GPU job using the custom scheduler
Example yaml is available at deploy/sample-job.yaml. Update the yaml with your Nimbix USERNAME and APIKEY. The job will be provisioned to Nimbix cloud if resource requirement is not met in the local cluster
kubectl create -f sample-job.yamlAbhishek Dasgupta (abdasgupta@in.ibm.com)
Pradipta Kumar Banerjee (bpradipt@in.ibm.com)