Mini Kube Scheduler

What this repo does are:

• Study kubernetes scheduler following 自作して学ぶKubernetes Scheduler with sanposhiho/mini-kube-scheduler.
  • Simplify the original repo https://github.com/sanposhiho/mini-kube-scheduler
  • Implement ToDos in sanposhiho/mini-kube-scheduler .
• Gain basic understanding of Scheduling Framework by hands-on.

Versions

• Go: 1.17
• Kubernetes: 1.23.4
• etcd: 3.5.2

Components

• Scheduler: Custom Scheduler (minisched)
  • SchedulingQueue: Store Pods to schedule to a node.
    • activeQ: A queue to store Pods to start scheduling.
    • unschedulableQ: A queue to store Pods that are failed to schedule (with the plugins that made it unschedulable.)
    • podBackoffQ: A queue to store Pods that are in back-off state.
  • client: Used to update Pod to bind it to a node.
• Informer with Handler (Scheduler.addPodToSchedulingQueue()) + FilterFunc
• Service:
  • Initialize Scheduler and set the event handler for informer with New()
  • Call Scheduler.Run() to start Scheduler.
• Plugin: Implement logic for extension points in each scheduling context and called in the scheduler.

Diagram:

Files:

1. hack: Shell scripts
   1. etcd.sh: Functions to start/stop/cleanup etcd.
   2. openapi.sh: Generate zz_generated.openapi.go for kube-apiserver.
   3. run.sh: Start etcd and run the scheduler.
2. k8sapiserver: Dependency to run a scheduler.
3. minisched: Implementation of mini-kube-scheduler.
4. sched.go: Run dependency (apiserver) and the scheduler within a scenario (create nodes and a pod).
5. scheduler: Scheduler service to manage minisched.

Steps

1. Initial Random Scheduler: Randomly schedule a Pod to available nodes.
2. Filter Plugins: Use NodeUnschedulable plugin.
3. Score Plugins: Schedule based on the score of each node. (NodeName)
4. PreScore Plugins: Move common logic in the ScorePlugin to PreScorePlugin to reuse it.
5. Permit Plugins: Delay n seconds based on the name of the node.
6. Scheduling Queue: Make the simple queue closer to the one used in kube-scheduler. (podBackoffQ and UnschedulableQ + Logic to put back a queue)
7. Event Handler: Trigger MoveAllToActiveOrBackoffQueue when a new node is added.
8. Flush Queue: Flush podBackoffQ and UnschedulableQ periodically.

Tips: Development Tips

References

1. kubernetes.io/docs
   1. https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework
   2. https://kubernetes.io/docs/reference/scheduling/config
2. github.com
   1. https://github.com/sanposhiho/mini-kube-scheduler
   2. https://github.com/draios/kubernetes-scheduler
   3. https://github.com/kubernetes/kubernetes/blob/v1.23.4/pkg/scheduler/internal/queue/scheduling_queue.go
   4. https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/624-scheduling-framework
   5. https://github.com/kubernetes-sigs/kube-scheduler-simulator
   6. https://github.com/kubernetes-sigs/scheduler-plugins
3. Others
   1. 自作して学ぶKubernetes Scheduler (mercari engineering)
   2. 自作して学ぶKubernetes scheduler入門 (CloudNative Days 2021)
   3. Writing custom Kubernetes schedulers
   4. Kubernetes Scheduler 自作入門
   5. Building a Kubernetes Scheduler using Custom Metrics - Mateo Burillo, Sysdig (YouTube)
   6. 'unknown revision v0.0.0' errors, seemingly due to 'require k8s.io/foo v0.0.0' #79384