Simple time-based lock service with HTTP interface.

Designed at Oath to solve the Thundering herd problem during multiple applications startup in the Kubernetes clusters.

The Problem

Starting multiple applications simultaneously on the same host may cause a performance bottleneck. In Kubernetes this usually happens when applications are automatically deployed to a newly added Node. In the worst-case scenario, application startup may be slowed down so dramatically that they fail to pass the healthcheck. They are then restarted by Kubernetes just to start fighting for shared resources again, in an endless loop.

The Solution

Kubernetes allows a Pod to have additional, Init container, and postpone application startup until Init container finishes execution. The solution is to deploy Lock service as a DaemonSet on a Pod, and each init container will sequentially acquire this lock. So moments of application container starts will be distributed in time.

Components

See Readmes in subfolders for details.

• Lock

  HTTP service to be deployed one instance per Node (as a DaemonSet). Returns code 200 OK as a response to the first request. Returns 423 Locked to the subsequent requests until timeout exceeded. May depend on additional endpoint check.

• Init

  Lightweight client for the Lock service. To be deployed as Init Container alongside the main application container. Periodically tries to acquire the lock. Once succeeded, terminates, allowing the main container to start running.

• K8s-health

  Optional component. Performs healthcheck of Kubernetes DaemonSets and Node CPU load. May be used by Lock service to postpone lock acquiring until all DaemonSets on the Node are up and running or Node is not overloaded.

Images

The OCI images are available at

• Quay
  • K8s health
  • Lock
  • Init
• GHCR
  • K8s health
  • Lock
  • Init
• AWS ECR Public
  • K8s health
  • Lock
  • Init
• FlakyBit's Harbor

How to build locally

1. Set target platform for Go binaries.

   Optional step, default is linux. You can use Make task as shown below.

   export GOOS=darwin

2. Change directory.

   Open directory with a service you want to build.

   # cd <service directory>
   cd init
   # cd lock
   # cd k8s-health

3. Build a binary.

   # go build -v -a -o <output binary file path>
   go build -v -a -o ../bin/init
   # go build -v -a -o ../bin/lock
   # go build -v -a -o ../bin/health

4. Obtain the binaries.

   The binaries will be located in bin directory of project's root:

• pod-startup-lock/bin/init
• pod-startup-lock/bin/health
• pod-startup-lock/bin/lock

Release Notes

• 2.0.0
  • Added node load health check
  • Reworked configuration
  • Reworked logging
  • and other refactorings
• 1.1.0
  • Migrated to go modules
  • Updated dependencies
  • Fixed K8s API
  • Releasing on multiple registries
  • Building debug images
  • Updated documentation with examples
  • Updated copyright and module name
  • Migrated to Woodpecker CI
• 1.0.1
  • Added connection timeouts for http and tcp connections
  • Added keep-alive for http connections
• 1.0.0
  • Initial version

Contributing

Please feel free to submit issues, fork the repository and send pull requests!

Source

Source code is available at Gitea and mirrored to GitHub.

Links

1. Containers startup throttling Kubernetes issue
2. Thundering Herd Scheduler