title | linkTitle | weight | description |
---|---|---|---|
Development Guide |
Development Guide |
80 |
Set up a Karpenter development environment
|
The following tools are required for contributing to the Karpenter project.
Package | Version | Install |
---|---|---|
go | v1.19+ | Instructions |
kubectl | brew install kubectl |
|
helm | brew install helm |
|
Other tools | make toolchain |
Based on how you are running your Kubernetes cluster, follow the Environment specific setup to configure your environment before you continue. You can choose to either run the Karpenter controller locally on your machine, pointing to the Kubernetes cluster specified in your ~/.kube/config
or inside the Kubernetes cluster specified in your ~/.kube/config
deployed with Helm.
Once you have your environment set up, run the following commands to run the Karpenter Go binary against the Kubernetes cluster specified in your ~/.kube/config
make run
Once you have your environment set up, to install Karpenter in the Kubernetes cluster specified in your ~/.kube/config
run the following commands.
make apply # Install Karpenter
make delete # Uninstall Karpenter
- Make sure dependencies are installed
- Run
make codegen
to make sure yaml manifests are generated - Run
make toolchain
to install cli tools for building and testing the project
- Run
- You will need a personal development image repository (e.g. ECR)
- Make sure you have valid credentials to your development repository.
$KO_DOCKER_REPO
must point to your development repository- Your cluster must have permissions to read from the repository
Note: these commands do not rely on each other and may be executed independently
make apply # quickly deploy changes to your cluster
make presubmit # run codegen, lint, and tests
If you are only interested in building the Karpenter images and not deploying the updated release to your cluster immediately with Helm, you can run
make build # build and push the karpenter images
make test # E2E correctness tests
make battletest # More rigorous tests run in CI environment
kubectl patch configmap config-logging -n karpenter --patch '{"data":{"loglevel.controller":"debug"}}' # Debug Level
kubectl patch configmap config-logging -n karpenter --patch '{"data":{"loglevel.controller":"info"}}' # Info Level
OSX:
open http://localhost:8080/metrics && kubectl port-forward service/karpenter -n karpenter 8080
Linux:
gio open http://localhost:8080/metrics && kubectl port-forward service/karpenter -n karpenter 8080
While you can tail Karpenter's logs with kubectl, there's a number of tools out there that enhance the experience. We recommend Stern:
stern -n karpenter -l app.kubernetes.io/name=karpenter
For local development on Karpenter you will need a Docker repo which can manage your images for Karpenter components. You can use the following command to provision an ECR repository.
aws ecr create-repository \
--repository-name karpenter/controller \
--image-scanning-configuration scanOnPush=true \
--region "${AWS_DEFAULT_REGION}"
aws ecr create-repository \
--repository-name karpenter/webhook \
--image-scanning-configuration scanOnPush=true \
--region "${AWS_DEFAULT_REGION}"
Once you have your ECR repository provisioned, configure your Docker daemon to authenticate with your newly created repository.
export KO_DOCKER_REPO="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/karpenter"
aws ecr get-login-password --region "${AWS_DEFAULT_REGION}" | docker login --username AWS --password-stdin "${KO_DOCKER_REPO}"
Finally, to deploy the correct IAM permissions, including the instance profile for provisioned nodes, run
make setup
Karpenter exposes a pprof endpoint on its metrics port.
Learn about profiling with pprof: https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/
brew install graphviz
go install github.com/google/pprof@latest
# Connect to the metrics endpoint
kubectl port-forward service/karpenter -n karpenter 8080
open http://localhost:8080/debug/pprof/
# Visualize the memory
go tool pprof -http 0.0.0.0:9000 localhost:8080/debug/pprof/heap