Open
Description
Description
Observed Behavior:
We currently receive no information from Karpenter as to why a node was deleted (e.g. due to consolidation, because it was empty, because it is drifted). In the Karpenter logs there are only log entries that a node was deleted by Karpenter. Unfortunately, the reason is missing. Here are the logs that can be viewed:
{"time":"2025-06-25T07:59:24.583353879Z","stream":"stdout","_p":"F","log":"{\"level\":\"INFO\",\"time\":\"2025-06-25T07:59:24.583Z\",\"logger\":\"controller\",\"message\":\"deleted nodeclaim\",\"commit\":\"9458bb5\",\"controller\":\"nodeclaim.lifecycle\",\"controllerGroup\":\"karpenter.sh\",\"controllerKind\":\"NodeClaim\",\"NodeClaim\":{\"name\":\"flux-system-karpenter--riff--on-demand-zp78h\"},\"namespace\":\"\",\"name\":\"flux-system-karpenter--riff--on-demand-zp78h\",\"reconcileID\":\"d1a9f9d4-c1ae-4763-98ae-8057536a87b5\",\"provider-id\":\"aws:///eu-central-1c/i-03d3728ddaea9a1e4\",\"Node\":{\"name\":\"ip-10-110-106-175.eu-central-1.compute.internal\"}}","kubernetes":{"pod_name":"karpenter-57f84c84b9-9xknz","namespace_name":"system-karpenter","pod_id":"c7d81ce1-f64c-4796-8f51-e51f81b16815","labels":{"app.kubernetes.io/instance":"system-karpenter-karpenter","app.kubernetes.io/name":"karpenter","pod-template-hash":"57f84c84b9"},"annotations":{"vpc.amazonaws.com/pod-ips":"100.64.128.148"},"host":"ip-10-110-106-61.eu-central-1.compute.internal","container_name":"controller","docker_id":"307cfdd01462863b30b8b77db55bbbeb662db2460fc9c90c9cceb2d771bcc7f8","container_hash":"983981291973.dkr.ecr.eu-central-1.amazonaws.com/saso-ops/infra/prd/images/karpenter/controller@sha256:339aef3f5ecdf6f94d1c7cc9d0e1d359c281b4f9b842877bdbf2acd3fa360521","container_image":"983981291973.dkr.ecr.eu-central-1.amazonaws.com/saso-ops/infra/prd/images/karpenter/controller:1.5.0"}}
The metric karpenter_voluntary_disruption_eligible_nodes
also does not allow the reason to be clearly traced back to a specific node.
Expected Behavior:
Karpenter logs and/or submits metrics as to why a specific node was deleted.
Reproduction Steps (Please include YAML):
- Create node
- Wait until it is deleted by Karpenter due to emptiness (allow via disruption control)
- Check logs of Karpenter for reason for deletion (not available)
Versions:
- Chart version: 1.5.0
- Kubernetes version (
kubectl version
):
Client Version: v1.33.2
Kustomize Version: v5.6.0
Server Version: v1.32.5-eks-5d4a308
Translated with DeepL.com (free version)
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment