Karpenter controller can't schedule/add any nodes #4698
Labels
bug
Something isn't working
Comments
|
This is my provisioner YAML. apiVersion: v1
items:
- apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
annotations:
karpenter.sh/provisioner-hash: "1599235016353422174"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"karpenter.sh/v1alpha5","kind":"Provisioner","metadata":{"labels":{"app.kubernetes.io/managed-by":"pulumi"},"name":"default"},"spec":{"providerRef":{"name":"aws"},"requirements":[{"key":"karpenter.k8s.aws/instance-categor
y","operator":"In","values":["m","r","c"]},{"key":"karpenter.k8s.aws/instance-cpu","operator":"In","values":["4","8","16","32"]}]}}
creationTimestamp: "2023-09-26T09:10:55Z"
generation: 1
labels:
app.kubernetes.io/managed-by: pulumi
name: default
resourceVersion: "130107"
uid: a7b41239-623d-4a50-a040-dfcdae716968
spec:
providerRef:
name: aws
requirements:
- key: karpenter.k8s.aws/instance-category
operator: In
values:
- m
- r
- c
- key: karpenter.k8s.aws/instance-cpu
operator: In
values:
- "4"
- "8"
- "16"
- "32"
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- amd64
- key: karpenter.sh/capacity-type
operator: In
values:
- on-demand
kind: List
metadata:
resourceVersion: ""These are my logs: The rest of the logs are pretty much the same. |
|
What does the node template look like? |
|
Seeing the same thing in one of our clusters. All 20 clusters have a pretty much identical configuration yet this one refuses to launch any nodes. Provisioner: |
|
What does the node template look like and can you show the provisioner's status (status section from |
|
@tzneal please accept my apologies! - It would appear the helm chart we use to install our custom nodetemplate had been failing silently in CI and there was no node template present! |
|
No problem, we should have a much better indicator of the problem in this case. I'll fix that. |
3 tasks
|
Hey thanks, the problem was with my providerRef. I wasn't specifying the correct one. Would be great to have a better error message here |
|
@tzneal I do have the same error, this is odd |
|
@hassammustafaa did you managed to solve the error?I have the same problem and it's driving me crazy... |
|
Yes, I did, can you share your AWSNodeTemplate and output of |
|
I've just realize that i had a typo...:D |
|
I had the same issue. I resolved it by adding the right tag to the subnet and security group. |
My SG and Subnets tag whas missing the last character in "karpenter.sh/discovery" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Observed Behavior:
Karpenter controller can't schedule any nodes in my cluster. The following are the logs I see in the controller pod:
$ kubectl logs -f -c controller -l app.kubernetes.io/name=karpenter 2023-09-26T10:00:27.381Z DEBUG controller.pricing pricing/pricing.go:382 updated spot pricing with instance types and offerings {"commit": "637a642", "instance-type-count": 702, "offering-count": 3315} 2023-09-26T10:00:27.779Z INFO controller.provisioner provisioning/provisioner.go:238 skipping, no resolved instance types found {"commit": "637a642", "provisioner": "default"} 2023-09-26T10:00:27.781Z ERROR controller.provisioner scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default" {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"} github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190 github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172 github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330 github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130 github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94 github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82 sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1 sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219Expected Behavior:
Karpenter to have added new nodes, but it didn't.
Reproduction Steps (Please include YAML):
Versions:
kubectl version):The text was updated successfully, but these errors were encountered: