NAP is creating NodeClaims and VMs but not registering Nodes to the cluster

[WillEdMatsypie]

Version

Karpenter Overlay Version: N/A - we are using managed NAP

Kubernetes Version: v1.28.5

Expected Behavior

Karpenter should create NodeClaims and provision VMs which then register to the cluster and are capable of scheduling Pods.

Actual Behavior

Pods go into pending state and create a NodeClaim with Karpenter i.e.

The NodeClaims then create and provision a VM but never reach Ready state

The reason given in the Claim is NodeNotFound:

We can see the VMs are up and running in the account:

Steps to Reproduce the Problem

N/A - this is a managed service. In our cluster this was working until yesterday afternoon and then Karpenter stopped being able to register Nodes with the cluster altogether.

Resource Specs and Logs

N/A - issue is with managed NAP not registering nodes to the cluster, no logs we can see outside of those shared in the "Actual behaviour" above.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[Bryce-Soghigian]

I suspect its related to #247

[Bryce-Soghigian]

Will take a look today

[Bryce-Soghigian]

I took a look at your cluster.

2024-04-01T00:37:27.778Z we see that a NodeImageUpgrade occurred on the agentpool, which triggered #247 where we see the bootstrap token isn't rotated when one of the agentpools is upgraded. We can fix this by doing az aks update -g rg -n clusterName, which simply reapplys the token and passes that configuration to NAP.

I have mitigated the problem for you by running the az aks update -g rg -n clusterName with a powerful azure employee command.

The fix for this on the nap side is currently rolling out, but this should fix things for now

[WillEdMatsypie]

Awesome, let us know when it is fully rolled out but cheers for the support!

[Bryce-Soghigian]

@WillEdMatsypie thank you for opening an issue and making us aware of the problem! Please feel free to keep opening more anytime you experience some problems!