Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Abnormal Time Between disruption runs #191

Open
Bryce-Soghigian opened this issue Mar 11, 2024 · 1 comment
Open

Abnormal Time Between disruption runs #191

Bryce-Soghigian opened this issue Mar 11, 2024 · 1 comment
Labels
area/consolidation Issues or PRs related to consolidation area/deprovisioning Issues or PRs related to deprovisioning kind/bug Categorizes issue or PR as related to a bug.

Comments

@Bryce-Soghigian
Copy link
Collaborator

Bryce-Soghigian commented Mar 11, 2024 •
edited

Version

Karpenter Version: v0.3.0
__Karpenter Core Version: v0.33.1

Kubernetes Version: v1.27.9

Expected Behavior

We should ideally never see these logs as seeing a 15 minute delay in disruption breaks the scale down SLOs we promise.

Actual Behavior

When running long running deployments, every couple of days, we will see some abnormal time of runs for all of the disruption actions. Note that we do see some garbage collection of an instance shortly before we see all of the abnormal consolidation errors. Since TTL is 15m for GC, it might be possible that there is some conflict there.

We should look into

  1. What time was general-purpose-hcdgt created.
  2. What time was it garbage collected
  3. Was it marked for disruption beforehand
  4. Learn more about the abnormal time between disruption logs emitted from core to see if this really is problematic

Steps to Reproduce the Problem

Unsure how to repro, need to dig further to find the RCA for this issue.

Resource Specs and Logs

Separating logs as they are too verbose and long github says maximum characters reached

{"level":"DEBUG","time":"2024-03-09T03:44:56.773Z","logger":"controller.disruption","message":"abnormal time between runs of *disruption.Emptiness = 15m0.285297573s","commit":"c32dbc1-dirty"}
{"level":"DEBUG","time":"2024-03-09T03:44:56.773Z","logger":"controller.disruption","message":"abnormal time between runs of *disruption.EmptyNodeConsolidation = 15m0.171398206s","commit":"c32dbc1-dirty"}
{"level":"DEBUG","time":"2024-03-09T03:44:56.773Z","logger":"controller.disruption","message":"abnormal time between runs of *disruption.MultiNodeConsolidation = 15m0.141458065s","commit":"c32dbc1-dirty"}
{"level":"DEBUG","time":"2024-03-09T03:44:56.773Z","logger":"controller.disruption","message":"abnormal time between runs of *disruption.SingleNodeConsolidation = 15m0.060651402s","commit":"c32dbc1-dirty"}
{"level":"DEBUG","time":"2024-03-09T03:44:56.773Z","logger":"controller.disruption","message":"abnormal time between runs of *disruption.Expiration = 15m0.465905477s","commit":"c32dbc1-dirty"}
...... There are 100 additional logs here but removing for brevity
{"level":"DEBUG","time":"2024-03-09T03:45:28.457Z","logger":"controller.disruption","message":"abnormal time between runs of *disruption.SingleNodeConsolidation = 15m31.744886108s","commit":"c32dbc1-dirty"}
{"level":"DEBUG","time":"2024-03-09T03:45:29.457Z","logger":"controller.disruption","message":"abnormal time between runs of *disruption.MultiNodeConsolidation = 15m32.826051368s","commit":"c32dbc1-dirty"}
{"level":"DEBUG","time":"2024-03-09T03:45:29.457Z","logger":"controller.disruption","message":"abnormal time between runs of *disruption.SingleNodeConsolidation = 15m32.745308809s","commit":"c32dbc1-dirty"}

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment
@Bryce-Soghigian
Copy link
Collaborator Author

Controller logs for that nodeclaim that was problematic and possibly related. I could be way off base here as well

{"level":"INFO","time":"2024-03-09T03:30:06.616Z","logger":"controller.provisioner","message":"created nodeclaim","commit":"c32dbc1-dirty","nodepool":"general-purpose","nodeclaim":"general-purpose-hcdgt","requests":{"cpu":"53490m","memory":"14530000Ki","pods":"60"},"instance-types":"Standard_D64_v3, Standard_D64_v4, Standard_D64_v5, Standard_D64a_v4, Standard_D64ads_v5 and 21 other(s)"}
{"level":"INFO","time":"2024-03-09T03:30:06.696Z","logger":"controller.nodeclaim.lifecycle","message":"Selected instance type Standard_D64ls_v5","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:30:06.696Z","logger":"controller.nodeclaim.lifecycle","message":"Resolved image /CommunityGalleries/AKSUbuntu-38d80f77-467a-481f-a8d4-09b6d4220bd2/images/2204gen2containerd/versions/202403.04.0 for instance type Standard_D64ls_v5","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:30:06.697Z","logger":"controller.nodeclaim.lifecycle","message":"Querying load balancers in resource group MC_redacted_RG","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:30:07.289Z","logger":"controller.nodeclaim.lifecycle","message":"Found 1 load balancers of interest","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:07.289Z","logger":"controller.nodeclaim.lifecycle","message":"Returning 2 IPv4 backend pools: [/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes]","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:07.289Z","logger":"controller.nodeclaim.lifecycle","message":"Creating network interface aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:10.598Z","logger":"controller.nodeclaim.lifecycle","message":"Successfully created network interface: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:10.598Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine aks-general-purpose-hcdgt (Standard_D64ls_v5)","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:30:14.102Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine \"aks-general-purpose-hcdgt\" failed: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"OperationNotAllowed\",\n    \"message\": \"Operation could not be completed as it results in exceeding approved standardDLSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 100, Current Usage: 56, Additional Required: 64, (Minimum) New Limit Required: 120. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22redacted%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDLSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:120,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDLSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:30:14.102Z","logger":"controller.nodeclaim.lifecycle","message":"virtualMachine.BeginCreateOrUpdate for VM \"aks-general-purpose-hcdgt\" failed: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"OperationNotAllowed\",\n    \"message\": \"Operation could not be completed as it results in exceeding approved standardDLSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 100, Current Usage: 56, Additional Required: 64, (Minimum) New Limit Required: 120. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22redacted%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDLSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:120,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDLSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:14.103Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64ls_v5","zone":"eastus-3","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"DEBUG","time":"2024-03-09T03:30:14.103Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64ls_v5","zone":"eastus-2","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"DEBUG","time":"2024-03-09T03:30:14.103Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64ls_v5","zone":"eastus-1","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"ERROR","time":"2024-03-09T03:30:15.233Z","logger":"controller.nodeclaim.lifecycle","message":"networkInterface.Delete for aks-general-purpose-hcdgt failed: DELETE https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: NicReservedForAnotherVm\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"NicReservedForAnotherVm\",\n    \"message\": \"Nic(s) in request is reserved for another Virtual Machine for 180 seconds. Please provide another nic(s) or retry after 180 seconds. Reserved VM: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\",\n    \"details\": []\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:30:15.233Z","logger":"controller.nodeclaim.lifecycle","message":"failed to cleanup resources for node claim general-purpose-hcdgt, %!w(*errors.joinError=&{[0xc0047b5460]})","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:30:16.256Z","logger":"controller","message":"Reconciler error","commit":"c32dbc1-dirty","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"general-purpose-hcdgt"},"namespace":"","name":"general-purpose-hcdgt","reconcileID":"a3ae6ccf-e956-4e6e-8481-da7bf8b5b340","error":"launching nodeclaim, creating instance, subscription level on-demand vCPU quota for Standard_D64ls_v5 has been reached (may try provision an alternative instance type)"}
{"level":"INFO","time":"2024-03-09T03:30:17.284Z","logger":"controller.nodeclaim.lifecycle","message":"Selected instance type Standard_D64as_v5","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:30:17.284Z","logger":"controller.nodeclaim.lifecycle","message":"Resolved image /CommunityGalleries/AKSUbuntu-38d80f77-467a-481f-a8d4-09b6d4220bd2/images/2204gen2containerd/versions/202403.04.0 for instance type Standard_D64as_v5","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:17.284Z","logger":"controller.nodeclaim.lifecycle","message":"Returning 2 IPv4 backend pools: [/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes]","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:17.284Z","logger":"controller.nodeclaim.lifecycle","message":"Creating network interface aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:22.270Z","logger":"controller.nodeclaim.lifecycle","message":"Successfully created network interface: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:22.270Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine aks-general-purpose-hcdgt (Standard_D64as_v5)","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:30:25.286Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine \"aks-general-purpose-hcdgt\" failed: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"OperationNotAllowed\",\n    \"message\": \"Operation could not be completed as it results in exceeding approved standardDASv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 100, Current Usage: 96, Additional Required: 64, (Minimum) New Limit Required: 160. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22redacted%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDASv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:160,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDASv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:30:25.286Z","logger":"controller.nodeclaim.lifecycle","message":"virtualMachine.BeginCreateOrUpdate for VM \"aks-general-purpose-hcdgt\" failed: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"OperationNotAllowed\",\n    \"message\": \"Operation could not be completed as it results in exceeding approved standardDASv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 100, Current Usage: 96, Additional Required: 64, (Minimum) New Limit Required: 160. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22redacted%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDASv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:160,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDASv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:25.287Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64as_v5","zone":"eastus-3","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"DEBUG","time":"2024-03-09T03:30:25.287Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64as_v5","zone":"eastus-2","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"DEBUG","time":"2024-03-09T03:30:25.287Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64as_v5","zone":"eastus-1","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"ERROR","time":"2024-03-09T03:30:26.439Z","logger":"controller.nodeclaim.lifecycle","message":"networkInterface.Delete for aks-general-purpose-hcdgt failed: DELETE https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: NicReservedForAnotherVm\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"NicReservedForAnotherVm\",\n    \"message\": \"Nic(s) in request is reserved for another Virtual Machine for 180 seconds. Please provide another nic(s) or retry after 180 seconds. Reserved VM: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\",\n    \"details\": []\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:30:26.439Z","logger":"controller.nodeclaim.lifecycle","message":"failed to cleanup resources for node claim general-purpose-hcdgt, %!w(*errors.joinError=&{[0xc00670f600]})","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:30:27.462Z","logger":"controller","message":"Reconciler error","commit":"c32dbc1-dirty","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"general-purpose-hcdgt"},"namespace":"","name":"general-purpose-hcdgt","reconcileID":"2f4b65bd-de41-4fda-8fe5-a26f519e6c4d","error":"launching nodeclaim, creating instance, subscription level on-demand vCPU quota for Standard_D64as_v5 has been reached (may try provision an alternative instance type)"}
{"level":"INFO","time":"2024-03-09T03:30:29.511Z","logger":"controller.nodeclaim.lifecycle","message":"Selected instance type Standard_D64_v3","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:30:29.517Z","logger":"controller.nodeclaim.lifecycle","message":"Resolved image /CommunityGalleries/AKSUbuntu-38d80f77-467a-481f-a8d4-09b6d4220bd2/images/2204containerd/versions/202403.04.0 for instance type Standard_D64_v3","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:29.517Z","logger":"controller.nodeclaim.lifecycle","message":"Returning 2 IPv4 backend pools: [/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes]","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:29.517Z","logger":"controller.nodeclaim.lifecycle","message":"Creating network interface aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:33.199Z","logger":"controller.nodeclaim.lifecycle","message":"Successfully created network interface: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:30:33.199Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine aks-general-purpose-hcdgt (Standard_D64_v3)","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:36:14.569Z","logger":"controller.nodeclaim.garbagecollection","message":"Deleting virtual machine %s and associated resources","commit":"c32dbc1-dirty","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","nodeclaim":""}
{"level":"DEBUG","time":"2024-03-09T03:43:29.836Z","logger":"controller.nodeclaim.garbagecollection","message":"garbage collected cloudprovider instance","commit":"c32dbc1-dirty","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:30.218Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine \"aks-general-purpose-hcdgt\" failed: GET https://management.azure.com/subscriptions/redacted/providers/Microsoft.Compute/locations/eastus/operations/41036c9a-f306-43fa-b939-ca26e6905913\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: OperationPreempted\n--------------------------------------------------------------------------------\n{\n  \"startTime\": \"2024-03-09T03:30:36.0771057+00:00\",\n  \"endTime\": \"2024-03-09T03:43:10.7448205+00:00\",\n  \"status\": \"Canceled\",\n  \"error\": {\n    \"code\": \"OperationPreempted\",\n    \"message\": \"Operation execution has been preempted by a more recent operation.\"\n  },\n  \"name\": \"41036c9a-f306-43fa-b939-ca26e6905913\"\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:31.676Z","logger":"controller","message":"Reconciler error","commit":"c32dbc1-dirty","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"general-purpose-hcdgt"},"namespace":"","name":"general-purpose-hcdgt","reconcileID":"79858cdf-b36e-4614-86c3-f33449d7718a","error":"launching nodeclaim, creating instance, virtualMachine.BeginCreateOrUpdate for VM \"aks-general-purpose-hcdgt\" failed: GET https://management.azure.com/subscriptions/redacted/providers/Microsoft.Compute/locations/eastus/operations/41036c9a-f306-43fa-b939-ca26e6905913\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: OperationPreempted\n--------------------------------------------------------------------------------\n{\n  \"startTime\": \"2024-03-09T03:30:36.0771057+00:00\",\n  \"endTime\": \"2024-03-09T03:43:10.7448205+00:00\",\n  \"status\": \"Canceled\",\n  \"error\": {\n    \"code\": \"OperationPreempted\",\n    \"message\": \"Operation execution has been preempted by a more recent operation.\"\n  },\n  \"name\": \"41036c9a-f306-43fa-b939-ca26e6905913\"\n}\n--------------------------------------------------------------------------------\n"}
{"level":"INFO","time":"2024-03-09T03:43:31.703Z","logger":"controller.nodeclaim.lifecycle","message":"Selected instance type Standard_D64ls_v5","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:43:31.703Z","logger":"controller.nodeclaim.lifecycle","message":"Resolved image /CommunityGalleries/AKSUbuntu-38d80f77-467a-481f-a8d4-09b6d4220bd2/images/2204gen2containerd/versions/202403.04.0 for instance type Standard_D64ls_v5","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:31.704Z","logger":"controller.nodeclaim.lifecycle","message":"Returning 2 IPv4 backend pools: [/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes]","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:31.704Z","logger":"controller.nodeclaim.lifecycle","message":"Creating network interface aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:34.945Z","logger":"controller.nodeclaim.lifecycle","message":"Successfully created network interface: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:34.945Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine aks-general-purpose-hcdgt (Standard_D64ls_v5)","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:37.797Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine \"aks-general-purpose-hcdgt\" failed: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"OperationNotAllowed\",\n    \"message\": \"Operation could not be completed as it results in exceeding approved standardDLSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 100, Current Usage: 56, Additional Required: 64, (Minimum) New Limit Required: 120. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22redacted%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDLSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:120,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDLSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:37.797Z","logger":"controller.nodeclaim.lifecycle","message":"virtualMachine.BeginCreateOrUpdate for VM \"aks-general-purpose-hcdgt\" failed: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"OperationNotAllowed\",\n    \"message\": \"Operation could not be completed as it results in exceeding approved standardDLSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 100, Current Usage: 56, Additional Required: 64, (Minimum) New Limit Required: 120. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22redacted%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDLSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:120,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDLSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:37.797Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64ls_v5","zone":"eastus-1","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"DEBUG","time":"2024-03-09T03:43:37.797Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64ls_v5","zone":"eastus-3","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"DEBUG","time":"2024-03-09T03:43:37.797Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64ls_v5","zone":"eastus-2","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"ERROR","time":"2024-03-09T03:43:39.388Z","logger":"controller.nodeclaim.lifecycle","message":"networkInterface.Delete for aks-general-purpose-hcdgt failed: DELETE https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: NicReservedForAnotherVm\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"NicReservedForAnotherVm\",\n    \"message\": \"Nic(s) in request is reserved for another Virtual Machine for 180 seconds. Please provide another nic(s) or retry after 180 seconds. Reserved VM: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\",\n    \"details\": []\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:39.388Z","logger":"controller.nodeclaim.lifecycle","message":"failed to cleanup resources for node claim general-purpose-hcdgt, %!w(*errors.joinError=&{[0xc00314d120]})","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:40.407Z","logger":"controller","message":"Reconciler error","commit":"c32dbc1-dirty","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"general-purpose-hcdgt"},"namespace":"","name":"general-purpose-hcdgt","reconcileID":"ab3be226-74d1-4da7-a586-ba1aef3d3766","error":"launching nodeclaim, creating instance, subscription level on-demand vCPU quota for Standard_D64ls_v5 has been reached (may try provision an alternative instance type)"}
{"level":"INFO","time":"2024-03-09T03:43:40.525Z","logger":"controller.nodeclaim.lifecycle","message":"Selected instance type Standard_D64as_v5","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:43:40.526Z","logger":"controller.nodeclaim.lifecycle","message":"Resolved image /CommunityGalleries/AKSUbuntu-38d80f77-467a-481f-a8d4-09b6d4220bd2/images/2204gen2containerd/versions/202403.04.0 for instance type Standard_D64as_v5","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:40.528Z","logger":"controller.nodeclaim.lifecycle","message":"Returning 2 IPv4 backend pools: [/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes]","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:40.528Z","logger":"controller.nodeclaim.lifecycle","message":"Creating network interface aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:44.130Z","logger":"controller.nodeclaim.lifecycle","message":"Successfully created network interface: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:44.130Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine aks-general-purpose-hcdgt (Standard_D64as_v5)","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:47.142Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine \"aks-general-purpose-hcdgt\" failed: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"OperationNotAllowed\",\n    \"message\": \"Operation could not be completed as it results in exceeding approved standardDASv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 100, Current Usage: 96, Additional Required: 64, (Minimum) New Limit Required: 160. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22redacted%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDASv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:160,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDASv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:47.143Z","logger":"controller.nodeclaim.lifecycle","message":"virtualMachine.BeginCreateOrUpdate for VM \"aks-general-purpose-hcdgt\" failed: PUT https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: OperationNotAllowed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"OperationNotAllowed\",\n    \"message\": \"Operation could not be completed as it results in exceeding approved standardDASv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 100, Current Usage: 96, Additional Required: 64, (Minimum) New Limit Required: 160. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22redacted%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDASv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:160,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDASv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:47.143Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64as_v5","zone":"eastus-2","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"DEBUG","time":"2024-03-09T03:43:47.143Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64as_v5","zone":"eastus-1","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"DEBUG","time":"2024-03-09T03:43:47.143Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","unavailable":"SubscriptionQuotaReached","instance-type":"Standard_D64as_v5","zone":"eastus-3","capacity-type":"on-demand","ttl":"3m0s"}
{"level":"ERROR","time":"2024-03-09T03:43:48.264Z","logger":"controller.nodeclaim.lifecycle","message":"networkInterface.Delete for aks-general-purpose-hcdgt failed: DELETE https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: NicReservedForAnotherVm\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"NicReservedForAnotherVm\",\n    \"message\": \"Nic(s) in request is reserved for another Virtual Machine for 180 seconds. Please provide another nic(s) or retry after 180 seconds. Reserved VM: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\",\n    \"details\": []\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:48.264Z","logger":"controller.nodeclaim.lifecycle","message":"failed to cleanup resources for node claim general-purpose-hcdgt, %!w(*errors.joinError=&{[0xc006724da0]})","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:43:49.284Z","logger":"controller","message":"Reconciler error","commit":"c32dbc1-dirty","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"general-purpose-hcdgt"},"namespace":"","name":"general-purpose-hcdgt","reconcileID":"bdd1bf2a-8eb3-46d7-b7f0-3e3801cd3a2f","error":"launching nodeclaim, creating instance, subscription level on-demand vCPU quota for Standard_D64as_v5 has been reached (may try provision an alternative instance type)"}
{"level":"INFO","time":"2024-03-09T03:43:49.312Z","logger":"controller.nodeclaim.lifecycle","message":"Selected instance type Standard_D64_v3","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:43:49.312Z","logger":"controller.nodeclaim.lifecycle","message":"Resolved image /CommunityGalleries/AKSUbuntu-38d80f77-467a-481f-a8d4-09b6d4220bd2/images/2204containerd/versions/202403.04.0 for instance type Standard_D64_v3","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:49.313Z","logger":"controller.nodeclaim.lifecycle","message":"Returning 2 IPv4 backend pools: [/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes]","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:49.313Z","logger":"controller.nodeclaim.lifecycle","message":"Creating network interface aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:50.370Z","logger":"controller.nodeclaim.lifecycle","message":"Successfully created network interface: /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Network/networkInterfaces/aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:43:50.370Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine aks-general-purpose-hcdgt (Standard_D64_v3)","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:44:40.447Z","logger":"controller.nodeclaim.lifecycle","message":"Created virtual machine /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:44:40.447Z","logger":"controller.nodeclaim.lifecycle","message":"Creating virtual machine AKS identifying extension for aks-general-purpose-hcdgt","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:45:12.133Z","logger":"controller.nodeclaim.lifecycle","message":"Created  virtual machine AKS identifying extension for aks-general-purpose-hcdgt, with an id of /subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt/extensions/computeAksLinuxBilling","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:45:12.133Z","logger":"controller.nodeclaim.lifecycle","message":"launched new instance","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","launched-instance":"/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","hostname":"aks-general-purpose-hcdgt","type":"Standard_D64_v3","zone":"2","capacity-type":"on-demand"}
{"level":"INFO","time":"2024-03-09T03:45:12.133Z","logger":"controller.nodeclaim.lifecycle","message":"launched nodeclaim","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","instance-type":"Standard_D64_v3","zone":"","capacity-type":"on-demand","allocatable":{"cpu":"63260m","ephemeral-storage":"128G","memory":"229568Mi","pods":"110"}}
{"level":"DEBUG","time":"2024-03-09T03:45:12.149Z","logger":"controller.nodeclaim.lifecycle","message":"registered nodeclaim","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","node":"aks-general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:45:12.164Z","logger":"controller.nodeclaim.lifecycle","message":"initialized nodeclaim","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","node":"aks-general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:45:28.331Z","logger":"controller.disruption","message":"disrupting via consolidation delete, terminating 4 candidates aks-general-purpose-mcwzr/Standard_D16ls_v5/on-demand, aks-general-purpose-kstg2/Standard_D96as_v5/on-demand, aks-general-purpose-dwg68/Standard_D32ls_v5/on-demand, aks-general-purpose-hcdgt/Standard_D64_v3/on-demand","commit":"c32dbc1-dirty"}
{"level":"INFO","time":"2024-03-09T03:45:28.733Z","logger":"controller.node.termination","message":"tainted node","commit":"c32dbc1-dirty","node":"aks-general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:45:28.734Z","logger":"controller.node.termination","message":"Deleting virtual machine %s and associated resources","commit":"c32dbc1-dirty","node":"aks-general-purpose-hcdgt","nodeclaim":"aks-general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:47:41.620Z","logger":"controller.node.termination","message":"deleted node","commit":"c32dbc1-dirty","node":"aks-general-purpose-hcdgt"}
{"level":"DEBUG","time":"2024-03-09T03:47:41.622Z","logger":"controller.nodeclaim.termination","message":"Deleting virtual machine %s and associated resources","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","node":"aks-general-purpose-hcdgt","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:47:42.088Z","logger":"controller.nodeclaim.termination","message":"virtualMachine.Delete for aks-general-purpose-hcdgt failed: GET https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: NotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"NotFound\",\n    \"message\": \"The entity was not found in this Azure location.\"\n  }\n}\n--------------------------------------------------------------------------------\n","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","node":"aks-general-purpose-hcdgt","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","nodeclaim":"general-purpose-hcdgt"}
{"level":"ERROR","time":"2024-03-09T03:47:42.307Z","logger":"controller","message":"Reconciler error","commit":"c32dbc1-dirty","controller":"nodeclaim.termination","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"general-purpose-hcdgt"},"namespace":"","name":"general-purpose-hcdgt","reconcileID":"01bc0172-1267-4ad0-9113-b4742920e64b","error":"terminating cloudprovider instance, GET https://management.azure.com/subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: NotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"NotFound\",\n    \"message\": \"The entity was not found in this Azure location.\"\n  }\n}\n--------------------------------------------------------------------------------\n"}
{"level":"DEBUG","time":"2024-03-09T03:47:43.308Z","logger":"controller.nodeclaim.termination","message":"Deleting virtual machine %s and associated resources","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","node":"aks-general-purpose-hcdgt","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt","nodeclaim":"general-purpose-hcdgt"}
{"level":"INFO","time":"2024-03-09T03:47:43.740Z","logger":"controller.nodeclaim.termination","message":"deleted nodeclaim","commit":"c32dbc1-dirty","nodeclaim":"general-purpose-hcdgt","node":"aks-general-purpose-hcdgt","provider-id":"azure:///subscriptions/redacted/resourceGroups/MC_redacted_RG/providers/Microsoft.Compute/virtualMachines/aks-general-purpose-hcdgt"}

@Bryce-Soghigian Bryce-Soghigian added area/consolidation Issues or PRs related to consolidation area/deprovisioning Issues or PRs related to deprovisioning kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/bug Categorizes issue or PR as related to a bug. and removed kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Mar 11, 2024
@Bryce-Soghigian Bryce-Soghigian mentioned this issue Apr 3, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/consolidation Issues or PRs related to consolidation area/deprovisioning Issues or PRs related to deprovisioning kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Bryce-Soghigian