Skip to content

Commit

Permalink
Merge pull request #11728 from MicrosoftDocs/repo_sync_working_branch
Browse files Browse the repository at this point in the history
Confirm merge from repo_sync_working_branch to main to sync with https://github.com/microsoftdocs/architecture-center (branch main)
  • Loading branch information
prmerger-automator[bot] committed Mar 12, 2024
2 parents fb62071 + f37c0a1 commit cf8b921
Showing 1 changed file with 43 additions and 5 deletions.
48 changes: 43 additions & 5 deletions docs/operator-guides/aks/troubleshoot-network-aks.md
Expand Up @@ -78,18 +78,56 @@ create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to
delegate: Failed to allocate address: No available addresses
```

Or a `not enough IPs available` error:

```output
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox
'ac1b1354613465324654c1588ac64f1a756aa32f14732246ac4132133ba21364': plugin type='azure-vnet'
failed (add): IPAM Invoker Add failed with error: Failed to get IP address from CNS with error:
%w: AllocateIPConfig failed: not enough IPs available for 9c6a7f37-dd43-4f7c-a01f-1ff41653609c,
waiting on Azure CNS to allocate more with NC Status: , IP config request is [IPConfigRequest:
DesiredIPAddress , PodInterfaceID a1876957-eth0, InfraContainerID
a1231464635654a123646565456cc146841c1313546a515432161a45a5316541, OrchestratorContext
{'PodName':'a_podname','PodNamespace':'my_namespace'}]
```

Check the allocated IP addresses in the plugin IPAM store. You might find that all IP addresses are allocated, but the number is much less than the number of running Pods:

**If using [kubenet](/azure/aks/configure-kubenet):**

```bash
# Kubenet, for example. The actual path of the IPAM store file depends on network plugin implementation.
cd /var/lib/cni/networks/kubenet
ls -al|wc -l
258
chroot /host/
ls -la "/var/lib/cni/networks/$(ls /var/lib/cni/networks/ | grep -e "k8s-pod-network" -e "kubenet")" | grep -v -e "lock\|last\|total" -e '\.$' | wc -l
244
```

> [!NOTE]
> For kubenet without Calico, the path is `/var/lib/cni/networks/kubenet`. For kubenet with Calico, the path is `/var/lib/cni/networks/k8s-pod-network`. The script above will auto select the path while executing the command.
docker ps | grep POD | wc -l
```bash
# Check running Pod IPs
kubectl get pods --field-selector spec.nodeName=<your_node_name>,status.phase=Running -A -o json | jq -r '.items[] | select(.spec.hostNetwork != 'true').status.podIP' | wc -l
7
```

**If using [Azure CNI for dynamic IP allocation](/azure/aks/configure-azure-cni-dynamic-ip-allocation):**

```bash
kubectl get nnc -n kube-system -o wide
```

```output
NAME REQUESTED IPS ALLOCATED IPS SUBNET SUBNET CIDR NC ID NC MODE NC TYPE NC VERSION
aks-agentpool-12345678-vmss000000 32 32 subnet 10.18.0.0/15 559e239d-f744-4f84-bbe0-c7c6fd12ec17 dynamic vnet 1
```

```bash
# Check running Pod IPs
kubectl get pods --field-selector spec.nodeName=aks-agentpool-12345678-vmss000000,status.phase=Running -A -o json | jq -r '.items[] | select(.spec.hostNetwork != 'true').status.podIP' | wc -l
21
```

**Cause 1**

This error can be caused by a bug in the network plugin. The plugin can fail to deallocate the IP address when a Pod is terminated.
Expand Down Expand Up @@ -254,4 +292,4 @@ Other contributors:
- [AKS day-2 operations guide](../../operator-guides/aks/day-2-operations-guide.md)
- [Triage practices](../../operator-guides/aks/aks-triage-practices.md)
- [Patching and upgrade guidance](../../operator-guides/aks/aks-upgrade-practices.md)
- [Monitoring AKS with Azure Monitor](/azure/aks/monitor-aks)
- [Monitoring AKS with Azure Monitor](/azure/aks/monitor-aks)

0 comments on commit cf8b921

Please sign in to comment.