Handle InvalidParameterValue as well for PD fallback

[hemanthmalla]

#30536 prematurely concluded that AWS now uses InsufficientCidrBlocks to indicate the subnet is out of prefixes. Looks like AWS still uses InvalidParameterValue and There aren't sufficient free Ipv4 addresses or prefixes to indicate subnet is at capacity. However, when subnet is at capacity potentially due to fragmentation, InsufficientCidrBlocks is returned. In either case, it's worth trying to fallback since /32 IPs might still be available compared to /28. Details from the support ticket we opened with AWS for clarification

I would like to inform you that I have received an update from our team :

• Client.InvalidParameterValue : There aren't sufficient free Ipv4 addresses or prefixes

  The exception 'Client.InvalidParameterValue : There aren't sufficient free Ipv4 addresses or prefixes' occurs when the number of requested IP addresses exceeds the number of available IP addresses in a subnet. This exception can occur during CreateNetworkInterface, AttachNetworkInterface, and AssignPrivateIpAddresses (please refer to Common client error codes section).

    [+] [https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateNetworkInterface.html ](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateNetworkInterface.html)
    [+] [https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AttachNetworkInterface.html ](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AttachNetworkInterface.html)
    [+] [https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AssignPrivateIpAddresses.html ](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AssignPrivateIpAddresses.html)
    [+] [https://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html#CommonErrors ](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html#CommonErrors)
  

• Client.InsufficientCidrBlocks : The specified subnet does not have enough free cidr blocks to satisfy the request

  The exception 'Client.InsufficientCidrBlocks : The specified subnet does not have enough free cidr blocks to satisfy the request' occurs when subnet does not have any contiguous /28 blocks available as available in our public docs.

    [+] [https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html ](https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html)
  

  Even if your subnet has available IP addresses, if the subnet does not have any contiguous /28 blocks available, you will see the following error.

  This can happen due to fragmentation of existing secondary IP addresses spread out across a subnet. To resolve this error, either create a new subnet and launch Pods there, or use an Amazon EC2 subnet CIDR reservation to reserve space within a subnet for use with prefix assignment. For more information, see Subnet CIDR reservations in the Amazon VPC User Guide.

---

Depending on subnet IP utilization during API call, the response for API call can be reaching either of the two scenarios above.

Additional References :-

  [+] https://repost.aws/knowledge-center/vpc-insufficient-ip-errors

[hemanthmalla]

/test

[christarazi]

Thanks for the fix, I note that I mentioned in the previous PR a sort lack of testing in this area. Would it be worth to write some unit tests (even some small mocks) just to validate the behavior here?

[hemanthmalla]

@christarazi updated the prefix delegation unit test to cover the fallback case. This wouldn't really catch scenarios we're hitting here (new error codes), but the fallback feature itself should now be tested based on mocked API implementation.

➜   go test ./pkg/aws/eni -test.v -test.run Test/ENISuite/TestNodeManagerPrefixDelegation
=== RUN   Test
=== RUN   Test/ENISuite
=== RUN   Test/ENISuite/TestNodeManagerPrefixDelegation
level=info msg="Synchronized ENI information" numInstances=1 numSecurityGroups=2 numSubnets=1 numVPCs=1 subsys=eni
level=info msg="Discovered new CiliumNode custom resource" name=node1 subsys=ipam
level=warning msg="Unable to compute pending pods, will not surge-allocate" error="pod store uninitialized" instanceID=i-testNodeManagerDefaultAllocation-0 name=node1 subsys=ipam
level=info msg="Resolving IP deficit of node" available=16 availableForAllocation=128 emptyInterfaceSlots=2 instanceID=i-testNodeManagerDefaultAllocation-0 maxIPsToAllocate=4 name=node1 neededIPs=4 remainingInterfaces=3 selectedInterface=5a76a247-3699-4c74-aca6-b074c7151dde selectedPoolID=s-1 subsys=ipam used=12
level=info msg="Synchronized ENI information for the corresponding instance" instance=i-testNodeManagerDefaultAllocation-0 numSecurityGroups=2 numSubnets=1 numVPCs=1 subsys=eni
level=warning msg="Unable to compute pending pods, will not surge-allocate" error="pod store uninitialized" instanceID=i-testNodeManagerDefaultAllocation-0 name=node1 subsys=ipam
level=info msg="Resolving IP deficit of node" available=32 availableForAllocation=112 emptyInterfaceSlots=2 instanceID=i-testNodeManagerDefaultAllocation-0 maxIPsToAllocate=1 name=node1 neededIPs=1 remainingInterfaces=3 selectedInterface=5a76a247-3699-4c74-aca6-b074c7151dde selectedPoolID=s-1 subsys=ipam used=25
level=warning msg="Subnet might be out of prefixes, Cilium will not allocate prefixes on this node anymore" node=node1 subsys=eni
level=info msg="Synchronized ENI information for the corresponding instance" instance=i-testNodeManagerDefaultAllocation-0 numSecurityGroups=2 numSubnets=1 numVPCs=1 subsys=eni
--- PASS: Test (0.02s)
    --- PASS: Test/ENISuite (0.02s)
        --- PASS: Test/ENISuite/TestNodeManagerPrefixDelegation (0.02s)
=== RUN   TestENIIPAMCapacityAccounting
--- PASS: TestENIIPAMCapacityAccounting (0.00s)
PASS
ok  	github.com/cilium/cilium/pkg/aws/eni	0.071s

[hemanthmalla]

/test