Wait for SecondaryIPs in waitENIAttached()

[mogren]

We have seen an increase of PrivateIpAddressLimitExceeded since we started logging these errors back in May. #989

The issue seems to be that waitENIAttached() only check for the newly created ENI to be attached, not that the Secondary IPs have actually been assigned to that ENI. Because of that, the ENI gets added to the ipamd data store with no IPs, and on the next reconcile we think is empty, try to call AllocIPAddresses() on the same ENI again, which fails with PrivateIpAddressLimitExceeded that we ignore, and then go on to add those existing IPs to the data store.

We should instead wait for the secondary IPs to actually show up before adding the ENI to the data store.

[naveenb29]

what CNI version does this affect ?

[mogren]

@naveenb29 All versions seem to have had this issue since before 1.0.0. We did not log this until May, just quietly ignoring. The issue is in this check:

amazon-vpc-cni-k8s/pkg/ipamd/ipamd.go

Lines 787 to 792 in f2ad9ba

	// Verify that the ENI we are waiting for is in the returned list
	for _, returnedENI := range enis {
	if eni == returnedENI.ENIID {
	return returnedENI, nil
	}
	}

Just getting the ENI attached is not enough, we also need to get the secondary IPv4 addresses. Added in #1174

One reason it was not found earlier is that in the first call to create a new ENI and attach it, we fail to add any new IPs to the datastore. When we 5s later try again, we find the ENI that the CNI think is empty, the AssignPrivateIPAddresses call fails since they are already attached. We ignore that error since we now have the IPs, and just add them to the datastore and make them available for pods.

[mogren]

Fix merged