Do not allocate IPs or prefixes to trunk ENIs; enable Custom Networking before Security Groups for Pods

[jdn5126]

What type of PR is this?
bug

Which issue does this PR fix:
#2792

What does this PR do / Why do we need it:
This PR contains two updates:

• GetENINeedsIP, which tries to find an ENI in the datastore that new IPs/prefixes can be assigned to, now skips trunk ENIs, as they are managed by the VPC Resource Controller. This fixes the first issue in aws eniconfig not being honored and pods using trunk instead of ENI #2792.
• When patching CNINode with enabled features, Custom Networking is patched before Security Groups for Pods. This order is required for the VPC Resource Controller to use the proper Security Group when creating the trunk ENI. This fixes the second issue in aws eniconfig not being honored and pods using trunk instead of ENI #2792.

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:
N/A

Testing done on this change:
All existing unit tests and integration tests pass. I spun up a 1.28 cluster with Custom Networking and Security Groups for Pods configured, and I verified the security groups assigned to trunk ENIs, the placement of new pods, etc. From the issues described in #2792, everything seems to be resolved.

Will this PR introduce any new dependencies?:
No

Will this break upgrades or downgrades? Has updating a running cluster been tested?:
No, Yes

Does this change require updates to the CNI daemonset config files to work?:
No

Does this PR introduce any user-facing change?:
Yes

Stop IPAMD from assigning prefixes/IPs to trunk ENIs. Enable Custom Networking before Security Groups for Pods.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jchen6585]

LGTM

[orsenthil]

Updating the README with this detail (keeping that comprehensive might help), and aws docs separately with this detail might help too.

[orsenthil]

Please see if you would like to update the README in appropriate section.

[haouc]

lgtm

[jayanthvn]

lgtm...

[Shadowssong]

Hey @jdn5126 , has this been officially released yet? I was trying to find it in the changelog or in the github diffs but I can't seem to find it anywhere. We ran into this when upgrading our CNI from 1.15 -> 1.16 -> 1.17. Thanks!

[jdn5126]

@Shadowssong this PR was released with https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.16.4

[Shadowssong]

@jdn5126 thanks! We were on 1.17 and still experiencing the exact same issue described in the associated issue. We reverted to 1.15 and the issue resolved. I will open a support ticket to further debug this issue.

[jayanthvn]

When you upgraded from 1.15 to 1.16, can you share which patch version of 1.15 and 1.16?

[Shadowssong]

@jayanthvn We specified v1.16.4-eksbuild.2 for 1.16 and then for 1.17 we did v1.17.1-eksbuild.1. We were on 1.17 when we observed the behavior was occurring and reverting the EKS addon back to 1.15 resolved the issue once the daemonset restarted.