1.x: Add optional provisioning of ipv4 delegated prefix IPs

[jblosser]

Issue #, if available:
57

Description of changes:

Adds management of the individual addresses inside an ipv4 delegated prefix if the ifcfg option 'EC2PROVISIONPFXIPS' is set. The option defaults to 'no' for back-compat with current behavior.

This uses ipcalc(1) to determine start and stop addresses of the range, which is already included in the initscripts package. I added it as Requires in the spec file, but didn't adjust the Changelog.

Addresses #57

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[nmeyerhans]

Hi @jblosser. Thanks for putting this together. At first glance, I don't have any problems with it, but will need to spend a bit more time on it. It's an interesting use case that I had not specifically considered before. Most prefix delegation use cases involve assigning IPs to locally hosted containers, not directly to the ENI interface itself...

[jblosser]

Yeah, the docs for prefixes are pretty clear that's the expected use case. As a high volume email SaaS provider we have to manage a lot of outgoing IP traffic as well, from thousands of IPs. We used to publicly egress from AWS EIPs but eventually moved to our own router hardware because of limitations there, however we're evaluating moving some burst back through EC2 NAT and the option of increased IP density is a big deal for that use case. I can have our AWS TAM reach out if you'd like to discuss specifics more. Ultimately we'll need this if we go that route so if nothing else we'll be using it locally.

[nmeyerhans]

The questions I'd like to answer before merging are:

1. What's the story for the main branch (2.x releases)? Generating the additional systemd-networkd configuration won't be difficult, but we don't have single configuration file comparable to the old ifcfg-ethX files, so there's no clear place to add the toggle.
2. What's the IPv6 story? We can't enumerate and locally assign all 4 billion+ addresses in the /80 prefixes. My inclination is to continue to ignore IPv6 for the 1.x branch, as you've done.

For the 2.x branch, with either protocol, additional IPs can be assigned to an interface by generating drop-in conf files for the interface in question, and the configuration defined within them will be respected by systemd-networkd and (as long as the conf file names don't conflict with the ones we generate) will not be removed or overwritten. Because these drop-in files will be respected, it may be reasonable to again leave all of this out of scope for 2.x, instead allowing the administrator to drop config files into place as needed to assign some subset of addresses of either protocol to a local interface.

[jblosser]

Good questions and I agree with your inclinations.

1. For the 2.x branch, existing support for conf files covers what we'd need. Hooks or conf files like that are a normal model to add network configuration on various distros even pre-systemd and it was actually the first thing I checked the code for. It was only after I didn't find anything obvious that I went the route of adding support for auto detection and provisioning since that's how other secondary IPs are handled today. I think for 1.x this approach is fine for consistency. But having the option in 2.x to just have glue read the meta and write config files that are respected would be fine for our use case when we move to AL2022.
2. I agree with continuing to ignore IPv6 for this as well. Having all the IPs provisioned on the node directly is inherently about NAT use cases, and NAT is inherently a v4 problem. In v6 the use cases are probably exclusively going to be routing, and that's already covered.

[halfdime-code]

Should we set some error handling or bounds on the size of the delegation here? It looks to me like I could conceivably delegate a /8 here. What would those bounds sensibly be?

[jblosser]

The prefix delegation feature today allows /28 and only /28 as the prefix for IPv4[1]. You can add one of those per IP slot on each ENI other than the primary IP, which today is a max of 15 ENIs * 49 slots. That's 11,775 IPs max on one instance. Our use case could conceivably use all of that, though we're more likely to spread around more.

Of course, the current /28 limit could get raised later? Still, this is only about auto-provisioning--if they have a way to delegate an /8 to the instance they're going to be able to provision it manually up to any limits the kernel imposes. I don't think the tooling should second guess that, especially since they have to set the config flag to get the effect at all.

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-prefix-eni.html

[halfdime-code]

TIL. Works for me.

[nmeyerhans]

This will be included in an upcoming 1.7.0 release.

[nmeyerhans]

4 billion ipv6 addresses on an interface might lead to problems. ;)