libnet/ipams/default: introduce a linear allocator

[akerouanton]

• Close IPv6 address pool subnet smaller than /80 causes dockerd to consume all available RAM #40275
• Close splitNetwork and addIntToIP fails for IPv6 addresses on /64+ size pools #42801
• Supersede / Close libnet/ipam: Lazily sub-divide pools into subnets #43033
• Stacked on top of libnet/ipam: Various clean-ups #47727

- What I did

This previous allocator was subnetting address pools eagerly when the daemon started, and would then just iterate over that list whenever RequestPool was called. This was leading to high memory usage whenever IPv6 pools were configured with a target subnet size too different from the pools prefix size.

For instance: pool = fd00::/8, target size = /64 -- 2 ^ (64-8) subnets would be generated upfront. This would take approx. 9 * 10^18 bits -- way too much for any human computer in 2024.

Another noteworthy issue, the previous implementation was allocating a subnet, and then in another layer was checking whether the allocation was conflicting with some 'reserved networks'. If so, the allocation would be retried, etc... To make it worse, 'reserved networks' would be recomputed on every iteration. This is totally ineffective as there could be 'reserved networks' that fully overlap a given address pool (or many!).

To fix this issue, a new field Exclude is added to RequestPool. It's up to each driver to take it into account. Since we don't know whether this retry loop is useful for some remote IPAM driver, it's reimplemented bug-for-bug directly in the remote driver.

The new allocator uses a linear-search algorithm. It takes advantage of all lists (predefined pools, allocated subnets and reserved networks) being sorted and logically combines 'allocated' and 'reserved' through a 'double cursor' to iterate on both lists at the same time while preserving the total order. At the same time, it iterates over 'predefined' pools and looks for the first empty space that would be a good fit.

Currently, the size of the allocated subnet is still dictated by each 'predefined' pools. We should consider hardcoding that size instead, and let users specify what subnet size they want. This wasn't possible before as the subnets were generated upfront. This new allocator should be able to deal with this easily.

The method used for static allocation has been updated to make sure the ascending order of 'allocated' is preserved. It's bug-for-bug compatible with the previous implementation.

One consequence of this new algorithm is that we don't keep track of where the last allocation happened, we just allocate the first free subnet we find.

Before:

• Allocate: 10.0.1.0/24, 10.0.2.0/24 ; Deallocate: 10.0.1.0/24 ; Allocate 10.0.3.0/24.

Now, the 3rd allocation would yield 10.0.1.0/24 once again.

As it doesn't change the semantics of the allocator, there's no reason to worry about that.

Finally, about 'reserved networks'. The heuristics we use are now properly documented. It was discovered that we don't check routes for IPv6 allocations -- this can't be changed because there's no such thing as on-link routes for IPv6.

(Kudos to Rob Murray for coming up with the linear-search idea.)

- How to verify it

CI -- a bunch of tests have been added, some have been rewritten.

Or manually by creating, deleting and re-creating networks.

- Description for the changelog

- Introduce a new subnet allocator that can deal with IPv6 address pools of any size

- A picture of a cute animal (not mandatory but encouraged)

[robmry]

LGTM!

[robmry]

This hasn't changed - but Is4 is false for an IPv4-mapped IPv6 addresses. It might be worth checking for that and storing the unmapped prefix in the IPv4 list, or just bailing out?

(We don't deal with mapped addresses in command line options either, but it might be less obvious here - the address pool just won't do anything useful. I'm not quite sure why someone would want to write IPv4 addresses as IPv6, but there's an issue somewhere asking us allow it.)

[akerouanton]

@corhere Please make sure this won't break Swarm in any way (eg. when a new leader is elected, etc...).

[corhere]

Review WIP; I haven't even finished with address_space.go. I'll be back tomorrow.

[corhere]

Since the slices package is already being used, may as well take full advantage.

Suggested change

	var last *ipamutils.NetworkToSplit
	var discarded int
	for i, imax := 0, len(predefined); i < imax; i++ {
	p := predefined[i-discarded]
	if last != nil && last.Overlaps(p.Base) {
	predefined = slices.Delete(predefined, i-discarded, i-discarded+1)
	discarded++
	continue
	}
	last = p
	}
	predefined = slices.CompactFunc(predefined, func(last, p *ipamutils.NetworkToSplit) bool {
	return last.Overlaps(p.Base)
	})

[akerouanton]

slices.CompactFunc works a bit differently. It expects a strict equality as it doesn't compare to the last non-duplicate found, but to 'current-1'. If you have the following subnets:

1. 10.0.0.0/8
2. 10.0.0.0/16
3. 10.10.0.0/16

It tries to compare s1 == s2, and then s2 == s3. That's not what we want.

[corhere]

slices.CompactFunc works a bit differently.

That's too bad; it would have been such an elegant solution!

---

I don't like that slices.Delete is being called in a loop:

Delete is O(len(s)-i)

Therefore the worst case time complexity is O(N^2).

It is doable in O(len(predefined)) time.

[akerouanton]

Yeah, at first I thought predefined wouldn't be filled with that many entries to really matter. But better safe than sorry, I guess. That's now fixed.

[corhere]

aSpace.allocated is a sorted slice, which means binary searching is possible. Turn that O(n) search into O(log n) time complexity!

func (aSpace *addrSpace) allocatePool(nw netip.Prefix) error {
	n, _ := slices.BinarySearchFunc(aSpace.allocated, nw, func(allocated, nw netip.Prefix) int { return nw.Addr().Compare(allocated.Addr()) })
	aSpace.allocated = slices.Insert(aSpace.allocated, n, nw)
	aSpace.subnets[nw] = newPoolData(nw)
	return nil
}

Also, are duplicate allocations allowed? It would be trivial to detect this situation and return an error instead of inserting the duplicate entry into the slice.

[corhere]

The new allocator should work fine with Swarm. The CNM network allocator replays allocations as static assignments if there is an existing allocation in the Swarm state.

moby/libnetwork/cnmallocator/networkallocator.go

Lines 866 to 873 in 4554d87

	// If there is non-nil IPAM state always prefer those subnet
	// configs over Spec configs.
	if n.IPAM != nil {
	ipamConfigs = n.IPAM.Configs
	} else if n.Spec.IPAM != nil {
	ipamConfigs = make([]*api.IPAMConfig, len(n.Spec.IPAM.Configs))
	copy(ipamConfigs, n.Spec.IPAM.Configs)
	}