implement CIDR matching

[BenTheElder]

This is inspired by how the kernel does routing (with a LPC Trie), thanks @aojea for the pointer to https://vincent.bernat.ch/en/blog/2017-ipv4-route-lookup-linux

There is one existing go library for this at https://github.com/yl2chen/cidranger but it implements more of a set type (we need a map to AWS regions), doesn't use the newer net/netip types which are more efficient, etc.

We implement a reasonably efficient generic map type in relatively little code here (~260 lines TrieMap including comments, whitespace, license header etc.).

While the current implementation here lacks level and path compression, cidranger only implements path compression.
For AWS data memory usage is acceptable without compression (ballpark less than 1MB), while lookup performance is with the AWS IPv4 data is ~40ns / op for this implementation vs ~400ns / op for cidranger on my 16" intel macbook laptop in microbenchmarking (and both drastically better than brute force looping over CIDRs).

See: #42 (comment) with the full dataset ipv4 + ipv6 matching, performance is ~fine. We can continue optimizing in follow-ups as needed.

See: #39

/cc @dims @ameukam

[BenTheElder]

note: named return values seem to be the easiest way to return a default generic type cleanly while also documenting the return values

otherwise we would have a local type like var value V to achieve the same thing, you cannot do say return nil, false

[aojea]

👁️ /cc

[BenTheElder]

preliminary benchmarking has brute force taking 150x time, even with the current relatively simplistic implementation.

it does take longer than i'd like to initialize the trie, we may want to pre-initialize it (expected, but didn't want to do that before measuring...)

[BenTheElder]

NOTE: most of the lines in the diff are just pkg/net/cidrs/aws/internal/ranges2go/zz_generated_rawdata.go which is https://ip-ranges.amazonaws.com/ip-ranges.json stuffed into a go string by a small script. The actual diff is not that large.

[BenTheElder]

go test -benchmem -run=^$ -coverprofile=$elided -bench . sigs.k8s.io/oci-proxy/pkg/net/cidrs/aws

goos: darwin
goarch: amd64
pkg: sigs.k8s.io/oci-proxy/pkg/net/cidrs/aws
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkNewAWSRegionMapper-16       	     786	   1487210 ns/op	  870320 B/op	   29875 allocs/op
BenchmarkNewAWSegionBruteForce-16    	   96062	     12387 ns/op	       8 B/op	       1 allocs/op
BenchmarkAWSRegionTrieMap-16         	10781480	       111.2 ns/op	       0 B/op	       0 allocs/op
BenchmarkAWSRegionBruteForce-16      	   76730	     14975 ns/op	       0 B/op	       0 allocs/op
PASS
coverage: 100.0% of statements
ok  	sigs.k8s.io/oci-proxy/pkg/net/cidrs/aws	5.771s

With the real AWS data loaded including IPv6 (unless I've somehow misread these results, I don't find myself benchmarking go all that often...)

~ 0.0015 s to initialize the TrieMap and ~ 0.87 MB memory overhead (BenchmarkNewAWSRegionMapper-16).
~ 111 ns/op lookup times (BenchmarkAWSRegionTrieMap-16).

We have room to improve these if need be, but they should be plenty acceptable already.

[BenTheElder]

Also a conscious decision: we could fairly easily minify pkg/net/cidrs/aws/internal/ranges2go/zz_generated_rawdata.go but I chose not to do so, because when the data does change occasionally it will be easy to read the diff (we will see some new ranges inserted in some regions), versus it will not be so easy minified.

The file is not included at runtime anyhow, only in the repo. The raw data from AWS is pre-processed by ranges2go into pre-parsed CIDRs in a more useful format for our purposes (map[string][]netip.Prefix region : CIDRs), and that file is much smaller.

[BenTheElder]

I think the unused lint flaked:

pkg/net/cidrs/triemap.go:53:2: field `keyToValue` is unused (unused)
	keyToValue map[int]V
	^
pkg/net/cidrs/bruteforce.go:34:2: field `mapping` is unused (unused)
	mapping map[V][]netip.Prefix
	^
pkg/net/cidrs/triemap.go:54:2: field `valueToKey` is unused (unused)
	valueToKey map[V]int

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_oci-proxy/42/pull-oci-proxy-verify/1514800357945905152

does not fail locally, and by trivial inspection these errors are not correct
/retest

[BenTheElder]

/retest

[dims]

/approve
/lgtm

thanks @BenTheElder looking forward to hooking up this to the main code base next!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [BenTheElder,dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment