cmd/{k8s-nameserver,k8s-operator},k8s-operator: add a kube nameserver, make operator deploy it

[irbekrm]

This is #11017 that was already once merged, but I reverted it just before 1.64 release in #11669 as we did not have the bandwidth to review the second part (#11019) and I did not want to release half of a feature.
Below is the original PR description, the contents haven't changed:

This PR is the first part of the work towards enabling support for MagicDNS name resolution from within Kubernetes cluster.
Second part that updates the nameserver config with 'DNS records' is in #11019.
This PR:

• implements a simple nameserver, that reads hosts records from a file and responds to A record DNS queries over UDP and TCP
• adds a DNSConfig custom resource that can be used to tell the operator to deploy the nameserver
• Adds a new control loop that deploys/syncs nameserver resources in response to users creating DNSConfigs
• adds some infra for nameserver (make target etc) and CRD generation

Operator advertizes the nameserver's Service IP address on the DNSConfig status. Users can read it from there to update clusterDNS.
Example flow:

1. Apply a DNSConfig:

apiVersion: tailscale.com/v1alpha1
kind: DNSConfig 
metadata:
  name: ts-dns
spec:
  nameserver:
    image:
      repo: gcr.io/csi-test-290908/nameserver // official images are not published yet- this is from a registry of mine
      tag: v0.0.64dns

2. Wait for the DNSConfig's status to be updated:

$ kubectl get dnsconfig ts-dns
NAME     NAMESERVERIP
ts-dns   10.44.11.171

3. Update cluster DNS to add a stub nameserver for ts.net (the following example is for kube-dns, Corefile has different syntax):

apiVersion: v1
data:
  stubDomains: |
    {
      "ts.net": ["10.44.11.171"]
    }
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kube-dns
  namespace: kube-system

[oxtoacart]

Since this is in-cluster and I assume(?) peers can join/leave the tailnet fairly rapidly, I think a low TTL would be good.

[irbekrm]

I've changed that comment to state that TTL is currently 0, but may want to look at what's better in future

[oxtoacart]

Worth doing now?

[irbekrm]

As we discussed later, added a comment now to mention that IPv6 Pod CIDR ranges are as of now less common, so should be ok to not do it in this PR

[oxtoacart]

Neat pattern!

[oxtoacart]

You can achieve better concurrency here by using a sync.RWMutex and using n.mu.RLock() here.

[irbekrm]

So, I was thinking about this earlier. If I undestand it correctly sync.RWMutex will prefer writers if writers are around. And cluster updates that would result in new writes often tend to come in batches (i.e a user applies an update that creates X number of things that are new tailscale proxies or a cluster is being upgraded, so a lot of things move around and change Pod IPs). With that (and also bc the records are not cached), I'd be worried that any such cluster change would result in MagicDNS names not resolvable for a period of time because all reads are waiting for all writes to finish.
Although it's true that we don't need to lock on reads..

[irbekrm]

maybe the answer is to allow caching
Or, perhaps we need to add some metrics to this so we can trace what's actually happening

[oxtoacart]

Why not just configure a.logger itself with "dnsConfig"?

[irbekrm]

This logger takes as args key/value pairs where, in this case, key is dnsConfig (type of the resource) and value will be whatever is the name of the dnsConfig resource this method is currently reconciling (req.Name), example output with a different keys/values from here

{"level":"debug","ts":"2024-04-30T10:26:33Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-kuard-gccxj","service-ns":"default","service-name":"kuard"}

[oxtoacart]

Worked in my testing, generally LGTM, just a few suggestions.

[irbekrm]

Thanks for review @oxtoacart , I am going to merge this and rebase #11019 .
I am going to think about and experiment with #11919 (comment) a bit more separately (will create an issue once these three initial pieces get merged)