cmd/k8s-operator,k8s-operator: allow the operator to deploy exit nodes via Connector custom resource

[irbekrm]

Context

This PR adds a new .spec.exitNode field to the Connector custom resource that allows users to configure the Tailscale node created for the Connector to act as an exit node.

It also redesigns the API:

• there is a 1<->1 connection between a Connector and a Tailscale node
• a Tailscale node created for a Connector can be an exit node, a subnet router or both (and in future also an app connector)

apiVersion: tailscale.com/v1alpha1
kind: Connector
metadata:
  name: prod
spec:
  tags:
  - "tag:prod"
  hostname: ts-prod
  subnetRouter:
    advertiseRoutes:
    - "10.40.0.0/14"
    - "192.168.0.0/14"
  exitNode: true

(For the above Connector the operator configures one Tailscale node that acts as an exit node and advertises routes 10.40.0.0/14, 192.168.0.0/14)

UI

• A Connector that acts as both exit node and advertises two routes:

$ kubectl get connector
NAME   SUBNETROUTES                  ISEXITNODE   STATUS
prod   10.40.0.0/14,192.168.0.0/14   true         ConnectorCreated

• A Connector that only advertises routes:

$ kubectl get connector
NAME   SUBNETROUTES                  ISEXITNODE   STATUS
prod   10.40.0.0/14,192.168.0.0/14   false        ConnectorCreated

• A Connector that is only an exit node:

$ kubectl get connector
NAME   SUBNETROUTES   ISEXITNODE   STATUS
prod                  true         ConnectorCreated

Notes:

• (same as with other resources managed by the operator) tags currently cannot be updated post-creation
• (same as with other resources managed by the operator) multiple replicas are not supported
• I have tested this on GKE (with ubuntu nodes) and using a linux client to reach out to internet via the exit node and access Pods via the subnet router.

See details below for how to try this out:

• create an operator image from this PR

• apply the CRD kubectl apply -f ./cmd/k8s-operator/deploy/crds/tailscale.com_connectors.yaml

• install the operator with the custom image and connector functionality enabled i.e helm install operator tailscale/tailscale-dev --set enableConnector=true.... Ensure that you are using unstable-v1.57.42 or newer

• (if using the example manifest below) update ACL tags to allow the operator to own tag:prod and to allow tag:prod to auto-approve exit nodes and the desired subnet routes

• apply a Connector CR, such as the example one added in this PR- kubectl apply -f ./cmd/k8s-operator/deploy/examples/connector.yaml

• verify ready Connector status and that the subnet router and exit node are functioning

Updates #10708

[irbekrm]

This change is moving existing STS configuration around so that it is ordered by type of configuration options.

[irbekrm]

This new file contains utils shared between ingress/egress tests and connector tests, this is partially just moved from operator_test.go, connector_test.go

[irbekrm]

Thank you for the review @knyar , I have now addressed all the comments PTAL

[btobolaski]

I'm not sure this is the appropriate place for this given that it has been merged. I'm trying to utilize this functionality as it would greatly simplify something I'm currently attempting to do. I'm on the specified unstable-v1.57.42, I tried modifying the example and eventually just deployed the example as is and I get continuous errors during reconciliation rather than the connector being configured. The error is failed to add finalizer: Connector.tailscale.com \"prod\" is invalid. The full log of a reconciliation run are here.

[irbekrm]

Hi @btobolaski thanks for trying this out!

unstable-1.57.42 tag does not yet have contents of this PR and we have changed the CRD structure since unstable-1.57.42, I suspect that the error is because you have the CRD installed from this PR, but the operator version that tries to apply a patch with contents built against a different version.

I just pushed unstable-v1.57.65 tag that has the contents of this PR if you'd like to try it. You might want to delete the old Connector resource before upgrade.

(Just for the record- the Connector CRD has not been released yet, which is why we allowed a breaking change like this. Any changes made post 1.58 release should be backwards compatible)

[btobolaski]

That was it, thank you. Everything appears to be working as expected.