cmd/k8s-operator,ssh/tailssh,tsnet: optionally record 'kubectl exec' sessions via Kubernetes operator's API server proxy

[irbekrm]

Adds the ability to forward contents of 'kubectl exec' sessions over the API server proxy to a configured tsrecorder instance.

Users can now configure via grants that kubectl exec sessions over an API server proxy should be recorded.

If so, the proxy hijacks the connection, parses the byte stream as SPDY frames and, for frames that SPDY data frames for stdout/stderr data streams, forwards the payload to the configured session recorder.
If the connection to session recorder fails (either at the point where the proxy is establishing the connection or during the session) the session is only allowed to continue (without recording) if the enforceRecorder is not set to true.

Example user workflow

Scenario: enforce that tailnet users in group group:engineering must have their kubectl exec sessions recorded and sent to (one of) tsrecorder instances tagged by tag:recorder when execing over an API server proxy tagged by tag:k8s-operator:

• deploy a tsrecorder with ACL tag tag:recorder, deploy a Kubernetes operator with tag:k8s-operator configured to run API server proxy. Ensure that operator has access to the recorder (ACLs).

• configure ACLs

"grants": [
		{
			"src": ["group:engineering"],
			"dst": ["tag:k8s-operator"],
			"app": {
				"tailscale.com/cap/kubernetes": [{
					"recorder":        ["tag:recorder"],
					"enforceRecorder": true, // fail closed
				}],
			},
		},

kubectl exec sessions from group:engineering via tag:k8s-operator should now be recorded, if no recorders can be reached, the session will be refused.

Notes

• We check if the session appears to be SPDY. If it is not, we error out and the connection will be refused (even if the recording policy was enforceRecorder = false, see 40e30e5#r1649006397)

• Tailscale SSH servers are also able to store the recording on disk. For now I am not planning to add this to the operator unless there is a use case (latency?) and am assuming that everyone will be ok to deploy a tsrecorder. We are also planning on making it possible to configure the operator to deploy tsrecorder.

• Tsrecorder treats these session recordings same as any other. See UI screenshot:

Next steps

• support for websocket clients

• additional configuration in tsrecorder to show Kubernetes-specific info such as Pod name/namespace

• metrics

• look into dropping connections if ACLs are changed mid-session

• more testing around performance and the behaviour in cases of ACL reconfig etc. I have, a few times, obeserved a command on recorded session lag for a noticeable period of time before returning

Updates tailscale/corp#19821

[irbekrm]

Here we could also, if the policy was to fail open, forward the raw bytes to the original destination anyway (and stop attempting to record the next reads and writes as well?) . Perhaps we can revisit that part in a next iteration