FR: socks/http proxy by kubernetes operator

[mweibel]

What are you trying to do?

We have ArgoCD running which needs to access our git instance accessible via tailnet. ArgoCD pulls certain repositories via git URL and therefore needs access to the tailnet.

The Kubernetes operator does not yet seem to support a way to set the TS_SOCKS5_SERVER environment variable (or other proxy env variables). This means we had to create a separate sidecar which we then can use as a proxy target.

How should we solve this?

Add the ability to specify proxy options for exposing an egress service. Could be done by adding more annotations or using a separate CRD (or maybe the ProxyClass CRD can be reused?).

What is the impact of not solving this?

As mentioned - separate sidecar. The trouble with managing a separate sidecar is that we didn't want to use an auth key since they auto-expire after 90 days and the pods running can be recreated any time. This means we had to settle for an Oauth2 client which means the container has (a) too much permissions (device write) and (b) every time a new pod gets scheduled, the respective machine needs to be manually approved.

Anything else?

No response

[irbekrm]

@mweibel you can configure env vars for the operator proxies via ProxyClass https://github.com/tailscale/tailscale/blob/main/k8s-operator/apis/v1alpha1/types_proxyclass.go#L179

[mweibel]

Thanks @irbekrm! While I looked at ProxyClass I somehow overlooked this possibility. Will test and report back 👍

[irbekrm]

Thanks, it was added quite recently. Let us know how it goes- it might anyway make sense to make something custom for this.

[mweibel]

I tried to make it work but couldn't. I believe the problem is that we can't specify ports for service or the sts pod so the socks5 port doesn't get exposed.

setup to reproduce:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: socks5-proxy
spec:
  statefulSet:
    pod:
      tailscaleContainer:
        env:
          - name: TS_SOCKS5_SERVER
            value: ":1055"
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    tailscale.com/tailnet-fqdn: service.example.ts.net
  labels:
    tailscale.com/proxy-class: "socks5-proxy"
  name: service
spec:
  externalName: placeholder
  type: ExternalName

$ kubectl debug -it pod/<existing-pod> --image=nicolaka/netshoot

existing-pod ~ curl -v -x socks5://service:1055 https://device-name-in-tailnet
* Host service:1055 was resolved.
* IPv6: (none)
* IPv4: <stsPodIp>
*   Trying <stsPodIp>:1055...
* connect to <stsPodIp> port 1055 from <sourcePod> port 47024 failed: Connection refused
* Failed to connect to service port 1055 after 82 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to service port 1055 after 82 ms: Couldn't connect to server

I believe to make this work we'd need to be able to specify which port to expose in the STS and possibly also in the associated services.
I tried to manually edit the service/sts but of course the operator immediately overwrites any changes. I didn't yet try to stop the operator to test it out - that's one thing I could try. But maybe another day 😄

[irbekrm]

I believe to make this work we'd need to be able to specify which port to expose in the STS and possibly also in the associated services.

Could you give us a bit more detail about the setup - is the ArgoCD what is running in the cluster and if so and you are trying to access git over tailnet why does cluster egress not work for you?

I didn't yet try to stop the operator to test it out - that's one thing I could try.

Yes, the operator will override changes :) You can temporarily scale it down kubectl scale deploy/operator -n tailscale --replicas=0

[mweibel]

Yes, the operator will override changes :) You can temporarily scale it down kubectl scale deploy/operator -n tailscale --replicas=0

Yes, thanks! I didn't want to do that today but probably will need to, to be able to fully test it.

Could you give us a bit more detail about the setup - is the ArgoCD what is running in the cluster and if so and you are trying to access git over tailnet why does cluster egress not work for you?

Setup is as follows:

• git.example.com -> 100.x (public DNS entry)
• ArgoCD runs in cluster
• ArgoCD has a list of permitted repos
• each ArgoCD application specifies as the sourceURL the full repository URL which needs to match a permitted repo
• in many places (~200 places maybe) we have git@git.example.com:path/to/repo.git as the source repo URL

We were able to add the socks5 proxy to the list of permitted repositories in a central place.
While we could rewrite git.example.com to the service egress name (e.g. git) in all ArgoCD applications and in the permitted repo list, it would require some work and for readers/contributors to those YAML files, it would be less clear than fully specifying the existing git.example.com URL.

We migrated from a OpenVPN system and that's why we have existing domains which we map in a public DNS to the tailnet IP instead of e.g. specifying the magic DNS name directly.

I'm not sure if it's clear what/why - a little hard to write it succinctly. Let me know if something's not clear yet.

[mweibel]

ok I tested it but it doesn't work because with the externalName service the STS will proxy all non tailscale traffic to the target instead of being used as a proxy.