Tailscale-operator not working on k3s cluster

[clarkezone]

### What is the issue?

I’m attempting a simple helloworld level scenario for tailscale-operator on k3s on ubnuntu 22.04 and the scenario isn't working correctly.

Steps to reproduce

1. backup kubeconfig:

mv ~/.kube/config ~/.kube/backup

2. install k3s and confirm running:

curl -sfL https://get.k3s.io | sh -
sudo mv /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chmod 777 ~/.kube/config
➜  tailscaleoperator git:(c4updates) ✗ k get nodes
NAME                   STATUS   ROLES                  AGE   VERSION
clarkezonedevbox5-tr   Ready    control-plane,master   94s   v1.27.3+k3s1

3. install tailscale operator from instructions here: https://tailscale.com/kb/1236/kubernetes-operator/

➜  tailscaleoperator git:(c4updates) ✗ k apply -f operator.yaml
namespace/tailscale created
serviceaccount/proxies created
role.rbac.authorization.k8s.io/proxies created
rolebinding.rbac.authorization.k8s.io/proxies created
serviceaccount/operator created
clusterrole.rbac.authorization.k8s.io/tailscale-operator created
clusterrolebinding.rbac.authorization.k8s.io/tailscale-operator created
role.rbac.authorization.k8s.io/operator created
rolebinding.rbac.authorization.k8s.io/operator created
secret/operator-oauth created
deployment.apps/operator created
➜  tailscaleoperator git:(c4updates) ✗ k get pods -n tailscale
NAME                       READY   STATUS    RESTARTS   AGE
operator-74cdfb6f5-gj7dw   1/1     Running   0          8s

# verify operator showing in tailnet
➜  tailscaleoperator git:(c4updates) ✗ tailscale status | grep operator
100.82.108.42   tailscale-operator   tagged-devices linux   -

4. install test workload:

k apply -f https://gist.github.com/clarkezone/b22a5851f2e4229f5fd29f1115ddee32/raw/766708eee8f614d846dc12afe4dfaa819a678ee9/tailscaletest.yaml

➜  tailscaleoperatortest git:(master) k get pods -n tailscaletest
NAME                              READY   STATUS    RESTARTS   AGE
nginx-tailscale-7bbbb87bf-2f7lc   1/1     Running   0          30s
nginx-tailscale-7bbbb87bf-wkm24   1/1     Running   0          30s
➜  tailscaleoperatortest git:(master) k get services -n tailscaletest
NAME              TYPE           CLUSTER-IP     EXTERNAL-IP                                      PORT(S)        AGE
nginx-tailscale   LoadBalancer   10.43.167.59   tailscaletest-nginx-tailscale.tail967d8.ts.net   80:30110/TCP   37s

5. Attempt to curl / ping the tailscale URL for the service (Result: it isn't working)

➜  tailscaleoperatortest git:(master) curl tailscaletest-nginx-tailscale.tail967d8.ts.net

^C
➜  tailscaleoperatortest git:(master) ping tailscaletest-nginx-tailscale.tail967d8.ts.net

PING tailscaletest-nginx-tailscale.tail967d8.ts.net (100.70.204.105) 56(84) bytes of data.
^C
--- tailscaletest-nginx-tailscale.tail967d8.ts.net ping statistics ---
11 packets transmitted, 0 received, 100% packet loss, time 10223ms

Logs from proxy pod:
logs.txt

Are there any recent changes that introduced the issue?

First time I’ve tried hence can’t comment

OS

Linux

OS version

Ubuntu 22.04.2 LTS

Tailscale version

1.47.36

Other software

v1.27.3+k3s1

Bug report

No response

[clarkezone]

At end or repro uninstall k3s and restore config,

/usr/local/bin/k3s-uninstall.sh
mv ~/.kube/backup ~/.kube/config

[DentonGentry]

2023/07/29 00:12:16 health("router"): error: setting up filter/ts-input: running [/sbin/ip6tables -t filter -N ts-input --wait]: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory ip6tables v1.8.8 (legacy): can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)                                                                                                           Perhaps ip6tables or your kernel needs to be upgraded. 

I think the fix for this is the auto-detection tracked in #391
Closing as a duplicate of #391

[spasche]

I had the same issue: on a k3s cluster, impossible to reach a service exposed on tailscale.

The error from the proxy Pod:

ts-nginx-ts-8f2zt-0 tailscale 2023/09/28 21:09:21 health("router"): error: setting up filter/ts-input: running [/sbin/ip6tables -t filter -N ts-input --wait]: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory
ts-nginx-ts-8f2zt-0 tailscale ip6tables v1.8.8 (legacy): can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
ts-nginx-ts-8f2zt-0 tailscale Perhaps ip6tables or your kernel needs to be upgraded.

I could fix it by loading the ip6table_filter module on the node:

modprobe ip6table_filter

And to make it persist on reboots:

echo ip6table_filter | sudo tee -a /etc/modules-load.d/k3s.conf