tailscale-operator not working on AKS #8111

tiesmaster · 2023-05-10T09:05:43Z

What is the issue?

I want to expose my AKS kubernetes cluster via Tailscale, and tried to do that via the Tailscale proxy, however, that doesn't seems to proxy the traffic to the destination IP. As suggested here, I tried out the Tailscale Operator, hoping that that might work, however, I'm running into the same issue here.

To troubleshoot the issue, I've ramped up a secondary cluster, using microk8s, and I've got the Tailscale Operator working there, as you can see in the logs of the proxy pod:

Proxy pod on AKS

$ kubectl logs -n tailscale pod/ts-sample-workload-one-bjvhs-0
...
2023/05/10 08:27:55 Accept: TCP{100.115.48.77:64882 > 100.79.209.22:80} 52 tcp ok
2023/05/10 08:27:56 Accept: TCP{100.115.48.77:64882 > 100.79.209.22:80} 52 tcp ok

Proxy pod on microk8s

$ kubectl logs -n tailscale pod/ts-sample-workload-one-7nst7-0
...
2023/05/10 08:27:40 Accept: TCP{100.115.48.77:64880 > 100.67.186.56:80} 52 tcp ok
2023/05/10 08:27:40 Accept: TCP{100.67.186.56:80 > 100.115.48.77:64880} 52 ok out
2023/05/10 08:27:40 Accept: TCP{100.115.48.77:64880 > 100.67.186.56:80} 40 tcp non-syn

This is what I get when I curl the Tailscale IP of the sample workload (which is this Deployment, and Service that I've used for that). When I hit the AKS cluster, nothing happens, when I hit the microk8s cluster, I get the expected output.

I've looked at the source of the Tailscale proxy image (which turns out to also be used under the hood of the Tailscale Operator), I see that it basically only sets up some iptables rules. So this is something you should also see in the proxy containers on both clusters. I don't have any experience with iptables, and I don't really know how to troubleshoot this, but I've found a couple commands to inspect things. I see that the AKS one has some rules installed, but also missing quite a few.

Is this a bug in the Tailscale proxy image? Or should I report this to AKS?

iptables output on AKS

/ # iptables -nvL
Chain INPUT (policy ACCEPT 682 packets, 287K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 605 packets, 173K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain ts-forward (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ts-input (0 references)
 pkts bytes target     prot opt in     out     source               destination
/ # iptables -vL
Chain INPUT (policy ACCEPT 682 packets, 287K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 605 packets, 173K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain ts-forward (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ts-input (0 references)
 pkts bytes target     prot opt in     out     source               destination
/ # iptables -w -t nat -L "PREROUTING"
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       all  --  anywhere             100.79.209.22        to:10.0.45.41

iptables output on microk8s

/ # iptables -nvL
Chain INPUT (policy ACCEPT 100 packets, 25080 bytes)
 pkts bytes target     prot opt in     out     source               destination
  100 25080 ts-input   all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ts-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 93 packets, 18922 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain ts-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  tailscale0 *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x40000/0xff0000
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x40000/0xff0000
    0     0 DROP       all  --  *      tailscale0  100.64.0.0/10        0.0.0.0/0
    0     0 ACCEPT     all  --  *      tailscale0  0.0.0.0/0            0.0.0.0/0

Chain ts-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  lo     *       100.67.186.56        0.0.0.0/0
    0     0 RETURN     all  --  !tailscale0 *       100.115.92.0/23      0.0.0.0/0
    0     0 DROP       all  --  !tailscale0 *       100.64.0.0/10        0.0.0.0/0
/ # iptables -w -t nat -L "PREROUTING"
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       all  --  anywhere             default-sample-workload-one-1.shark-egret.ts.net  to:10.152.183.137

Steps to reproduce

Deploy an AKS cluster:
Deploy the Tailscale Operator
Deploy some sample workload that has the loadBalancerClass set to tailscale, like this one:
- kubectl apply -f https://gist.githubusercontent.com/tiesmaster/d7b397f19015514451fd0cd58b37fb06/raw/ed83bca958ee43ab107f2468a849918d4d0da87f/sample-workload-one.yaml
Hit the sample workload endpoint with curl

Are there any recent changes that introduced the issue?

No response

OS

Other

OS version

AKS with Kubernetes version 1.26.3

Tailscale version

No response

Other software

No response

Bug report

No response

The text was updated successfully, but these errors were encountered:

rodrigc · 2023-06-16T15:06:22Z

I suspect that these issues might be related:

#8244
#5621
#391

tiesmaster · 2023-06-19T08:52:02Z

@rodrigc Awesome! Thanks for the links. I'm gonna do some reading up on those, and report back here with my findings

wadhah101 · 2023-06-29T10:10:34Z

@tiesmaster Exact same issue with AKS and tailscale proxy

tiesmaster · 2023-06-30T08:45:52Z

@wadhah101 Thanks for the +1! I suspected it would brake for others on AKS (should be, right?), though, good to know, and good to know that others would be helped with getting this fixed (and perhaps pushed upstream). I looked into the other issues briefly, but didn't manage to do a write-up here.

What I found out is that it looked like that AKS is not using iptables-ng, as opposed to what you would be expecting from the other issues. So I cannot explain this, so far. I going to look into this over the course of the next week. I'll keep you posted when I have some results.

wadhah101 · 2023-06-30T08:50:23Z

@tiesmaster Thanks for your response, I lack the skills to debug this unfortunately. Thanks for keeping everyone posted 🙏

tiesmaster · 2023-07-20T17:34:28Z

@wadhah101 Quick update here, as I didn't manage to look into this the last couple of weeks, and I will be on holiday from now on, for over 2 weeks. Next, I don't know if I'll be able to work on this even after that for the short term, so I cannot give any guarantees. I'll keep you posted when I do.

rodrigc · 2023-07-20T20:36:15Z

@tiesmaster @wadhah101 wadhah101 Major update posted by @KevinLiang10 here:
#391 (comment)

It may or may not help with your issue, but is worth testing

clarkezone · 2023-07-29T00:07:56Z

I'm having a similar issue on a k3s cluster #8733 wondering if this could be related.

…le TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555 Fixes tailscale#8111, tailscale#8733

Add flag to k8s-operator to enable TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555 Fixes tailscale#8111, tailscale#8733

Add flag to k8s-operator to enable TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555 Fixes tailscale#8111, tailscale#8733 Signed-off-by: James Clarke <james@clarkezone.net>

clarkezone · 2023-07-30T18:22:28Z

Based on my testing, this issue appears to be resolve by the experimental NFTABLES support that was recently merged. I developed a PR to enable this in the k8s-operator and was able to get the operator working in an AKS cluster correctly as well as in my k3s case above. If you want to try my fix, you can patch the operator.yaml file as follows to pick up my privates and enable NFT mode. Pertinent changes are operator image, PROXY_IMAGE and new OPERATOR_USENFT env var:

 containers:
         - name: operator
           image: clarkezone/tsoperatornftfix:latest
           resources:

…

          - name: CLIENT_SECRET_FILE
             value: /oauth/client_secret
          - name: PROXY_IMAGE
             value: "clarkezone/tsclientnftfix:latest"
          - name: PROXY_TAGS
…

          - name: OPERATOR_USENFT
             value: "true"

rodrigc · 2023-07-30T18:51:53Z

@clarkezone That's great that you confirmed the fix.
Even better you submitted a patch!

In your patch to the operator, is it possible to somehow detect that nftables exists on the
system, and then spit out a log informing the user that they should set this particular variable to get things working?

Bravo for your work and @KevinLiang10 's original patch!

KevinLiang10 · 2023-07-31T00:11:42Z

@rodrigc Hi I'm working on detecting wether nftables/iptables available or used on machine, and just use it. Tho the implementation is not targeting to solve k8s problems, rather an implementation to relief user from having to explicit set env var to use the new nftables feature.

spit out a log informing the user that they should set this particular variable to get things working

Thanks for this advice we will discuss about adding it to our run time logging. It's because we are detecting nftables/iptables support at runtime.

I'll link these k8s issues when I put the pr up and test if the solution would help on this issue!

rodrigc · 2023-08-02T12:23:38Z

@KevinLiang10 Nice work. Long term the heuristics you are implementing as part of #8762 to auto-select
iptables/nftables and choose iptables or nftables is the right approach.
Long term, having the user depend on too many weird knobs is not ideal.

In #8762, if you can add good logging, so that as you develop the feature, end-users can get some
idea how the feature is working in their environment, that would be good.
That will also help identify any weird corner cases where your heuristic logic breaks down.

KevinLiang10 · 2023-08-02T19:53:16Z

@rodrigc I've added logs for which firewall tool is being used, and the tool's condition. Apart from that what logs do you think are particularly interesting for you?

Really would love to hear what you are interested, your opinions will definitely help us to improve user experience!

rodrigc · 2023-08-02T20:18:41Z

@KevinLiang10 These are some ideas I have of things to log:

What firewall tool is being used?
What is the version of the firewall tool?
What is the version of the kernel?
What firewall capability is available in the kernel?
What is the version of the firewall capabilty compiled into the kernel?
If it is available, output of lsb_release -a is sometimes helpful for quickly figuring out version/distro information

Basically, anything that can help the user debug
when things go wrong, with iptables / nftables

rodrigc · 2023-08-08T19:00:52Z

@KevinLiang10 the other perspective I would suggest to you, is
from your perspective at Tailscale,
what kinds of information/logs would you want the user to provide to you in a bug report,
to better troubleshoot this particular feature?

I gave you a list of what I think are good pieces of information to log,
but as the author of this feature, you might have more ideas.

mts-dyt · 2024-06-26T09:52:14Z

HI,

I still have the issue with Tailscale proxy event after setting the following flags:

    - name: TS_DEBUG_FIREWALL_MODE
      value: "auto"
    - name: TS_TEST_USENFT
      value: "true"

Here are the logs that confirm the new options are taken into account:

024/06/26 09:47:42 envknob: TS_DEBUG_FIREWALL_MODE="auto"
2024/06/26 09:47:42 logpolicy: using system state directory "/var/lib/tailscale"
2024/06/26 09:47:42 got LocalBackend in 607ms
2024/06/26 09:47:42 Start
2024/06/26 09:47:43 timeout waiting for initial portlist
...
2024/06/26 09:47:41 link state: interfaces.State{defaultRoute=eth0 ifs={eth0:[172.22.16.55/20 llu6]} v4=true v6=false}
2024/06/26 09:47:41 onPortUpdate(port=47377, network=udp6)
2024/06/26 09:47:42 router: nftables rule count: 0, iptables rule count: 0
2024/06/26 09:47:42 router: nftables is available
2024/06/26 09:47:42 router: netfilter running in nftables mode, v6 = true

The service is unreachable from my laptop:

 ✘ curl http://proxy-sandbox -vvv
* Host proxy-sandbox:80 was resolved.
* IPv6: (none)
* IPv4: 100.107.41.58
*   Trying 100.107.41.58:80...
* connect to 100.107.41.58 port 80 from 100.107.67.15 port 55649 failed: Connection refused
* Failed to connect to proxy-sandbox port 80 after 52 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to proxy-sandbox port 80 after 52 ms: Couldn't connect to server

Logs on proxy side:

2024/06/26 09:50:06 Accept: TCP{100.107.67.15:55541 > 100.107.41.58:80} 64 tcp ok
2024/06/26 09:50:06 Accept: TCP{100.107.41.58:80 > 100.107.67.15:55541} 40 ok out
2024/06/26 09:50:06 magicsock: disco: node [uYL6L] d:a26334348afac49c now using 109.26.56.23:41641 mtu=1360 tx=b6c1d2257a8a
2024/06/26 09:50:22 Accept: TCP{100.107.67.15:55647 > 100.107.41.58:80} 64 tcp ok
2024/06/26 09:50:22 Accept: TCP{100.107.41.58:80 > 100.107.67.15:55647} 40 ok out

tiesmaster added bug Bug needs-triage labels May 10, 2023

DentonGentry added kubernetes L2 Few Likelihood P2 Aggravating Priority level T5 Usability Issue type and removed needs-triage labels May 27, 2023

DentonGentry changed the title ~~Tailscale proxy or tailscale-operator not working on AKS?~~ tailscale-operator not working on AKS Jun 4, 2023

clarkezone added a commit to clarkezone/tailscale that referenced this issue Jul 30, 2023

cmd/containerboot, cmd/k8s-operator:

32da9b5

Add flag to k8s-operator to enable TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555 Fixes tailscale#8111, tailscale#8733

clarkezone mentioned this issue Jul 30, 2023

cmd/containerboot, cmd/k8s-operator: Add flag to k8s-operator and proxy to enable NFTABLES #8749

Closed

KevinLiang10 mentioned this issue Aug 1, 2023

wgengine/router: add auto selection heuristic for iptables/nftables #8762

Merged

KevinLiang10 closed this as completed in #8762 Aug 8, 2023

rodrigc mentioned this issue Sep 11, 2023

Tailscale Operator does not pass TS_DEBUG_FIREWALL_MODE to pods that it creates #9310

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tailscale-operator not working on AKS #8111

tailscale-operator not working on AKS #8111

tiesmaster commented May 10, 2023

rodrigc commented Jun 16, 2023

tiesmaster commented Jun 19, 2023

wadhah101 commented Jun 29, 2023

tiesmaster commented Jun 30, 2023

wadhah101 commented Jun 30, 2023 •

edited

Loading

tiesmaster commented Jul 20, 2023

rodrigc commented Jul 20, 2023 •

edited

Loading

clarkezone commented Jul 29, 2023

clarkezone commented Jul 30, 2023

rodrigc commented Jul 30, 2023

KevinLiang10 commented Jul 31, 2023

rodrigc commented Aug 2, 2023

KevinLiang10 commented Aug 2, 2023 •

edited

Loading

rodrigc commented Aug 2, 2023

rodrigc commented Aug 8, 2023 •

edited

Loading

mts-dyt commented Jun 26, 2024 •

edited

Loading

tailscale-operator not working on AKS #8111

tailscale-operator not working on AKS #8111

Comments

tiesmaster commented May 10, 2023

What is the issue?

Steps to reproduce

Are there any recent changes that introduced the issue?

OS

OS version

Tailscale version

Other software

Bug report

rodrigc commented Jun 16, 2023

tiesmaster commented Jun 19, 2023

wadhah101 commented Jun 29, 2023

tiesmaster commented Jun 30, 2023

wadhah101 commented Jun 30, 2023 • edited Loading

tiesmaster commented Jul 20, 2023

rodrigc commented Jul 20, 2023 • edited Loading

clarkezone commented Jul 29, 2023

clarkezone commented Jul 30, 2023

rodrigc commented Jul 30, 2023

KevinLiang10 commented Jul 31, 2023

rodrigc commented Aug 2, 2023

KevinLiang10 commented Aug 2, 2023 • edited Loading

rodrigc commented Aug 2, 2023

rodrigc commented Aug 8, 2023 • edited Loading

mts-dyt commented Jun 26, 2024 • edited Loading

wadhah101 commented Jun 30, 2023 •

edited

Loading

rodrigc commented Jul 20, 2023 •

edited

Loading

KevinLiang10 commented Aug 2, 2023 •

edited

Loading

rodrigc commented Aug 8, 2023 •

edited

Loading

mts-dyt commented Jun 26, 2024 •

edited

Loading