High CPU and network bandwidth usage at special gateways

[mritd]

What is the issue?

Operating environment

• run using the command line
• tailscale(d) version: v1.30.2
• golang build version: go1.19.2
• System version: macOS 12.6 (21G115) / M1 Max(arm64)

What happened

The tailscaled process consumes a lot of CPU and sends a lot of traffic to the derper server.

Steps to reproduce

Allow me to start with my findings:

When used with some other network proxy tools (such as Surge), if other tools set a proxy gateway (such as Surge enhanced mode will set the default gateway to 198.18.0.1), tailsaled seems to be unable to read the default gateway, which will cause tailscaled Stuck in some infinite loop.

How to reproduce

• Download Surge App
• Turn on enhanced mode in the top menu
• Use the netstat -rn command to confirm that the default gateway is set to 198.18.0.1
• Restart tailscaled several times with tailscale down && tailscale up command
• The tailscaled process will continue to consume a lot of CPU and keep sending a lot of traffic to the derper

Log

To describe the problem more clearly, I modified the launchd configuration and generated the log(I submitted a PR #5878 incidentally to enable log files by default):

tailscaled.log

Are there any recent changes that introduced the issue?

I tried looking at the code and it seems that there is a problem in the interfaces_darwin.go code, guessing that the impactful change might start with 04c2c5bd80e86b2113d4e04e840d79ec66981fda.

OS

macOS

OS version

12.6 (21G115) / M1 Max(arm64)

Tailscale version

v1.30.2

Bug report

BUG-9f5e02a01c6c13045de0c809a1920718b3a3e585e740cb2555da520c13491e40-20221009030245Z-b4a892d63e1da8c7

[mritd]

The current location is caused by the receiveIPv4 method in magicsock.go

[mritd]

Well, I don't fully understand the code; but I have it working fine with some modifications:

diff --git a/net/interfaces/interfaces_darwin.go b/net/interfaces/interfaces_darwin.go
index 60335bf8..a72c6fd5 100644
--- a/net/interfaces/interfaces_darwin.go
+++ b/net/interfaces/interfaces_darwin.go
@@ -59,7 +59,6 @@ func DefaultRouteInterfaceIndex() (int, error) {
 	if err != nil {
 		return 0, fmt.Errorf("route.ParseRIB: %w", err)
 	}
-	indexSeen := map[int]int{} // index => count
 	for _, m := range msgs {
 		rm, ok := m.(*route.RouteMessage)
 		if !ok {
@@ -73,17 +72,9 @@ func DefaultRouteInterfaceIndex() (int, error) {
 		if rm.Flags&RTF_IFSCOPE != 0 {
 			continue
 		}
-		indexSeen[rm.Index]++
+		return rm.Index, nil
 	}
-	if len(indexSeen) == 0 {
-		return 0, errors.New("no gateway index found")
-	}
-	if len(indexSeen) == 1 {
-		for idx := range indexSeen {
-			return idx, nil
-		}
-	}
-	return 0, fmt.Errorf("ambiguous gateway interfaces found: %v", indexSeen)
+	return 0, errors.New("no gateway index found")
 }
 
 func init() {

My understanding is that the DefaultRouteInterfaceIndex method is trying to find the main route interface, but I think if multiple interfaces are found, it is better to use the first one found instead of the hardcoded index 0 interface used (my understanding may be wrong) . Finally I provide my routing table for reference:

Routing tables

Internet:
Destination        Gateway            Flags           Netif Expire
default            192.168.8.1        UGScg             en6       
default            192.168.8.1        UGScIg            en0       
default            link#18            UCSIg       bridge100      !
1                  198.18.0.1         UGSc            utun0       
2/7                198.18.0.1         UGSc            utun0       
4/6                198.18.0.1         UGSc            utun0       
8/5                198.18.0.1         UGSc            utun0       
9.111.111.11/32    utun4              USc             utun4       
10                 192.168.8.1        UGSc              en6       
10.10.1/24         utun4              USc             utun4       
10.24.0.1/32       utun4              USc             utun4       
10.24.0.2/32       utun4              USc             utun4       
10.24.0.3/32       utun4              USc             utun4       
10.24.0.4/32       utun4              USc             utun4       
10.24.0.5/32       utun4              USc             utun4       
10.24.0.6/32       utun4              USc             utun4       
10.24.0.7/32       utun4              USc             utun4       
10.24.0.8          10.24.0.8          UH              utun4       
10.24.0.9/32       utun4              USc             utun4       
10.24.0.10/32      utun4              USc             utun4       
10.24.0.11/32      utun4              USc             utun4       
10.24.0.12/32      utun4              USc             utun4       
10.24.0.13/32      utun4              USc             utun4       
10.24.0.14/32      utun4              USc             utun4       
10.24.0.15/32      utun4              USc             utun4       
10.24.0.16/32      utun4              USc             utun4       
10.24.0.17/32      utun4              USc             utun4       
16/4               198.18.0.1         UGSc            utun0       
32/3               198.18.0.1         UGSc            utun0       
64/2               198.18.0.1         UGSc            utun0       
82.111.111.111/32  192.168.8.1        UGSc              en6       
100.100.100.100/32 utun4              USc             utun4       
103.111.11.1/32    192.168.8.1        UGSc              en6       
124.111.111.111/32 192.168.8.1        UGSc              en6       
127                192.168.8.1        UGSc              en6       
127.0.0.1          127.0.0.1          UH                lo0       
128.0/1            198.18.0.1         UGSc            utun0       
169.254            link#13            UCS               en6      !
169.254            link#15            UCSI              en0      !
172.16/12          192.168.8.1        UGSc              en6       
172.16.4/24        utun4              USc             utun4       
172.16.11/24       utun4              USc             utun4       
173.111.111.111/32 192.168.8.1        UGSc              en6       
192.168.0/16       192.168.8.1        UGSc              en6       
192.168.8          link#13            UCS               en6      !
192.168.8          link#15            UCSI              en0      !
192.168.8.1/32     link#13            UCS               en6      !
192.168.8.1        94:83:c4:11:ac:cc  UHLWIir           en6   1193
192.168.8.1        94:83:c4:11:ac:cc  UHLWIir           en0   1193
192.168.8.1/32     link#15            UCSI              en0      !
192.168.8.11/32    link#13            UCS               en6      !
192.168.8.100      4e:6a:7c:7:55:55   UHLWI             en6    516
192.168.8.100      4e:6a:7c:7:55:55   UHLWI             en0    516
192.168.8.147      a2:11:8b:c9:11:b2  UHLWI             en6    376
192.168.8.147      a2:11:8b:c9:11:b2  UHLWIi            en0    737
192.168.8.163/32   link#15            UCS               en0      !
192.168.8.222      3c:11:fb:18:f8:4b  UHLWI             en6    986
192.168.8.222      3c:11:fb:18:f8:4b  UHLWI             en0    986
192.168.8.249      4e:11:41:18:ee:6c  UHLWIi            en6   1189
192.168.8.249      4e:11:41:18:ee:6c  UHLWI             en0    991
192.168.11         utun4              USc             utun4       
192.168.20         utun4              USc             utun4       
192.168.30         utun4              USc             utun4       
192.168.40         utun4              USc             utun4       
192.168.105        link#18            UC          bridge100      !
192.168.105.1      f6.d4.81.15.1b.64  UHLWIi            lo0       
198.18.0/15        198.18.0.1         UGSc            utun0       
198.18.0.1         198.18.0.1         UH              utun0       
224.0.0/4          link#13            UmCS              en6      !
224.0.0/4          link#15            UmCSI             en0      !
224.0.0.251        1:0:5e:0:0:fb      UHmLWI            en6       
224.0.0.251        1:0:5e:0:0:fb      UHmLWI            en0       
224.0.0.251        1:0:5e:0:0:fb      UHmLWIg     bridge100       
239.255.255.250    1:0:5e:11:ff:fa    UHmLWI            en6       
255.255.255.255/32 link#13            UCS               en6      !
255.255.255.255/32 link#15            UCSI              en0      !

[knyar]

My understanding is that the DefaultRouteInterfaceIndex method is trying to find the main route interface, but I think if multiple interfaces are found, it is better to use the first one found instead of the hardcoded index 0 interface used (my understanding may be wrong)

DefaultRouteInterfaceIndex expects to find at most one network interface that has a route configured without the RTF_IFSCOPE flag (the I in netstat -r output). Since it examines both v4 and v6 routing tables and does not actually look at the route prefix, the ambiguous gateway interfaces found error can appear when v4 and v6 routes use different interfaces, or when a machine has static routes configured on a non-default interface.

For example, on my Mac that has two active interfaces and a static route, DefaultRouteInterfaceIndex will find three different interfaces as evidenced by a simple tool that duplicates its logic:

$ go run routes.go
interface 22, flags [RTF_UP RTF_STATIC RTF_PRCLONING RTF_GATEWAY]
- 0.0.0.0
- 10.11.12.1
- 0.0.0.0
- 10.11.12.33
interface 24, flags [RTF_STATIC RTF_GATEWAY RTF_PRCLONING RTF_UP]
- 10.99.99.0
- 192.168.56.2
- 255.255.255.0
- 192.168.56.1
interface 6, flags [RTF_PRCLONING RTF_GATEWAY RTF_UP]
- fd80:c6e0:719b::
- fe80::43c:cb5b:5fa9:aa4c
- ffff:ffff:ffff:ffff::
- fe80::8c4:7580:ff5e:a396

Routing table of this machine:

Internet:
Destination        Gateway            Flags           Netif Expire
default            10.11.12.1         UGScg            en11
default            10.11.12.1         UGScIg            en0
[dd]
10.99.99/24        192.168.56.2       UGSc         vboxnet0
[dd]

Internet6:
Destination                             Gateway                         Flags           Netif Expire
default                                 fe80::%utun0                    UGcIg           utun0
default                                 fe80::%utun1                    UGcIg           utun1
default                                 fe80::%utun2                    UGcIg           utun2
default                                 fe80::%utun4                    UGcIg           utun4
default                                 fe80::%utun5                    UGcIg           utun5
::1                                     ::1                             UHL               lo0
fd80:c6e0:719b::/64                     fe80::43c:cb5b:5fa9:aa4c%en0    UGc               en0

I think a reasonable way forward would be to adjust DefaultRouteInterfaceIndex to filter by route prefix and only care about interfaces that have routes to 0.0.0.0/0 or ::/0. If default routes for IPv4 and IPv6 go via different interfaces, we could either return the first one, or add logic to prefer one over the other. @bradfitz – any thoughts?