Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Immediately forward packets received on the nebula TUN device from self to self #501

Merged
merged 3 commits into from Jun 27, 2022

Conversation

brad-defined
Copy link
Collaborator

PR #192 updated Nebula to immediately drop packets read by the Nebula TUN device and destined to the local Nebula VPN IP.

On Linux machines, the local route table will send packets destined to the local nebula VPN IP through the loopback device, bypassing Nebula all together and permitting the packets to flow. On my mac, the route table instead sends these packets through the Nebula TUN device, where they are immediately dropped.

On my linux lighthouse - VPN IP routes to loopback device:

~# ip route get 172.16.200.1
local 172.16.200.1 dev lo src 172.16.200.1 uid 0 
    cache <local> 

On my mac - VPN IP routes to utun0, which is my Nebula TUN device:

nebula % route get 172.16.200.3
   route to: 172.16.200.3
destination: 172.16.200.3
  interface: utun0
      flags: <UP,HOST,DONE,LOCAL>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0      1300         0 

This diff updates Nebula to immediately forward these types of packets right back out the TUN device. This change will make my Mac (and presumably every other machine that routes the way Mac does) behave the same as Linux machines.

I also considered an alternative approach of having the locally destined packet be processed by the rest of the Nebula stack. This alternative approach would ensure the Nebula firewall rules are applied, for instance. However, this alternative approach would not behave the same as Linux (which does not apply the Nebula firewall rules to these local connections, as they bypass Nebula entirely), and would also require Nebula to handshake with itself locally - something which is not permitted elsewhere in the code for different reasons.

I also tried to address the issue by manually adding a route for my Nebula IP to route through the loopback device. This approach didn't reliably fix the issue for me - sometimes it worked, sometimes my manual route rule was applied after Nebula's route rules, or for some reason still sent my packets to the Nebula TUN device.

This PR fixes issue #493

@wadey wadey added this to the v1.6.0 milestone Nov 11, 2021
inside.go Outdated
if fwPacket.RemoteIP == f.myVpnIp {
_, err := f.readers[q].Write(packet)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In group discussion, we think this should be limited to macos only and leave the old behavior for all other oss

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrote it up with a const, in the hopes that a const gives the compiler the best opportunity to optimize the code path as it sees fit. Happy to implement the o/s specific path other ways if desired.

@brad-defined
Copy link
Collaborator Author

Also fixes issue #38

@brad-defined
Copy link
Collaborator Author

FWIW, I confirmed that on MacOS the conditional is optimized away -
there's no code associated with inside.go line 30 here:

	0x00a0 00160 (/Users/bradleyhiggins/Work/nebula/inside.go:26)	MOVD	"".fwPacket+32(FP), R4
	0x00a4 00164 (/Users/bradleyhiggins/Work/nebula/inside.go:26)	MOVWU	4(R4), R1
	0x00a8 00168 (/Users/bradleyhiggins/Work/nebula/inside.go:26)	MOVWU	124(R0), R5
	0x00ac 00172 (/Users/bradleyhiggins/Work/nebula/inside.go:26)	CMPW	R1, R5
	0x00b0 00176 (/Users/bradleyhiggins/Work/nebula/inside.go:26)	BNE	316
	0x00b4 00180 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	216(R0), R4
	0x00b8 00184 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	224(R0), R1
	0x00bc 00188 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	"".q+88(FP), R5
	0x00c0 00192 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	CMP	R1, R5
	0x00c4 00196 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	BHS	1816
	0x00c8 00200 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	LSL	$4, R5, R6
	0x00cc 00204 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	(R4)(R6), R6
	0x00d0 00208 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	ADD	R5<<4, R4, R4
	0x00d4 00212 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	8(R4), R0
	0x00d8 00216 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	40(R6), R4
	0x00dc 00220 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	"".packet+8(FP), R1
	0x00e0 00224 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	"".packet+16(FP), R2
	0x00e4 00228 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	MOVD	"".packet+24(FP), R3
	0x00e8 00232 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	PCDATA	$1, $2
	0x00e8 00232 (/Users/bradleyhiggins/Work/nebula/inside.go:31)	CALL	(R4)

I generated the output with the command go build -gcflags="-S -S" ./...

@brad-defined brad-defined merged commit 169cdbb into slackhq:master Jun 27, 2022
@brad-defined brad-defined deleted the forward-self branch June 30, 2022 15:25
johnmaguire added a commit to johnmaguire/nebula that referenced this pull request Jan 18, 2023
Fixes slackhq#493. This is borrowing the fix from slackhq#501 which was originally
scoped to use this behavior for all OSes. During review, it was scoped
down to just macOS which was known to be affected. Lacking support for
FreeBSD at the time was an oversight.

There may be other OSes which are affected by this bug, but we will wait
until we get reports to patch them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants