Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zerotier OPNSense interface binding #239

Closed
obrienmd opened this issue Aug 31, 2017 · 22 comments
Closed

Zerotier OPNSense interface binding #239

obrienmd opened this issue Aug 31, 2017 · 22 comments
Assignees
@dharrigan @fichtner
Labels
feature Adding new functionality

Comments

@obrienmd
Copy link

Continuing from opnsense/core#1775.

Per @dharrigan :

Let me do some more testing on binding to interfaces (I may need some help from the OPNsense core team on this one).

My eventual goal is to use BGP or OSPF over a Zerotier "virtual switch" between a bunch of OPNSense routers.

@dharrigan I'm no core team (or even peripheral team) member, but I'm happy to test and debug!
@obrienmd obrienmd changed the title Zerotier interface binding Zerotier OPNSense interface binding Aug 31, 2017
@fichtner
Copy link
Member

FWIW, there is a new "lock interface" feature since 17.7.1 as well, maybe that's already enough. It's untested with zerotier, so "don't do this at home" (yet).

@fichtner fichtner added the feature Adding new functionality label Aug 31, 2017
@fichtner fichtner self-assigned this Aug 31, 2017
@dharrigan
Copy link
Member

Will investigate! :-)

-=david=-

@obrienmd
Copy link
Author

Another note, it seems when I bind ZT to an interface in OPNSense and enable DHCP for IPv4 (wasn't sure what auto-assign in ZT uses), a few things happened:

  1. My system lost its default gateway, and had to be rebooted to regain it
  2. The interface didn't get an IP via auto-assign (though the ZT interface showed the IP in a shell session)

@obrienmd
Copy link
Author

obrienmd commented Aug 31, 2017 •
edited
Loading

Let me know how I can help! I'm a fairly experienced linux sysad, and semi-know my way around nets.

@StrikerTwo
Copy link

+1, lost my gateway as well when trying out Zerotier with DHCPv4 as IP scheme.

@dharrigan
Copy link
Member

Hi,

Thank you for the feedback. I hope to look into this over the course of this weekend. :-)

-=david=-

@dharrigan
Copy link
Member

@StrikerTwo can you clarify please - are you attempting to use DHCPv4 on the Zerotier interface within OPNsense, or have you defined the auto-assigned IP address range in my.zerotier.com? Thank you.

@obrienmd
Copy link
Author

obrienmd commented Sep 1, 2017

@dharrigan to speak for myself, I set that on the ZT interface in OPNSense, but also set auto-assigned IPs in my.zerotier.com (thinking ZT central would use DHCP as a protocol for ipv4, but that's probably naive).

@StrikerTwo
Copy link

@dharrigan I tried to use DHCPv4 on the Zerotier interface. My gateway was gone immediately, so I don't think it even had time to connect to Zerotier and get an IP.

@dharrigan
Copy link
Member

Hi,

I've been doing some investigation. More to do, but this works for me:

I created a new network on zerotier WITHOUT enabling autoassign of IP addresses (I believe it's best, otherwise OPNsense will get confused when it brings up/down interfaces - more investigation required). In effect, you are saying that you will assign the IP addresses to each node, either statically or via DHCP.

I can confirm that if you do that, then you can enable DHCP on the zerotier interface (after assignment) within OPNsense then other nodes joining the network will be issued DHCP addresses from the pool you specify.

I chose a subnet of 172.30.0.0/16 (others will work too) on my.zerotier.com.

I installed the plugin on OPNsense and enabled the service (so that the green play button shows).

I then added the network id I created above to the VPN...Zerotier form and enabled the network (from OPNsense's point of view).

I checked that the node (running opnsense) showed up on my.zerotier.com and authorized it to connect.

On OPNsense, I then when to interface assignments and assigned the new zerotier interface to a new interface (OPT1). I went into OPT1 and gave the interface a static IP address of 172.30.0.10/16.

On a completely separate computer in another country I installed zerotier one, joined the network created above, went into my.zerotier.com and authorized the node, then on the node I did:

ip addr add 172.30.0.20/16 dev zt0

I confirmed that the zt0 interface was assigned the IP address and that the route for 172.30.0.0/16 was set to go out via zt0.

Back on OPNsense, I went into the firewall rules configuration, selected OPT1 and allowed all traffic to pass (don't forget, that as soon as you assign an interface, that interface falls under the same firewall rules as other interfaces, so you have to allow IPV4/6 to flow as suitable for your requirements).

I was then able to ping (and ssh) from the remote node to the OPNsense box via 172.0.30.10 and vice-versa, I was able to ssh (and ping) to the remote node of 172.0.30.20 from OPNsense.

Further, I removed the IP from the remote node, then after configuring DHCP (with a pool of 172.0.30.100 to 172.0.30.200) on OPNsense on the OPT1 interface, I was able to do dhclient -v zt0 on the remote node and I was assigned an appropriate IP and was able to still ssh/ping to and from remote node and OPNsense node, and the DHCP lease showed up on OPNsense for the remote node.

So, right now, I have a bi-directional secure VPN network between two different nodes witin the zerotier network, with one of those nodes being OPNsense and allowing me to do all the interface things I would expect.

I believe, given this configuration so far, things like Quagga, or any other services that can bind to interfaces would work as expected.

I'm still doing some investigation on a few things, but let me know if you can replicate this.

I didn't observe my gateway go down. If it goes down, can you provide me some more details please?

Hope this helps!

-=david=-

@StrikerTwo
Copy link

I do not doubt that you can get it to work. Still, (maybe naively) selecting DHCPv4 on the Zerotier interface should not leave the system without a working gateway.

Maybe the problem is that my WAN address gets assigned via DHCPv4 as well?

@dharrigan
Copy link
Member

dharrigan commented Sep 3, 2017 via email

@StrikerTwo
Copy link

Yes, my LAN network (10.0.0.0/8) is quite different from the Zerotier range.

But again, I think the problem is that I selected DHCPv4 as IP method on the Zerotier interface. According to your description, you always used static IPs in OPNsense on the Zerotier interface (OPT1 in your post).

It would be okay if that didn't work and I had to assign IPs manually. But with the default gateway gone it didn't even connect to Zerotier. And Zerotier should never be the default gateway, anyway.

I can do some more testing myself next week to see if I can isolate the problem and/or provide you with logs.

@dharrigan
Copy link
Member

Hi,

Thank you. Do let me know how it goes. You can chat to me on freenode (#opnsense) if you want.

-=david=-

@StrikerTwo
Copy link

Okay, I tried again, on a fresh OPNsense VM.

zerotier-cli join worked, the device showed up on my.zerotier.com.

I then assigned the ZT interface to OPT1, activated it and set it to DHCP. And now my box is offline because it expects ZT to provide the default gateway:
image

Although my real gateway is still the default set in the GUI:
image

@dharrigan
Copy link
Member

Hi,

Did you try to set the interface to a static IP as discussed above, rather than DHCP?

@StrikerTwo
Copy link

Yes, I did. That works. But
a) I would like to use the ZT auto-assign method, and
b) the system shouldn't do something I didn't tell it to ;)

Are you saying DHCP is generally unsupported with the ZT plugin? Because I could live with that, but it should be documented somewhere.

@dharrigan
Copy link
Member

dharrigan commented Sep 4, 2017 via email

@StrikerTwo
Copy link

StrikerTwo commented Sep 4, 2017 •
edited
Loading

Thank you for your work and your comments.

Are there any drawbacks in running your own DHCP server on the ZT network instead of using the ZT assignment functionality? (Other than it being a SPOF, so it HAS to be online in order for the network to work)

Edit: Sure, go ahead and close this one.

@obrienmd obrienmd mentioned this issue Sep 4, 2017
Closed
@dharrigan
Copy link
Member

Hi,

I haven't observed any drawbacks from running DHCP on Zerotier - in fact, they support this model, although there is some discussion on the merits of DHCP on Zerotier (see zerotier/ZeroTierOne#322). However, my view at the moment is that since you trust OPNsense (so to speak) and are configuring DHCP on OPNsense, then handing out IP addresses to hosts on your Zerotier network is okay :)

-=david=-

@dharrigan
Copy link
Member

@fichtner Can you close this ticket please and raise a new one to provide Zerotier plugin how-to documentation. Thank you.

@fichtner fichtner mentioned this issue Sep 4, 2017
Closed
@fichtner
Copy link
Member

fichtner commented Sep 4, 2017

Sure, done. :)

@fichtner fichtner closed this as completed Sep 4, 2017
@StrikerTwo StrikerTwo mentioned this issue Sep 11, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dharrigan dharrigan

@fichtner fichtner

Labels
feature Adding new functionality
Milestone
No milestone
Development

No branches or pull requests
4 participants
@obrienmd @dharrigan @fichtner @StrikerTwo