Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zerotierone doesn't create dev zt0 on debian 8 with OpenVPN-Server installed. #699

Closed
quetsch opened this issue Mar 23, 2018 · 27 comments
Closed
Labels
Status: Can't Reproduce

Comments

@quetsch
Copy link

quetsch commented Mar 23, 2018

Hi!
I open up a new issue because 2 other threads with similiar issues were closed without a solution:
https://github.com/zerotier/ZeroTierOne/issues/497
https://github.com/zerotier/ZeroTierOne/issues/448
There is definitely an issue with creation of a zt0 interface on Debian 8 with openvpn server installed. The error message in /var/log/syslog is as follows:
zerotier-one[378]: ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory.

I installed zerotier-one freshly on two machines, a local LAN server and on a virtual server, both running debian 8 (uname -r):
LAN-Server: 3.16.0-5-amd64
vServer: 3.16.0
The vServer is configured as openvpn server with both a tun and a tap interface. However, joining my private network works on both machines, no traffic to the vServer however (PORT_ERROR)
sudo zerotier-cli listnetworks
200 listnetworks
200 listnetworks a09acf02333f90c3 Quetsch c2:26:be:0f:c7:29 PORT_ERROR PRIVATE fc93:a55f:c1b6:813c:c5e6:0000:0000:0001/40,10.100.79.1/24

Any help would be appreciated.
BTW: No change when I shut down openvpn and the tun/tap interfaces go down before installation. Seems like an issue in coexisting with openvpn.

PS: a similar issue was reported on centos7 here, thread closed. If I can provide any more information, I am glad to help.

@quetsch quetsch changed the title zerotierone doesn't create dev zt0 on debian 8 when with OpenVPN-Server installed. zerotierone doesn't create dev zt0 on debian 8 with OpenVPN-Server installed. Mar 23, 2018
@janjaapbos
Copy link
Contributor

janjaapbos commented Mar 23, 2018

Does /dev/net/tun exist?
Is it perhaps moved somewhere else in combination with OpenVPN?

@quetsch
Copy link
Author

quetsch commented Mar 23, 2018

Of course it exists on both machines:
sudo ls /dev/net/
tun

sudo ifconfig
`lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:25 errors:0 dropped:0 overruns:0 frame:0
TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:2224 (2.1 KiB) TX bytes:2224 (2.1 KiB)

tap0 Link encap:Ethernet HWaddr ee:05:76:13:a5:8b
inet addr:10.19.80.1 Bcast:10.19.80.255 Mask:255.255.255.0
inet6 addr: fe80::ec05:76ff:fe13:a58b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:6728 (6.5 KiB)

tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 -00
inet addr:10.19.81.1 P-t-P:10.19.81.2 Mask:255.255.255.255
inet6 addr: fe80::9e1f:ed86:c3a2:3c28/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1400 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B) TX bytes:144 (144.0 B)

venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 -00
inet addr:127.0.0.1 P-t-P:127.0.0.1 Bcast:0.0.0.0 Mask:255.255.255. 255
inet6 addr: ::2/128 Scope:Compat
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:172848 errors:0 dropped:0 overruns:0 frame:0
TX packets:171118 errors:0 dropped:7207 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:17150843 (16.3 MiB) TX bytes:18117412 (17.2 MiB)

venet0:0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 -00
inet addr:178.X.X.X P-t-P:178.X.X.X Bcast:178.X.X.255 Ma sk:255.255.255.0
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
`

As stated: OpenVPN works fine with 2 profiles, 1 for tun, 1 for tap. The interface zt0 just doesn't get created by the installer. It must habe something to do with openvpn because a nearly identical other machine with no openvpn installed does not have that issue.

@quetsch
Copy link
Author

quetsch commented Mar 23, 2018

And as mentioned in the other threads, apparmor or selinux is not installed/used as far as I can see:
sudo service apparmor status
● apparmor.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
sudo check-selinux-installation
sudo: check-selinux-installation: command not found
sudo selinux-activate
sudo: selinux-activate: command not found

@adamierymenko
Copy link
Contributor

adamierymenko commented Mar 29, 2018

This really kind of makes no sense. The kernel tap device driver supports thousands of devices and there isn't any restriction about which processes can open them.

Can you try shutting down the service and then running /usr/sbin/zerotier-one manually (via sudo) and telling us if it prints anything?

@quetsch
Copy link
Author

quetsch commented Apr 2, 2018

Hi!
Sorry, was on easter vacation. After stopping the service, the same error appears:
First stopping service:

sudo /etc/init.d/zerotier-one stop
[ ok ] Stopping zerotier-one (via systemctl): zerotier-one.service.
sudo /etc/init.d/zerotier-one status
● zerotier-one.service - ZeroTier One
Loaded: loaded (/lib/systemd/system/zerotier-one.service; enabled)
Active: inactive (dead) since Mon 2018-04-02 03:41:54 UTC; 1min 48s ago
Process: 14383 ExecStart=/usr/sbin/zerotier-one (code=exited, status=0/SUCCESS)
Main PID: 14383 (code=exited, status=0/SUCCESS)

Apr 02 03:41:52 vXXXXX.1blu.de systemd[1]: Started ZeroTier One.
Apr 02 03:41:52 vXXXXX.1blu.de zerotier-one[14383]: ERROR: unable to configur...
Apr 02 03:41:54 vXXXXX.1blu.de systemd[1]: Stopping ZeroTier One...
Apr 02 03:41:54 vXXXXX.1blu.de systemd[1]: Stopped ZeroTier One.
Apr 02 03:43:39 vXXXXX.1blu.de systemd[1]: Stopped ZeroTier One.
Hint: Some lines were ellipsized, use -l to show in full.

(I "Xed" the exact hostname, this forum is public.)

Then the command:
sudo /usr/sbin/zerotier-one
ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory

Can I provide anything else to help? It is definitely an issue with an already installed OpenVPN.

@adamierymenko
Copy link
Contributor

adamierymenko commented Apr 13, 2018

Is SELinux enabled? Maybe there's a rule or permission problem.

@quetsch
Copy link
Author

quetsch commented Apr 17, 2018

Hi!
Sorry for the late reply.
AND: As I already mentioned, no SELinux is NOT enabled, as well no apparmor is in use:
sudo sestatus
sudo: sestatus: command not found
selinuxenabled
-bash: selinuxenabled: command not found
sudo selinuxenabled
command not found
sudo cat /etc/sysconfig/selinux
cat: /etc/sysconfig/selinux: No such file or directory

@adamierymenko
Copy link
Contributor

adamierymenko commented Apr 18, 2018

I really don't know then... we use it alongside other things and I have never seen this issue. Linux has no limit on the number of tun/tap devices.

Can you shut down the ZeroTier service and try running it manually with "sudo /usr/sbin/zerotier-one"? See what it outputs and if there are any meaningful error messages.

@adamierymenko adamierymenko added the Status: Can't Reproduce label Apr 24, 2018
@quetsch
Copy link
Author

quetsch commented Apr 25, 2018

sudo service zerotier-one stop
sudo service zerotier-one status
● zerotier-one.service - ZeroTier One
Loaded: loaded (/lib/systemd/system/zerotier-one.service; enabled)
Active: inactive (dead) since Wed 2018-04-25 12:42:41 UTC; 4s ago
Process: 378 ExecStart=/usr/sbin/zerotier-one (code=exited, status=0/SUCCESS)
Main PID: 378 (code=exited, status=0/SUCCESS)

Apr 23 23:45:24 v65274.1blu.de systemd[1]: Started ZeroTier One.
Apr 23 23:45:24 v65274.1blu.de zerotier-one[378]: ERROR: unable to configure ...
Apr 25 12:42:41 v65274.1blu.de systemd[1]: Stopping ZeroTier One...
Apr 25 12:42:41 v65274.1blu.de systemd[1]: Stopped ZeroTier One.
Hint: Some lines were ellipsized, use -l to show in full.

sudo /usr/sbin/zerotier-one
ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory (Same as written 3 posts above).

I know it's a strange error. I can manually create tun/tap interfaces with the help of ip:
sudo ip tuntap add name tap0 mode tap
sudo ip link show

Now the thread is marked as "cantreproduce". I wonder if you installed OpenVPN prior to zerotier-one and also configured to use a TAP and a TUN device (see my 2nd post)?

Now I am just guessing. Can the error be related to venet0-00 network devices instead of eth0-devices?

However it's a bit frustrating answer the same questions over and over again with the same result. I know it ist an Open Source project and the support here is voluntarily, but I slowly get the impression that after asking the top 5 standard issues you are out of ideas and the threads' gonna die somehow.

So, can it have something to do with venet-0 devices on a virtual server, maybe in the routine on how tun/tap devices are created? It is possible via the "ip" command or with "openvpn -mktun".

Is there a way to increase verbosity level for logs???

1 similar comment
@quetsch
Copy link
Author

quetsch commented Apr 26, 2018

sudo service zerotier-one stop
sudo service zerotier-one status
● zerotier-one.service - ZeroTier One
Loaded: loaded (/lib/systemd/system/zerotier-one.service; enabled)
Active: inactive (dead) since Wed 2018-04-25 12:42:41 UTC; 4s ago
Process: 378 ExecStart=/usr/sbin/zerotier-one (code=exited, status=0/SUCCESS)
Main PID: 378 (code=exited, status=0/SUCCESS)

Apr 23 23:45:24 v65274.1blu.de systemd[1]: Started ZeroTier One.
Apr 23 23:45:24 v65274.1blu.de zerotier-one[378]: ERROR: unable to configure ...
Apr 25 12:42:41 v65274.1blu.de systemd[1]: Stopping ZeroTier One...
Apr 25 12:42:41 v65274.1blu.de systemd[1]: Stopped ZeroTier One.
Hint: Some lines were ellipsized, use -l to show in full.

sudo /usr/sbin/zerotier-one
ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory (Same as written 3 posts above).

I know it's a strange error. I can manually create tun/tap interfaces with the help of ip:
sudo ip tuntap add name tap0 mode tap
sudo ip link show

Now the thread is marked as "cantreproduce". I wonder if you installed OpenVPN prior to zerotier-one and also configured to use a TAP and a TUN device (see my 2nd post)?

Now I am just guessing. Can the error be related to venet0-00 network devices instead of eth0-devices?

However it's a bit frustrating answer the same questions over and over again with the same result. I know it ist an Open Source project and the support here is voluntarily, but I slowly get the impression that after asking the top 5 standard issues you are out of ideas and the threads' gonna die somehow.

So, can it have something to do with venet-0 devices on a virtual server, maybe in the routine on how tun/tap devices are created? It is possible via the "ip" command or with "openvpn -mktun".

Is there a way to increase verbosity level for logs???

@quetsch
Copy link
Author

quetsch commented May 8, 2018

Issue still persisting in 1.2.8

@maxnowack
Copy link

maxnowack commented Jun 4, 2018

Same issue here …

@s-frostick
Copy link

s-frostick commented Jun 5, 2018

So i'm not sure if this will help but i was experiencing the same problem, i did an strace of the zerotier process.

close(9)                                = 0
brk(0xc24000)                           = 0xc24000
open("/dev/net/tun", O_RDWR)            = -1 EACCES (Permission denied) 
open("/dev/tun", O_RDWR)                = -1 ENOENT (No such file or directory)
brk(0xc2d000)                           = 0xc2d000
brk(0xc36000)                           = 0xc36000 
writev(2, [{"ERROR: unable to configure virtu"..., 49}, {"could not open TUN/TAP device: N"..., 56}],   2ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory) = 105

So i checked the permission of /dev/net/tun

ls -la /dev/net/tun 
crw-rw---- 1 root 413 10, 200 Jun  5 01:08 /dev/net/tun

Now setting the permission to 0666 fixed the "No such file or directory" error for me.

https://www.kernel.org/doc/Documentation/networking/tuntap.txt

Set permissions:
e.g. chmod 0666 /dev/net/tun
There's no harm in allowing the device to be accessible by non-root users,
since CAP_NET_ADMIN is required for creating network devices or for
connecting to network devices which aren't owned by the user in question.
If you want to create persistent devices and give ownership of them to
unprivileged users, then you need the /dev/net/tun device to be usable by
those users.

@maxnowack
Copy link

maxnowack commented Jun 5, 2018

Thanks @s-frostick! Setting the permissions to 0666 fixed the issue for me as well 😊

@laduke
Copy link
Contributor

laduke commented Jun 5, 2018

why is the user "1" ?

@s-frostick
Copy link

s-frostick commented Jun 5, 2018

@laduke the user is root the number you are referencing is the number of hard links to the file.

https://www.debian.org/doc/manuals/debian-reference/ch01.en.html#_links

@laduke
Copy link
Contributor

laduke commented Jun 5, 2018

Oops, off by one. (group is 413)

@quetsch
Copy link
Author

quetsch commented Jun 7, 2018

Hi!
I checked the above on my both machines, one where zerotier is working, one where it is not (both Debian Jessie). The permissions seem identical:

ZT working:
ls -la /dev/net/tun
crw-rw-rw- 1 root root 10, 200 Jun 7 13:56 /dev/net/tun
sudo ls -la /dev/net/tun
crw-rw-rw- 1 root root 10, 200 Jun 7 13:56 /dev/net/tun

ZT not working:
ls -la /dev/net/tun
ls: cannot access /dev/net/tun: Permission denied
sudo ls -la /dev/net/tun
crw-rw-rw- 1 root root 10, 200 May 31 00:16 /dev/net/tun

The file permissions are obivously the same. However, I noticed on the machine where the issue persists, I can't "ls -la /dev/net/tun" as a normal user, on the other machine I can.
Well, I think that has nothing to do with my issue.

Still no zt0 interface is created:
sudo zerotier-cli listnetworks
200 listnetworks
200 listnetworks a09acf02333f90c3 Quetsch c2:26:be:0f:c7:29 PORT_ERROR PRIVATE fc93:a55f:c1b6:813c:c5e6:0000:0000:0001/40,10.100.79.1/24

strace behaves similar at the system with the issue:
12433 close(9) = 0
12433 chmod("/var/lib/zerotier-one/networks.d/a09acf02333f90c3.conf", 0600) = 0
12433 brk(0xed8000) = 0xed8000
12433 open("/dev/net/tun", O_RDWR) = -1 EACCES (Permission denied)
12433 open("/dev/tun", O_RDWR) = -1 ENOENT (No such file or directory)
12433 brk(0xee1000) = 0xee1000
12433 brk(0xeea000) = 0xeea000
12433 writev(2, [{"ERROR: unable to configure virtu"..., 49}, {"could not open TUN/TAP device: N"..., 56}], 2) = 105

Well, it seems like a permission issue, but chmod 666 or even chmod 777 on /dev/net/tun doesn't change it...

@factormystic
Copy link

factormystic commented Jun 9, 2018

FYI I found this issue via google after following the directions for getting started with docker in the knowledgebase article here. chmod 0666 /dev/net/tun did work for me.

@joseph-henry
Copy link
Contributor

joseph-henry commented Oct 3, 2018

Is anyone still experiencing this issues as of 1.2.12? It looks like a working solution has been found for at least a couple of those reporting the issue. I'm going to close this ticket for now but feel free to request that we re-open it.

@NeedsCoffee
Copy link

NeedsCoffee commented Nov 27, 2018

I just encountered this on v1.2.12
The chmod fix helped me and I had installed into a Scaleway VM that was running Debian 9
Permissions on /dev/net/tun were previously: crw-------
Afterwards permissions were: crw-rw-rw-

@quetsch
Copy link
Author

quetsch commented Jan 23, 2019

Hello!

This thread is closed. After further investigation I tried possible solutions to a bit different issues with ZT in linux.
I finally managed to get a working zt0 interface.
This thread helped by the the "fix":
#809

Apparently the issue was a "rights issue"; adding the -U option as described down below fixed it.

`/lib/systemd/system/zerotier-one.service:

[Unit]
Description=ZeroTier One
After=network.target

[Service]
ExecStart=/usr/sbin/zerotier-one -U
Restart=always
KillMode=process

[Install]
`WantedBy=multi-user.target``

@vamposdecampos
Copy link

vamposdecampos commented Feb 22, 2019

(on an openvz VPS) I've also had to chmod 777 /dev/net as well.

@NeverBehave
Copy link

NeverBehave commented Mar 30, 2020

Just a quick note if you google and find this issue: Don't forget to try rebooting

I have all settings correct (permission, etc.) but still encounter this problem, but it works after rebooting the machine.

@diwu1989
Copy link

diwu1989 commented Jun 26, 2020

Please don't 777 the /dev/net these are safer alternatives:

chmod 755 /dev/net
chmod 666 /dev/net/tun

@benhbell
Copy link

benhbell commented Jul 12, 2020

Both of these also helped me with a node on OpenVz

@rumym
Copy link

rumym commented Dec 21, 2021

chmod 666 /dev/net/tun

FWIW only 777 worked for me. I am logging in as root.
chmod 777 /dev/net
chmod 777 /dev/net/tun
And it starts working!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Can't Reproduce
Projects
None yet
Development

No branches or pull requests