Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zeronsd does not listen on port 53 on MacOS #199

Open
peterhoneder opened this issue Oct 18, 2022 · 18 comments
Open

zeronsd does not listen on port 53 on MacOS #199

peterhoneder opened this issue Oct 18, 2022 · 18 comments

Comments

@peterhoneder
Copy link

Hi!

I can see typical startup output:

/usr/local/bin/zeronsd start -t /private/var/lib/zerotier-one/token -d zerotier <network-id>
Oct 18 09:23:48.614  INFO zeronsd::init: Welcome to ZeroNS!
Oct 18 09:23:49.583  INFO zeronsd::init: Your IP for this network: 172.25.221.93
Oct 18 09:23:49.961  INFO zeronsd::authority: Adding new record <blanked>
Oct 18 09:23:49.961  INFO zeronsd::authority: Adding new record <blanked>
Oct 18 09:23:49.961  INFO zeronsd::authority: Adding/Replacing record <blanked>
...

The records that are added looked correct to me.

But then I tried to resolve using the IP (172.25.221.93) and nothing happened. Looking at the list of open ports (sudo lsof -i -P -n | grep LISTEN|grep 53) shows nothing listening on port 53.

I then tried to launch with debug and trace log levels, but did not see any other information that would point me to the issue. Do you have any other ideas on how to debug this?

OS: MacOS 12.4
installed: via brew (github repo instructions)
@laduke
Copy link
Contributor

laduke commented Oct 18, 2022

Hello,
Homebrew worked for me on macOS 12.6

zeronsd 63175 root 14u IPv4 0x7c52d7942caa8867 0t0 TCP 10.147.19.23:53 (LISTEN)

Are you on Arm? I'm on Intel.

@erikh
Copy link
Contributor

erikh commented Mar 2, 2023

I will try to comb over the issues in detail next week. Been away from the computer for a while.

A knee-jerk on this ticket however suggests that there is a firewall involved. I would look there first.

@erikh
Copy link
Contributor

erikh commented Mar 6, 2023

So, reviewing this again, can you run this command for me and paste the result? It should produce no secrets:

sudo lsof -p `pidof zeronsd`

It would help me diagnose your issue.

@laduke laduke mentioned this issue Mar 16, 2023
Open
@laduke
Copy link
Contributor

laduke commented Mar 28, 2023

One way to reproduce this is to not use be root/sudo when your start zeronsd. For whatever reason macos doesn't complain or crash the program, but the program doesn't work. sudo lsof -P -iUDP | grep zero does not contain any zeronsd unless you start it with sudo.

@glimberg
Copy link
Contributor

It should require root to run. Port 53 is < 1024, and thus a "privileged" port that requires root

@laduke
Copy link
Contributor

laduke commented Mar 28, 2023 •
edited
Loading

apple got rid of privileged ports at some point, but -get this- it's buggy.

https://news.ycombinator.com/item?id=18302380
https://developer.apple.com/forums/thread/674179
(there aren't any actual docs on the topic)

But yeah, zeronsd should exit when it's not root, or not able to use port 53 I guess. I tried on linux and the same thing happens. It starts but doesn't work.

@jordanarpad
Copy link

I have a Raspberry Pi 1 and Pi Zero for 2 different zerotier networks running exactly the same zeronsd binary (built over 12 hours with 512M swap) and listens port 53 on one but not on the other. Both are running pi-hole besides zeronsd.
Here is the requested output for one that does not listen:

colos@jozsi:~ $ sudo lsof -p pidof zeronsd
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND  PID USER   FD      TYPE     DEVICE SIZE/OFF   NODE NAME
zeronsd 2343 root  cwd       DIR      179,2     4096      2 /
zeronsd 2343 root  rtd       DIR      179,2     4096      2 /
zeronsd 2343 root  txt       REG      179,2 15558984 391959 /home/colos/zeronsd
zeronsd 2343 root  mem       REG      179,2    17972 137627 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so
zeronsd 2343 root  mem       REG      179,2    75568 137744 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so
zeronsd 2343 root  mem       REG      179,2    13760 137632 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2
zeronsd 2343 root  mem       REG      179,2    42628 137628 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so
zeronsd 2343 root  mem       REG      179,2  1319784 137204 /usr/lib/arm-linux-gnueabihf/libc-2.31.so
zeronsd 2343 root  mem       REG      179,2    13864 137274 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so
zeronsd 2343 root  mem       REG      179,2   386572 137539 /usr/lib/arm-linux-gnueabihf/libm-2.31.so
zeronsd 2343 root  mem       REG      179,2   137364 137721 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so
zeronsd 2343 root  mem       REG      179,2   116324 137362 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
zeronsd 2343 root  mem       REG      179,2  2150824 137243 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1
zeronsd 2343 root  mem       REG      179,2   459032 137813 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1
zeronsd 2343 root  mem       REG      179,2     9512 137135 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
zeronsd 2343 root  mem       REG      179,2   146888 136992 /usr/lib/arm-linux-gnueabihf/ld-2.31.so
zeronsd 2343 root    0r      CHR        1,3      0t0      4 /dev/null
zeronsd 2343 root    1u     unix 0xcf31dc27      0t0  43155 type=STREAM
zeronsd 2343 root    2u     unix 0xcf31dc27      0t0  43155 type=STREAM
zeronsd 2343 root    3u  a_inode       0,13        0   6967 [eventpoll]
zeronsd 2343 root    4u  a_inode       0,13        0   6967 [eventfd]
zeronsd 2343 root    5u  a_inode       0,13        0   6967 [eventpoll]
zeronsd 2343 root    6u     unix 0x2cdb490e      0t0  43171 type=STREAM
zeronsd 2343 root    7u     unix 0x2d73d79d      0t0  43172 type=STREAM
zeronsd 2343 root    8u     unix 0x2cdb490e      0t0  43171 type=STREAM
zeronsd 2343 root    9u     IPv4      43193      0t0    TCP pi.hole:59370->151.101.1.91:https (ESTABLISHED)
colos@jozsi:~ $

And here is the requested output for the one that does listen:

jordana@pi1b:~ $ sudo lsof -p `pidof zeronsd`
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND   PID USER   FD      TYPE     DEVICE SIZE/OFF   NODE NAME
zeronsd 15574 root  cwd       DIR      179,2     4096      2 /
zeronsd 15574 root  rtd       DIR      179,2     4096      2 /
zeronsd 15574 root  txt       REG      179,2 15558984   3184 /home/jordana/zeronsd
zeronsd 15574 root  mem       REG      179,2    17972   8494 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so
zeronsd 15574 root  mem       REG      179,2    75568   8519 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so
zeronsd 15574 root  mem       REG      179,2    13760   7546 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2
zeronsd 15574 root  mem       REG      179,2    42628   8495 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so
zeronsd 15574 root  mem       REG      179,2  1319784   8471 /usr/lib/arm-linux-gnueabihf/libc-2.31.so
zeronsd 15574 root  mem       REG      179,2    13864   8477 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so
zeronsd 15574 root  mem       REG      179,2   386572   8480 /usr/lib/arm-linux-gnueabihf/libm-2.31.so
zeronsd 15574 root  mem       REG      179,2   137364   8512 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so
zeronsd 15574 root  mem       REG      179,2   116324   7299 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
zeronsd 15574 root  mem       REG      179,2  2150824  11580 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1
zeronsd 15574 root  mem       REG      179,2   459032  11584 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1
zeronsd 15574 root  mem       REG      179,2     9512   7093 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
zeronsd 15574 root  mem       REG      179,2   146888   7656 /usr/lib/arm-linux-gnueabihf/ld-2.31.so
zeronsd 15574 root    0r      CHR        1,3      0t0      4 /dev/null
zeronsd 15574 root    1u     unix 0x61a3094a      0t0 283766 type=STREAM
zeronsd 15574 root    2u     unix 0x61a3094a      0t0 283766 type=STREAM
zeronsd 15574 root    3u  a_inode       0,13        0   6967 [eventpoll]
zeronsd 15574 root    4u  a_inode       0,13        0   6967 [eventfd]
zeronsd 15574 root    5u  a_inode       0,13        0   6967 [eventpoll]
zeronsd 15574 root    6u     unix 0xb7630605      0t0 283774 type=STREAM
zeronsd 15574 root    7u     unix 0x0adc4997      0t0 283775 type=STREAM
zeronsd 15574 root    8u     unix 0xb7630605      0t0 283774 type=STREAM
zeronsd 15574 root    9u     IPv4     283794      0t0    TCP pi.hole:41042->151.101.1.91:https (ESTABLISHED)
zeronsd 15574 root   10u     IPv4     283801      0t0    TCP 192.168.193.1:domain (LISTEN)
zeronsd 15574 root   11u     IPv4     283802      0t0    UDP 192.168.193.1:domain 
jordana@pi1b:~ $

There is slightly more logs in journal for the one that listens and here is the diff:

May 08 11:24:02 pi1b sudo[15582]:  jordana : TTY=pts/0 ; PWD=/home/jordana ; USER=root ; COMMAND=/usr/bin/lsof -p 15574
May 08 11:24:02 pi1b sudo[15582]: pam_unix(sudo:session): session opened for user root(uid=0) by jordana(uid=1000)
May 08 11:24:03 pi1b sudo[15582]: pam_unix(sudo:session): session closed for user root

@erikh
Copy link
Contributor

erikh commented May 8, 2023 via email

@jordanarpad
Copy link

jordanarpad commented May 8, 2023 •
edited
Loading

Not exactly sure what you mean but I realised the diff is more, again in favour of the one that listens as the following lines are also there and they are not there for the one that does not listen.
This other diff snippet with a few lines before and after:

May 09 00:14:37 pi1b zeronsd[20722]: May 09 00:14:37.333  INFO zeronsd::init: Welcome to ZeroNS!
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.694  INFO zeronsd::init: Your IP for this network: 192.168.x.x
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.709  INFO trust_dns_server::store::forwarder::authority: loading forwarder config: .
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.716  INFO trust_dns_server::store::forwarder::authority: forward resolver configured: .:
May 09 00:14:39 pi1b zeronsd[20722]: May 09 00:14:39.003  INFO zeronsd::authority: Adding new record zt-xxxxxx.zeronsd.: (192.168.x.x)

It can be seen from this snippet that the initial post of this issue also misses these lines that refer to the forwarder.
A bit difficult to share more publicly from journalctl without exposing my network.

@erikh
Copy link
Contributor

erikh commented May 9, 2023 via email

@jordanarpad
Copy link

there are no zeronsd logs with bind or listen in neither journal unfortunately even after setting ZERONSD_LOG and RUST_LOG to debug.
do I need to set there environmental variables during the build process for a debug binary?

@erikh
Copy link
Contributor

erikh commented May 9, 2023 via email

@jordanarpad
Copy link

jordanarpad commented May 9, 2023 •
edited
Loading

I figured out the difference. On the one where zeronsd listens I had configured pi-hole to bind only on eth0. On the other one zeronsd could not bind because pi-hole bind was not limited to an interface so bind them all.

just for the record I used these commands to build zeronsd on Raspberry Pi 1 running latest Raspberry Pi OS:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt install librust-openssl-dev
cargo install --git https://github.com/zerotier/zeronsd --branch main --jobs 1

and I had to increase swap memory to 512M for which I used this article as guidance: increase swap memory
the build process took a painful 12 hours but I can happily use the binary accross all my Raspberry Pi Zero W and Raspberry Pi 1

@erikh
Copy link
Contributor

erikh commented May 10, 2023 via email

@erikh
Copy link
Contributor

erikh commented May 10, 2023 via email

@laduke
Copy link
Contributor

laduke commented May 10, 2023

ah sorry guys. I think #218 would have helped (but I haven't double checked). I haven't been able to focus on this project recently.

side question: you can't cross compile from a big computer?

@erikh
Copy link
Contributor

erikh commented May 10, 2023 via email

@jordanarpad
Copy link

jordanarpad commented May 10, 2023 •
edited
Loading

I am conscious that I hijacked this issue with something ended up unrelated so this will be my last comment under this issue. @laduke great to see possible fixes are on the way - reading it definitely sounds related even with the original issue in this post. I will test if it fixes my issue once merged.
@erikh I am very grateful for your swift and continuous support in this issue and your questions did guide me through to find my solution. Will gladly take your offer and will drop you an email about cross-compilation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@laduke @erikh @glimberg @jordanarpad @peterhoneder