Unable to configure local resolver address and port using NB_DNS_RESOLVER_ADDRESS environment variable

[gene1wood]

Describe the problem

I'd like to be able to run the pihole DNS server and Netbird on the same computer. I'm figuring that the best way to do this is to

• Configure Netbird local resolver to listen on 5053
• Configure Pihole DNS server to listen on 53
• Configure Netbird in the web UI to use a custom nameserver of the local Pihole instance

This way netbird addresses resolve correctly, and all other addresses get sent onto Pihole.

My Netbird installation works and can set up the local netbird resolver on port 53 of the CGNAT IP.

My Ubuntu 22.04 server uses systemd. Netbird 0.28.9 launches via systemd and consumes environment variables out of /etc/sysconfig/netbird.

I've created an /etc/sysconfig/netbird file with the contents of

NB_DNS_RESOLVER_ADDRESS=127.0.0.1:5053
NB_LOG_LEVEL=debug

I've done this to try to get netbird's local resolver to listen not on port 53 but instead 5053. I'm figuring this would work as the --dns-resolver-address command line flag for the up command says that it

Sets a custom address for NetBird's local DNS resolver. If set, the agent won't attempt to discover the best ip and port to listen on. An empty string "" clears the previous configuration. E.g. --dns-resolver-address 127.0.0.1:5053 or --dns-resolver-address ""

I've confirmed that the /etc/sysconfig/netbird file is working because when I add the NB_LOG_LEVEL=debug line, the /var/log/netbird/client.log file begins logging debug information.

Netbird however continues to follow it's native logic for determining the local resolver address and port.

I know it's not following the NB_DNS_RESOLVER_ADDRESS environment variable because the logs show this

2024-09-04T00:52:54Z DEBG client/internal/dns/service_listener.go:69: starting dns on 100.105.159.203:53
2024-09-04T00:52:54Z DEBG client/internal/dns/file_unix.go:97: creating managed file /etc/resolv.conf
2024-09-04T00:52:54Z INFO client/internal/dns/file_unix.go:107: created a NetBird managed /etc/resolv.conf file with the DNS settings. Added 2 search domains. Search list: [netbird.cloud .]

And when I run lsof -i :53 it shows that netbird continues to listen on port 53

What am I overlooking that's preventing me from being able to override the DNS resolver address using the NB_DNS_RESOLVER_ADDRESS environment variable?

To Reproduce

Steps to reproduce the behavior:

1. Install Netbird on an Ubuntu 22.04 system or any system that uses systemd
2. Create an /etc/sysconfig/netbird file as described above to set the local resolver address and port to 127.0.0.1:5053
3. Restart netbird by running systemctl restart netbird.service

Expected behavior

The logs should show that netbird's local resolver is listening on 127.0.0.1:5053 and when checking what ports the process has open, it should show 5053

Are you using NetBird Cloud?

Yes

NetBird version

0.28.9

NetBird status -dA output:

This is included in the output below from netbird debug for 1m -AS

Do you face any (non-mobile) client issues?

Please provide the file created by netbird debug for 1m -AS.

netbird.debug.3677564285.zip

[gene1wood]

A tangential question : While I'm unable to get Netbird's local resolver to listen on a different port, is there a way to just disable the local resolver feature? I'm not using it and just disabling it would be fine.

I've looked through the CLI help options but have been unable to find something to tell Netbird to just not spawn a DNS local resolver.

It looks like my tangential question may relate to #2495

[gene1wood]

@lixmal Is there something obvious I'm doing wrong with the NB_DNS_RESOLVER_ADDRESS environment variable?

[gokuldas]

@gene1wood I don't know if this issue is still relevant to you, but I think I know the problem and the solution.

According to the docs, the environment variables are just replacements for CLI flags. However, one thing is clear. The netbird client command fails if you give it a wrong flag. On the other hand if you give netbird a wrong environment variable, it just neglects it and proceeds with its default behavior without warning or alerting you in any manner.

Now have a look at the netbird systemd service file. The following lines are of special interest:

[Service]
EnvironmentFile=-/etc/default/netbird
ExecStart=/usr/bin/netbird service run --log-file /var/log/netbird/client-%i.log --config /etc/netbird/%i.json --daemon-addr unix:///var/run/netbird/%i.sock $FLAGS

The environment file was /etc/sysconfig/netbird on my system instead of /etc/default/netbird here. Either way, the file is read (if present, as indicated by the hyphen prefix) and the environment is set for the command given by ExecStart. Therefore, those environment variables apply to the command netbird service run.

Trying netbird help service run gives me:

runs Netbird as service

Usage:
  netbird service run [flags]

Flags:
  -h, --help   help for run

Global Flags:
      --admin-url string        Admin Panel URL [http|https]://[host]:[port] (default "https://app.netbird.io:443")
  -A, --anonymize               anonymize IP addresses and non-netbird.io domains in logs and status output
  -c, --config string           Netbird config file location (default "/etc/netbird/config.json")
      --daemon-addr string      Daemon service address to serve CLI requests [unix|tcp]://[path|host:port] (default "unix:///var/run/netbird.sock")
  -n, --hostname string         Sets a custom hostname for the device
      --log-file string         sets Netbird log path. If console is specified the log will be output to stdout. If syslog is specified the log will be sent to syslog daemon. (default "/var/log/netbird/client.log")
  -l, --log-level string        sets Netbird log level (default "info")
  -m, --management-url string   Management Service URL [http|https]://[host]:[port] (default "https://api.netbird.io:443")
      --preshared-key string    Sets Wireguard PreSharedKey property. If set, then only peers that have the same key can communicate.
  -s, --service string          Netbird system service name (default "netbird")
  -k, --setup-key string        Setup key obtained from the Management Service Dashboard (used to register peer)
      --setup-key-file string   The path to a setup key obtained from the Management Service Dashboard (used to register peer) This is ignored if the setup-key flag is provided.

There is no --dns-resolver-address flag for this command. In fact, there is no option at all in this command to set the DNS resolver address. But the --config /etc/netbird/config.json is a possible candidate for what you need. Infact, this was found in my /etc/netbird/config.json file:

"CustomDNSAddress": "",

But I couldn't find any documentation on what it is or how to set it. Anyway, as you already noted, the --dns-resolver-address flag is part of the netbird up command. This can be confirmed by running netbird help up. So, what happens if you do this?:

netbird up --dns-resolver-address=127.0.0.1:5053

The netbird client service does in fact start listening on that specific interface (loopback) and port. Further, the line from /etc/netbird/config.json changes to this:

"CustomDNSAddress": "127.0.0.1:5053",

So this is where the DNS resolver setting is stored. I also confirmed that this value is persisted. The resolver starts consistently on the same interface until you use that flag again. You really don't need the environment variable at all. All you need to do is to stop netbird once and start it again, specifying your preferred interface.

There are some caveats though. The first is that you have to specify both the IP and the port. Neither can be omitted.

The other is that when netbird populates the /etc/resolv.conf file, it adds the line nameserver 127.0.0.1 at the top. Note that the port is not mentioned. I checked and confirmed that the other applications can't find the resolver on a non-standard port. It's better to stick with the standard port 53, whatever the interface may be. I wanted the port 53 on the wireguard interface to be free. This worked perfectly when I shifted the resolver to 127.0.0.1:53.

Finally,

While I'm unable to get Netbird's local resolver to listen on a different port, is there a way to just disable the local resolver feature? I'm not using it and just disabling it would be fine.

This feature is not present in the CLI as far as I can tell. It's in the DNS page of the server administration console. There is a rather obscure tab that leads to a box where you can add the client groups on which you don't want the resolver to start. This is explained in a comment from the issue you linked above. Interestingly, I had the opposite problem as you did. The resolver wouldn't start on certain clients. After failing to identify the reason from hours of online research, I accidentally stumbled on this page.

Hope this helps!

[nazarewk]

Hello @gene1wood,

We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.

Could you please confirm if the issue is still there?

We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.

Thanks for your contribution to improving the project!

[mlsmaycon]

closing issue due to no recent feedback. Feel free to open a new one if the issue persist or reopen if this was a feature request.

[mr-karan]

I encountered the same issue on Arch Linux with Netbird installed via AUR. The problem was that my netbird@.service systemd unit file was missing the --config /etc/netbird/%i.json flag in the ExecStart line.

Without the config flag, Netbird wasn't reading or persisting settings to /etc/netbird/config.json, so the --dns-resolver-address flag had no effect.

Solution:

sudo netbird service install --log-level info --config /etc/netbird/config.json

The setting now persists in /etc/netbird/config.json as:

"CustomDNSAddress": "127.0.0.1:5053"

And Netbird's DNS resolver correctly listens on the specified port instead of the default port 53.

Hope this helps anyone else stuck with same issue

[OTheNonE]

I am having an issue with this. I am using docker compose and running the commands:
command: ["up", "--dns-resolver-address", "127.0.0.1:5053"]
When running ss -tulnp, i see that the netbird instance 100.XXX.YYY.ZZZ is taking the port 53, eventhough i run the above command.

[timowevel1]

The issue is still open.

[lixmal]

The container needs NB_DNS_RESOLVER_ADDRESS, non-containers need the up command. This issue is convoluted, can you please state what the issue is for each one of you?

[gokuldas]

@lixmal I don't have anymore issues here, but I noticed something in the Dockerfile and its entry point.

The container needs NB_DNS_RESOLVER_ADDRESS, non-containers need the up command. This issue is convoluted ...

I can see why. Here is the relevant line from the Dockerfile:

ENTRYPOINT [ "/usr/local/bin/netbird-entrypoint.sh" ]

Note that the CMD directive is not defined in it. Meanwhile, here are the relevant lines from the netbird-entrypoint script:

main() {
  trap 'on_exit' SIGTERM SIGINT EXIT
  "${NETBIRD_BIN}" service run &
  service_pids+=("$!")
  info "registered new service process 'netbird service run', currently running: ${service_pids[@]@Q}"

  locate_log_file "${NB_LOG_FILE}"
  wait_for_daemon_startup "${NB_ENTRYPOINT_SERVICE_TIMEOUT}"
  login_if_needed "${NB_ENTRYPOINT_LOGIN_TIMEOUT}"

  wait "${service_pids[@]}"
}

main "$@"

The last line invokes the main function with the script's entire argument. However, those arguments are not used anywhere inside the main function. This means that @OTheNonE 's compose line command: ["up", "--dns-resolver-address", "127.0.0.1:5053"] doesn't go anywhere and doesn't have any effect whatsoever.

I don't know what the original intentions behind the script were - if it was to use those CLI args, but forgot to do so, or if it was to not use the args, but passed them during the function call by mistake. Either way, it's inconsistent and very confusing. Even setting the NB_DNS_RESOLVER_ADDRESS environment variable in docker-cli/compose-file probably won't have any effect, because the netbird up command will never be called. It will always be the netbird service run command that's executed.

I can suggest a solution if you're looking for one. This solution will have the following properties:

• Specifying the command in docker-cli/compose-file will work as expected, including the correct sub-command and the CLI flags that the admin specifies.
• Setting the NB_DNS_RESOLVER_ADDRESS will work if the admin is running the netbird up command
• The --dns-resolver-address=<interface:port> flag will also work for the netbird up command

So here is the modified code:

 client/Dockerfile            | 1 +
 client/netbird-entrypoint.sh | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/client/Dockerfile b/client/Dockerfile
index b2f62740..c0e244bc 100644
--- a/client/Dockerfile
+++ b/client/Dockerfile
@@ -21,6 +21,7 @@ ENV \
     NB_ENTRYPOINT_LOGIN_TIMEOUT="5"
 
 ENTRYPOINT [ "/usr/local/bin/netbird-entrypoint.sh" ]
+CMD ["service", "run"]
 
 ARG NETBIRD_BINARY=netbird
 COPY client/netbird-entrypoint.sh /usr/local/bin/netbird-entrypoint.sh
diff --git a/client/netbird-entrypoint.sh b/client/netbird-entrypoint.sh
index 7c9fa021..1675447b 100755
--- a/client/netbird-entrypoint.sh
+++ b/client/netbird-entrypoint.sh
@@ -91,7 +91,7 @@ login_if_needed() {
 
 main() {
   trap 'on_exit' SIGTERM SIGINT EXIT
-  "${NETBIRD_BIN}" service run &
+  "${NETBIRD_BIN}" $@ &
   service_pids+=("$!")
   info "registered new service process 'netbird service run', currently running: ${service_pids[@]@Q}"
 
@@ -102,4 +102,4 @@ main() {
   wait "${service_pids[@]}"
 }
 
-main "$@"
+main $@

This way, the admin can replace the entire subcommand (eg: service run with up --dns-resolver-address 127.0.0.1:5053 as in @OTheNonE 's solution) using the command spec in the docker-cli/compose-file. This is probably the solution that's will spring the least surprise on the admin.

Note that should you choose to implement this, you'll probably need to modify the rest of the main function in the bash script. The admin may not be trying to start the service at all. So adding the PID to the PID list wont make any sense.

[nazarewk]

@gokuldas the main problem is there are 2 commands to run taking a different set of input, which one should take the arguments passed down? netbird up actually does most of the meaningful configuration in practice, but there are also some configuration tweaks that can only be passed to a daemon (service run).

As a bit of background, every single CLI flag can be passed down as an Environment Variable, so there should never be any need to customize the argument-less netbird service run nor netbird up.