Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG Maybe: After start netbird, all Docker containers can't connect network #2015

Closed
johnson7788 opened this issue May 20, 2024 · 14 comments · Fixed by #2038
Closed

BUG Maybe: After start netbird, all Docker containers can't connect network #2015

johnson7788 opened this issue May 20, 2024 · 14 comments · Fixed by #2038

Comments

@johnson7788
Copy link

johnson7788 commented May 20, 2024

Describe the problem
When I start netbird, No matter from netbird command or netbird docker, netbird works very well, But it leads my other docker container can't connect network. i suspect netbird conflict with docker deamon network.

Test on Linux: Ubuntu16.04 and CentOS7.

To Reproduce

Steps to reproduce the behavior:

  1. install docker services
  2. start netbird(docker or command)
  3. Test other docker, like below
  4. sudo docker run --rm --net=host -it busybox ping 123.150.76.218
  5. sudo docker run --rm busybox ping so.com
  6. both are failed
  7. stop netbird, try again step4 and step5, success network now.

Expected behavior

A clear and concise description of what you expected to happen.

Are you using NetBird Cloud?

Please specify whether you use NetBird Cloud or self-host NetBird's control plane.

NetBird version

netbird version
0.23.3 and 0.27.7

NetBird status -d output:

If applicable, add the `netbird status -d' command output.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

@enyachoke
Copy link

enyachoke commented May 21, 2024

@johnson7788 I also experienced this issue on CentOS 7, where the docker containers on the host could not connect to the Internet after installing Netbird.

i suspect netbird conflict with docker deamon network.

@johnson7788 have you found a way to run Netbird on a different IP range

@johnson7788
Copy link
Author

johnson7788 commented May 22, 2024

@enyachoke My docker network is 172.17.0.0/16, and netbird is 100.124.0.0/16, I think they are already in different IP segment.

@johnson7788
Copy link
Author

I check docker network, iptables, and routes

Start Bird, the route changed
default via 192.168.50.1 dev eno1 proto static metric 100
100.124.0.0/16 dev wt0 proto kernel scope link src 100.124.182.112
169.254.0.0/16 dev eno1 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.179 metric 100

IPtables:
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere localhost tcp dpt:15672
ACCEPT tcp -- anywhere localhost tcp dpt:amqp

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere

Stop Bird, The route
default via 192.168.50.1 dev eno1 proto static metric 100
169.254.0.0/16 dev eno1 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.179 metric 100

Iptables
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:15672
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:amqp

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere

@mlsmaycon
Copy link
Collaborator

mlsmaycon commented May 22, 2024

@enyachoke @johnson7788, can you confirm whether you have any DNS or routes enabled in NetBird's dashboard?

Also can you try forcing the use of iptables instead of nftables with the following commands:

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird

@thorleifjacobsen
Copy link
Contributor

@enyachoke @johnson7788 the code @mlsmaycon wrote seems to be not working for me, I changed it to this to make it use iptables instead:

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird

Ref issue #2030 - seems to be a common issue on C7 or machines using nftables? I do not see this table in ubuntu on nft or iptables.

@mlsmaycon
Copy link
Collaborator

That's correct, I've updated the command.

The problem still happening with this flag enabled?

@thorleifjacobsen
Copy link
Contributor

thorleifjacobsen commented May 23, 2024

My issue with NAT rules being ignored by IPTables, specifically with SNAT not working, was resolved with the following flag:

-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233

This IPTables rule was ignored, possibly due to the presence of this chain in nftables. After removing the chain, SNAT started working again:

chain netbird-rt-nat {
    type nat hook postrouting priority 99; policy accept;
}

I can't speak for others, but this fixed the issue for me. My next question is whether this is a configuration bug in Netbird. It seems to be incorrectly configured since it overrides other NAT rules.

My understanding is that this setup allows postrouting without passing through more specialized rules. Should the specific IPs involved be defined more clearly? What might be the underlying issue? Can this be fixed in a patch for Netbird, or do we need to enforce IPTables every time?

Additionally, why does this issue not occur on Ubuntu?

@thorleifjacobsen
Copy link
Contributor

thorleifjacobsen commented May 23, 2024

Another addition, in Ubuntu this is the same chain:

chain netbird-rt-nat {
        type nat hook postrouting priority srcnat - 1; policy accept;
}

Maybe CentOS NFTable Chains needs patching?

@johnson7788
Copy link
Author

@mlsmaycon
I tried it ,but it not works, see below:
johnson@wacserver7:$ cat /etc/sysconfig/netbird
NB_SKIP_NFTABLES_CHECK=true
johnson@wacserver7:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
371bce5deb08 netbirdio/netbird:latest "/usr/local/bin/netb…" 2 minutes ago Up 2 minutes netbirdapp
3fe3273373b6 rabbitmq:management "docker-entrypoint.s…" 3 years ago Up 3 months 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp rabbit

Docker started, but Peers ip is offline on https://app.netbird.io/peers.

cat /etc/release
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

After delete /etc/sysconfig/netbird
sudo rm /etc/sysconfig/netbird
sudo docker run --name netbirdapp --network host --privileged --rm -d -e NB_SETUP_KEY=xxxxxxx -v netbird-client:/etc/netbird netbirdio/netbird

Peers ip is online now, i think we can't turn off NB_SKIP_NFTABLES_CHECK. but when i turn off NB_SKIP_NFTABLES_CHECK, docker network is work, like below
docker run --rm --net=host -it busybox ping 123.150.76.218
PING 123.150.76.218 (123.150.76.218): 56 data bytes
64 bytes from 123.150.76.218: seq=0 ttl=52 time=10.719 ms
64 bytes from 123.150.76.218: seq=1 ttl=52 time=9.302 ms

@mlsmaycon
Copy link
Collaborator

hello @johnson7788 for the container, please add the environment variable as part of its running config. e.g.:

sudo docker run --name netbirdapp --network host --privileged --rm -d -e NB_SETUP_KEY=xxxxxxx -e NB_SKIP_NFTABLES_CHECK=true -v netbird-client:/etc/netbird netbirdio/netbird

@mlsmaycon
Copy link
Collaborator

the release 0.27.8 contains the fix. Let us know if there is any other issue

@johnson7788
Copy link
Author

@mlsmaycon Great jobs!, thank you very much, it works for me, docker network is ok now. and netbird network also works. You are so amazing

sudo docker run --rm busybox ping so.com
PING so.com (180.163.237.15): 56 data bytes
64 bytes from 180.163.237.15: seq=0 ttl=51 time=29.483 ms

@johnson7788
Copy link
Author

the release 0.27.8 contains the fix. Let us know if there is any other issue

Yes, I install on an Centos server with latest 0.27.9, Both docker network and netbird network are working. That's great!!!

@johnson7788
Copy link
Author

My issue all be fixed. thank you everyone! @mlsmaycon @thorleifjacobsen @enyachoke

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants