Can't Ping Tailscale Machines From Docker #7382

NathanMagnus · 2023-02-26T07:06:00Z

What is the issue?

I have Tailscale running on Android Phone and as a docker container on Ubuntu 22.04.2. I am trying to set tailscale up in subnet routing mode from the Ubuntu machine, but this is not working. I also cannot ping any of the other machines on the tailnet.``

Steps to reproduce

I am using the Android app on the android phone.
On my Ubuntu machine, I am using this docker-compose file:

version: '3.8'
services:
  tailscale:
    image: tailscale/tailscale
    container_name: tailscale
    restart: always
    privileged: true
    command: /bin/sh -c "tailscaled --tun=userspace-networking"                                                                                              
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
    environment:
      - TS_AUTHKEY=${TAILSCALE_AUTH_KEY}
      - TS_ROUTES=192.168.50.0/24
    volumes:
      - /etc/docker/tailscale:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun

after the docker container is up, I am running sudo docker exec tailscale tailscale up --auth-key=tskey-auth-AUTHKEY --advertise-routes=192.168.50.0/24 --hostname=hostName --accept-routes

After this point, running

sudo docker exec tailscale tailscale ping AndroidPhoneName
pong from AndroidPhoneName (100.102.174.X) via DERP(tor) in 401ms

But a regular ping times out:

ping 100.102.174.X
PING 100.102.174.X (100.102.174.X) 56(84) bytes of data.

Additionally

sudo docker exec tailscale tailscale ping --tsmp 100.102.174.X
ping "100.102.174.X" timed out

Are there any recent changes that introduced the issue?

No response

OS

Linux, Android

OS version

Ubuntu 22.04.2

Tailscale version

No response

Other software

No response

Bug report

Docker container: BUG-5904a47322fd35a5404973b6f033d4f6f17d5e7ec072d3bd52d717ea7ca7657a-20230227181435Z-9d7d6058e8349c44
Android device: BUG-f349d36777b462ac621e2d53952e7b73df791b27f0d1fb831a8787501d19a3bd-20230227181604Z-07a087fb0dc032c1

The text was updated successfully, but these errors were encountered:

DentonGentry · 2023-02-27T13:20:58Z

Because /dev/net/tun is available:

    volumes:
      - /dev/net/tun:/dev/net/tun

There isn't a reason to use tailscaled --tun=userspace-networking. Dropping the --tun= argument will let it use /dev/net/tun, which is generally better for subnet routing as it lets the Linux kernel do the forwarding.

Cases where tailscale ping works and regular ping does not are usually due to a firewall blocking traffic, but I'd recommend switching to /dev/net/tun first and trying again before debugging further.

    cap_add:
      - NET_ADMIN
      - SYS_ADMIN

Adding NET_RAW is also useful for tailscaled

NathanMagnus · 2023-02-27T14:10:46Z

I just tried this out. I'm still getting the same behavior.

Here is my docker-compose now:

  tailscale:
    image: tailscale/tailscale
    container_name: tailscale
    restart: always
    privileged: true
    command: /bin/sh -c "tailscaled"
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
      - NET_RAW
    environment:
      - TS_AUTHKEY=${TAILSCALE_AUTH_KEY}
      - TS_ROUTES=192.168.50.0/24
    volumes:
      - /etc/docker/tailscale:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun

When running tailscale up, I get this warning:

sudo docker exec tailscale tailscale up --auth-key=tskey-auth-AUTHKEY --advertise-routes=192.168.50.0/24 --hostname=hostName --accept-routes
Warning: IPv6 forwarding is disabled.
Subnet routes and exit nodes may not work correctly.
See https://tailscale.com/kb/1104/enable-ip-forwarding/

My /etc/sysctl.d/99-tailscale.conf contains only two line:

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

My /etc/sysctl.conf is much larger but does have the following lines uncommented:

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

ufw appears to be inactive:

sudo ufw status
Status: inactive

firewalld is not installed:

apt list --installed | grep firewall
Returns an empty list.

I notice that this change no longer shows routes when executing

sudo ip route show table 52
Error: ipv4: FIB table does not exist.
Dump terminated

I can see in tailscale admin portal that my linux host is up and running 1.36.2 and my android connection is also up running 1.36.1. I also see the accepted subnet routes ephemeral flag and expiry disabled.

DentonGentry · 2023-02-27T14:18:57Z

At this point a bugreport would be best, so we can look at the configuration of the tailnet and what it reports. Immediately after ping 100.102.174.X would be most useful. If that is the Android device, there is a menu selection in the three dot menu at the top right to generate a bugreport.

NathanMagnus · 2023-02-27T18:17:37Z

Ok, here are the bugreports.

Docker container: BUG-5904a47322fd35a5404973b6f033d4f6f17d5e7ec072d3bd52d717ea7ca7657a-20230227181435Z-9d7d6058e8349c44

immediately following a failed ping
prior to that is another failed ping
prior to that is a successful tailscale ping

Android device: BUG-f349d36777b462ac621e2d53952e7b73df791b27f0d1fb831a8787501d19a3bd-20230227181604Z-07a087fb0dc032c1

immediately after a failed ping

DentonGentry · 2023-03-04T01:46:51Z

@NathanMagnus something appears to be wrong in the routing table, there are no ACL evaluations about ICMP around the time of the bugreport. Even if the ping fails, the packet should trigger ACL evaluations when it tries to egress the node. It is like the ping is being sent to a different network interface.

@nunofgs your issue is different. Near the BUG line:

2023-03-03 22:46:08.198153883 +0000 UTC: Program starting: v1.36.2-t0438c67e2, Go 1.19.4-tsdc0ce6324d: []string{"tailscaled", "--socket=/tmp/tailscaled.sock", "--statedir=/var/lib/tailscale", "--tun=userspace-networking"}

tailscaled is starting up in userspace-networking mode, meaning the /dev/net/tun device isn't available. In userspace-networking mode applications have to use a SOCKS5 proxy to be able to initiate connections to the tailnet, and ping does not.

I'd suggest setting TS_USERSPACE=false and seeing if it complains about what is wrong when it starts up.

NathanMagnus · 2023-03-06T15:50:36Z

@DentonGentry That is strange. Pinging google works as expected both within and outside of the container.

Here is another bugreport generated from this line:
sudo docker exec tailscale tailscale ping PHONE; ping google.ca -c 5; ping 100.102.174.3 -c 2; sudo docker exec tailscale ping google.ca -c 5; sudo docker exec tailscale ping 100.102.174.3 -c 2; sudo docker exec tailscale tailscale bugreport

BUG-5904a47322fd35a5404973b6f033d4f6f17d5e7ec072d3bd52d717ea7ca7657a-20230306154717Z-8d9b8cab24221578

I'm not really sure what other debugging steps I can take here.

A brief history, I have been running everything containerized for quite some time. However I recently needed to update because the ubuntu machine was running tailscale 1.9.0 and my android client could no longer connect to it successfully. Since then, nothing has been working.

NathanMagnus · 2023-03-25T20:41:36Z

After some experimenting including the recommendations above, I have gotten to the point where pinging from the tailscale container provides the following in logs:
command: sudo docker exec -it tailscale ping [phone ip]

2023/03/25 20:36:06 Accept: ICMPv4{[ubuntu ip]:0 > [phone ip]:0} 84 ok out
It doesn't return successes to the ping though.

My current docker-compose is:

  tailscale:
    image: tailscale/tailscale
    container_name: tailscale
    restart: always
    privileged: true
    command: /var/lib/tailscale/startup.sh
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
      - NET_RAW
    environment:
      - TS_USERSPACE=false
      - TS_AUTHKEY=${TAILSCALE_AUTH_KEY}
      - TS_ROUTES=[local subnet]/24
      - TS_EXTRA_ARGS=--accept-routes
      - TS_SOCKET=/var/run/tailscale/tailscaled.sock
    volumes:
      - /etc/docker/tailscale:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun

and startup.sh:

#!/bin/sh
echo "Executing Startup Script"

sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -t nat -A POSTROUTING -j MASQUERADE
echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.conf
echo 'net.ipv6.conf.all.forwarding = 1' | tee -a /etc/sysctl.conf
sysctl -p /etc/sysctl.conf

echo "starting tailscaled"
tailscaled --statedir=/tmp &

echo "sleeping"
sleep 10s

echo "starting tailscale"
tailscale up --reset --auth-key=[auth key] --accept-routes --advertise-routes=[local subnet]/24

wait $!

Here is a new
Bug Report for you @DentonGentry from the Ubuntu machine: BUG-5904a47322fd35a5404973b6f033d4f6f17d5e7ec072d3bd52d717ea7ca7657a-20230325203619Z-03437b657ddc9d6f

DentonGentry · 2023-04-01T09:35:19Z

We do now see ICMP ACL evaluations:

2023-03-25 20:38:40.123032366 +0000 UTC: Accept: ICMPv4{100.112.x.y:0 > 100.68.x.y:0} 84 ok out
2023-03-25 20:38:48.123378536 +0000 UTC: Accept: ICMPv4{100.112.x.y:0 > 100.68.x.y:0} 84 ok out
2023-03-25 20:38:58.123989229 +0000 UTC: Accept: ICMPv4{100.112.x.y:0 > 100.68.x.y:0} 84 ok out
2023-03-25 20:39:08.124442631 +0000 UTC: Accept: ICMPv4{100.112.x.y:0 > 100.68.x.y:0} 84 ok out

The 100.112.x.y node no longer exists, no way to check what it was reporting at the time.

NathanMagnus · 2023-04-29T18:27:21Z

Sorry for the slow response. Last month was very busy.

I am now testing on two Ubuntu machines.

Both machines are using the same docker-compose (with different env variables for auth key and different TS_ROUTES):

tailscale:
image: tailscale/tailscale
container_name: tailscale
restart: always
privileged: true
command: /var/lib/tailscale/startup.sh
cap_add:
- NET_ADMIN
- SYS_ADMIN
- NET_RAW
environment:
- TS_USERSPACE=false
- TS_AUTHKEY=${TAILSCALE_AUTH_KEY}
- TS_ROUTES=MachineSubnet/24
- TS_EXTRA_ARGS=--accept-routes
- TS_SOCKET=/var/run/tailscale/tailscaled.sock
volumes:
- /etc/docker/tailscale:/var/lib/tailscale
- /dev/net/tun:/dev/net/tun

Startup.sh:

#!/bin/sh
echo "Executing Startup Script"
sleep 10s

#wget -O - https://tailscale.com/install.sh | /bin/sh

echo "adding routing stuff"
sysctl -w net.ipv4.conf.all.route_localnet=1
#iptables -A FORWARD -i eth0 -j ACCEPT
#iptables -t nat -A POSTROUTING -o eth0 -j ACCEPT
iptables -t nat -A POSTROUTING -j MASQUERADE
cat /etc/sysctl.conf
echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.conf
echo 'net.ipv6.conf.all.forwarding = 1' | tee -a /etc/sysctl.conf
sysctl -p /etc/sysctl.conf

echo "starting tailscaled"
tailscaled --statedir=/tmp &

echo "sleeping"
sleep 10s

echo "starting tailscale"
tailscale up --reset --auth-key=tskey-auth-AUTHKEY --accept-routes --advertise-routes=192.168.XX.XX/24 --advertise-tags=tag:MachineAorB --hostname=MachineAorB

wait $!

Command from Machine A:

echo "Tailscale ping name through container"; \
sudo docker exec tailscale tailscale ping -c 2 MachineB; \
echo "Ping ip no container";  \
sudo ping -c 2 100.126.BB.BB; \
echo "Ping name through container";\
sudo docker exec tailscale ping -c 2 MachineB; \
echo "Ping ip through container"; \
sudo docker exec tailscale ping -c 100.126.BB.BB; \
echo "Tailscale bug report"; \
sudo docker exec tailscale tailscale bugreport

BUG-5904a47322fd35a5404973b6f033d4f6f17d5e7ec072d3bd52d717ea7ca7657a-20230429182437Z-77bafedeafab704e

Command from Machine B:

echo "Tailscale ping name through container"; \
sudo docker exec tailscale tailscale ping -c 2 MachineA; \
echo "Ping ip no container";  \
sudo ping -c 2 100.94.AA.AA; \
echo "Ping name through container";\
sudo docker exec tailscale ping -c 2 MachineA; \
echo "Ping ip through container"; \
sudo docker exec tailscale ping -c 100.94.AA.AA; \
echo "Tailscale bug report"; \
sudo docker exec tailscale tailscale bugreport

BUG-3d03e3f9b6a7d6f112c400de694198fe23a5066de76a68f23389454f911d6f61-20230429182445Z-12bc2d7af7bec190

I'll try to leave these containers running for as long as I can.

DentonGentry · 2023-06-04T04:12:55Z

Around the time of BUG-5904a47322fd35a5404973b6f033d4f6f17d5e7ec072d3bd52d717ea7ca7657a-20230429182437Z-77bafedeafab704e:

2023-04-29 18:24:13.994895387 +0000 UTC: ping(100.126.x.y): sending disco ping to [+BPK8] fileserver5 ...
2023-04-29 18:25:20.903820651 +0000 UTC: magicsock: derp route for [+BPK8] changed from derp-21 => derp-10 (their home)
2023-04-29 18:25:20.903986934 +0000 UTC: magicsock: derp route for [+BPK8] changed from derp-10 => derp-21 (shared home)

Around the time of BUG-3d03e3f9b6a7d6f112c400de694198fe23a5066de76a68f23389454f911d6f61-20230429182445Z-12bc2d7af7bec190

2023-04-29 18:20:07.607091783 +0000 UTC: magicsock: derp route for [lZKYk] changed from derp-21 => derp-10 (shared home)
2023-04-29 18:24:38.605193289 +0000 UTC: ping(100.94.x.y): sending disco ping to [lZKYk] zariel ...
2023-04-29 18:24:38.605415964 +0000 UTC: magicsock: adding connection to derp-21 for [lZKYk]
2023-04-29 18:24:38.605496301 +0000 UTC: magicsock: 2 active derp conns: derp-10=cr8m0s,wr20s derp-21=cr0s,wr0s
2023-04-29 18:24:38.605532032 +0000 UTC: magicsock: derp route for [lZKYk] changed from derp-10 => derp-21 (their home)
2023-04-29 18:24:38.605607632 +0000 UTC: magicsock: derp route for [lZKYk] changed from derp-21 => derp-10 (shared home)

So they are just saying that Disco Ping didn't respond.

NathanMagnus · 2023-06-06T04:11:16Z

@DentonGentry I'm not quite sure what that means for me. What do I need to look into in my end?

upsangel · 2023-10-30T23:18:39Z

I am also experiencing the same problem. Inside the docker exec shell, I can ping through both TS virtual IP and the advertised subnet IP. However, in the host Debian machine, I couldn't ping any of them. Hereby below my docker-compose file.

version: "3"

services:
  tailscale:
    container_name: tailscale
    hostname: tailscale-frigate-n100
    image: tailscale/tailscale:latest
    volumes:
      - ./tailscale:/var/lib # State data will be stored in this directory
      - /dev/net/tun:/dev/net/tun # Required for tailscale to work
    cap_add: # Required for tailscale to work
      - net_admin
      - sys_module
    command: tailscaled
    privileged: true
    restart: unless-stopped
    #environment:
      #TS_EXTRA_ARGS: "--ssh=true --advertise-exit-node" # enable Tailscale SSH
      #TS_ROUTES: 192.168.1.0/24,192.168.2.0/24

DentonGentry · 2023-10-30T23:25:54Z

However, in the host Debian machine, I couldn't ping any of them.

The Debian host machine has Tailscale installed?
Does it have tailscale up --accept-routes ?

VasylSamagala97 · 2023-11-03T17:35:24Z

Same issue here, i cant ping from container to my other Devices in network...this is my start lo:my startup : #!/bin/sh

/usr/sbin/tailscaled -tun=userspace-networking &
/usr/bin/tailscale up --authkey=${TAILSCALE_AUTHKEY} --hostname=azure-app
exec uwsgi --ini uwsgi.ini

my compose version: '3.8'
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8000:80"
      - "2222:22"
    environment:
      - PASSWORD=${PASSWORD}
      - DEBUG=${DEBUG}
      - DATABASE_HOST=${DATABASE_HOST}
      - DATABASE_NAME=${DATABASE_NAME}
      - DATABASE_USER=${DATABASE_USER}
      - DATABASE_PASSWORD=${DATABASE_PASSWORD}
      - FLASK_RUN_PORT=${FLASK_RUN_PORT}
      - TAILSCALE_AUTHKEY=${TAILSCALE_AUTHKEY}
    volumes:
      - brevo_webhook:/app
    

volumes:
  brevo_webhook:
    driver: local
  
    driver_opts:
      type: none
      o: bind 
      device: /Users/vasylsamagala/Desktop/Desarrollo/brevo_webhook/ 
IF I DO tailscale ping to other device, its response without problem.

NathanMagnus · 2024-03-24T19:38:19Z

My issue has gone away since I originally filed this bug. Only action I took was to use a newer version.

NathanMagnus added bug Bug needs-triage labels Feb 26, 2023

NathanMagnus changed the title ~~Can't Ping Tailscale Machines~~ Can't Ping Tailscale Machines From Docker Feb 26, 2023

This comment was marked as off-topic.

Sign in to view

DentonGentry added connectivity L1 Very few Likelihood P3 Can't get started Priority level T5 Usability Issue type and removed needs-triage labels Apr 1, 2023

NathanMagnus closed this as completed Mar 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't Ping Tailscale Machines From Docker #7382

Can't Ping Tailscale Machines From Docker #7382

NathanMagnus commented Feb 26, 2023 •

edited

DentonGentry commented Feb 27, 2023 •

edited

NathanMagnus commented Feb 27, 2023

DentonGentry commented Feb 27, 2023

NathanMagnus commented Feb 27, 2023

This comment was marked as off-topic.

DentonGentry commented Mar 4, 2023

NathanMagnus commented Mar 6, 2023

This comment was marked as off-topic.

NathanMagnus commented Mar 25, 2023 •

edited

This comment was marked as off-topic.

DentonGentry commented Apr 1, 2023

NathanMagnus commented Apr 29, 2023

DentonGentry commented Jun 4, 2023

NathanMagnus commented Jun 6, 2023

upsangel commented Oct 30, 2023

DentonGentry commented Oct 30, 2023

VasylSamagala97 commented Nov 3, 2023

NathanMagnus commented Mar 24, 2024

Can't Ping Tailscale Machines From Docker #7382

Can't Ping Tailscale Machines From Docker #7382

Comments

NathanMagnus commented Feb 26, 2023 • edited

What is the issue?

Steps to reproduce

Are there any recent changes that introduced the issue?

OS

OS version

Tailscale version

Other software

Bug report

DentonGentry commented Feb 27, 2023 • edited

NathanMagnus commented Feb 27, 2023

DentonGentry commented Feb 27, 2023

NathanMagnus commented Feb 27, 2023

This comment was marked as off-topic.

DentonGentry commented Mar 4, 2023

NathanMagnus commented Mar 6, 2023

This comment was marked as off-topic.

NathanMagnus commented Mar 25, 2023 • edited

This comment was marked as off-topic.

DentonGentry commented Apr 1, 2023

NathanMagnus commented Apr 29, 2023

DentonGentry commented Jun 4, 2023

NathanMagnus commented Jun 6, 2023

upsangel commented Oct 30, 2023

DentonGentry commented Oct 30, 2023

VasylSamagala97 commented Nov 3, 2023

NathanMagnus commented Mar 24, 2024

NathanMagnus commented Feb 26, 2023 •

edited

DentonGentry commented Feb 27, 2023 •

edited

NathanMagnus commented Mar 25, 2023 •

edited