Unable to access subnet router in Docker

[vanhtuan0409]

What is the issue?

I have a home lab with 2 nodes. Both running Tailscale and advertising the same list of subnets and in the same LAN subnet. The difference is 1 node running Tailscale natively in the host OS (node A) and 1 node is running Tailscale as K8s deployment (node B).

Both nodes are able to start Tailscale and I am able to ping both nodes via Tailscale ip (direct connect). However, if I allowed subnet routes for both nodes via Tailscale Admin Page (https://login.tailscale.com/admin/machines), I cannot access the advertised subnets. Only once I disable the subnet routes for node B and allowed for node A, then I can access the advertised subnets. In summary, things I have tried:

• Enabled subnet on both Node => cannot access
• Enabled subnet on Node A, disabled subnet on Node B => success
• Disabled subnet on node A, enabled subnet on Node B => cannot access

I'm expecting to reach the advertised subnets via both nodes

Steps to reproduce

Disable ipv6 supporting via net.ipv6.conf.all.disable_ipv6 = 1

K8s deployment manifest

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: tailscale
spec:
  template:
    spec:
      hostNetwork: true
      serviceAccountName: tailscale
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: tailscale
          image: tailscale/tailscale:v1.20.2
          imagePullPolicy: IfNotPresent
          command:
            - /bin/sh
            - /data/bootstrap/run.sh
          args:
            - --advertise-routes=<redacted>
          env:
            - name: KUBE_SECRET
              value: tailscale-state
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          envFrom:
            - secretRef:
                name: tailscale-secret
          volumeMounts:
            - name: bootstrap
              mountPath: /data/bootstrap
              readOnly: true
            - name: devnet
              mountPath: /dev/net
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
      volumes:
        - name: bootstrap
          configMap:
            name: tailscale-bootstrap
        - name: devnet
          hostPath:
            path: /dev/net

Bootstrap script

#! /bin/sh

export PATH=$PATH:/tailscale/bin

AUTH_KEY="${AUTH_KEY:-}"
KUBE_SECRET_PREFIX="${KUBE_SECRET:-tailscale}"
NODE_NAME="${NODE_NAME:-}"

set -e

if [[ ! -d /dev/net ]]; then mkdir -p /dev/net; fi
if [[ ! -c /dev/net/tun ]]; then mknod /dev/net/tun c 10 200; fi

echo "Starting tailscaled"
tailscaled \
  --state=kube:${KUBE_SECRET_PREFIX}-${NODE_NAME} \
  --socket=/tmp/tailscaled.sock &
PID=$!

echo "Running tailscale up"
tailscale --socket=/tmp/tailscaled.sock up \
  --accept-dns=false \
  --authkey=${AUTH_KEY} \
  --hostname=${NODE_NAME} \
  $@

wait ${PID}

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Linux merlion 5.13.0-28-generic #31-Ubuntu SMP Thu Jan 13 17:41:06 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Tailscale version

1.20.2

Bug report

BUG-f71c8045f39d571c5184bac8ef0e6e91fa16b0f11615547b7672915a56083762-20220210031749Z-82e119fa610d100d

[maisem]

@vanhtuan0409 can you verify you have IP Forwarding enabled on node B by running:

cat /proc/sys/net/ipv4/ip_forward

[vanhtuan0409]

cc @maisem yeah I have ip forward enabled for ipv4

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.all.disable_ipv6 = 1

FYI: node B is actually running K3s. Not sure how subnet failover behaviour on non-bussiness account
https://tailscale.com/kb/1115/subnet-failover/

[DentonGentry]

Disable ipv6 supporting via net.ipv6.conf.all.disable_ipv6 = 1

Is there a reason to do this?

[vanhtuan0409]

If I enabled ipv6, tailscale will try to insert rules into iptables6 which then failed and disabled ip forwarding completely (even if ipv4) was sucessfully added

[DentonGentry]

This is Alpine? Installing the ip6tables package can likely resolve that issue.

[vanhtuan0409]

Docker container running alpine but host os running ubuntu. Anw I think the ipv6 is not related to this problem

…

On Thu, 10 Feb 2022 at 12:57 Denton Gentry ***@***.***> wrote: This is Alpine? Installing the ip6tables package can likely resolve that issue. — Reply to this email directly, view it on GitHub <#3899 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6MCPEIZMYILWYU2TPRNCLU2NHTHANCNFSM5N7OH3OQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[vanhtuan0409]

I might found the reason behind this. On node B, default policy for chain FORWARD is DROP while on node A it is ACCEPT
Changing default policy to accept by iptables --policy FORWARD ACCEPT did fix the problem
Any suggestion for a more restrictive rules :D

[DentonGentry]

Since the routing issue is resolved, I'll close this. You're welcome to try to tighten the iptables rules, but that is out of scope here.

[vanhtuan0409]

I will just leave a note here in case anyone needs it. The root cause is Ubuntu 21.01 running 2 different iptables versions (nft and legacy). By default, Ubuntu 21.01 will use nft one (via iptables-nft binary), causing k3s to insert rules into NFT tables. Tailscale docker image using the legacy one instead.

The solution is to rewire Ubuntu iptables by update-alternatives --set iptables /usr/sbin/iptables-legacy (the same with ipt6tables) and then reboot the server