tailscaled doesn't start on Google Colaboratory

[demotomohiro]

Describe the bug
I tried to use tailscale on Google Colaboratory but I got error when I tried to run tailscaled --state=tailscaled.state.

System info

!cat /etc/lsb-release
!uname -a

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
Linux  4.19.104+ #1 SMP Wed Feb 19 05:26:34 PST 2020 x86_64 x86_64 x86_64 GNU/Linux

CONFIG_TUN is available:

!zcat /proc/config.gz | grep CONFIG_TUN

CONFIG_TUN=m
# CONFIG_TUN_VNET_CROSS_LE is not set

Container info using container introspection tool:
https://github.com/genuinetools/amicontained

!amicontained

Container Runtime: docker
Has Namespaces:
	pid: true
	user: false
AppArmor Profile: datalabvm (enforce)
Capabilities:
	BOUNDING -> chown dac_override fowner fsetid kill setgid setuid setpcap net_bind_service net_raw sys_chroot sys_ptrace sys_admin mknod audit_write setfcap
Seccomp: filtering
Blocked Syscalls (47):
	SETSID USELIB USTAT SYSFS VHANGUP PIVOT_ROOT _SYSCTL ACCT SETTIMEOFDAY SWAPON SWAPOFF REBOOT IOPL IOPERM CREATE_MODULE INIT_MODULE DELETE_MODULE GET_KERNEL_SYMS QUERY_MODULE NFSSERVCTL GETPMSG PUTPMSG AFS_SYSCALL TUXCALL SECURITY CLOCK_SETTIME VSERVER MBIND SET_MEMPOLICY GET_MEMPOLICY KEXEC_LOAD ADD_KEY REQUEST_KEY KEYCTL MIGRATE_PAGES MOVE_PAGES OPEN_BY_HANDLE_AT CLOCK_ADJTIME FINIT_MODULE KEXEC_FILE_LOAD USERFAULTFD MEMBARRIER PKEY_MPROTECT PKEY_ALLOC PKEY_FREE IO_PGETEVENTS RSEQ

To Reproduce
I installed tailscale with following command:

!curl https://pkgs.tailscale.com/stable/ubuntu/bionic.gpg | sudo apt-key add -
!curl https://pkgs.tailscale.com/stable/ubuntu/bionic.list | sudo tee /etc/apt/sources.list.d/tailscale.list
!sudo apt-get update
!sudo apt-get install tailscale

As systemd is not working, tailscaled need to be run manually but failed:

!tailscaled --state=tailscaled.state

2020/08/03 00:33:30 logpolicy.Read /root/.cache/Tailscale/tailscaled.log.conf: open /root/.cache/Tailscale/tailscaled.log.conf: no such file or directory
logtail started
Program starting: v0.100.0-153-gc6c838785, Go 1.14.4-ts56db765: []string{"tailscaled", "--state=tailscaled.state"}
LogID: 3f96acd67e70b830a5a2b0482eb3e276917c58fc9774164c4c7cd1e3f83250a5
6.4M/93.0M Starting userspace wireguard engine with tun device "tailscale0"
3.1M/93.0M Linux kernel version: 4.19.104+
3.1M/93.0M is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: FATAL: Module tun not found in directory /lib/modules/4.19.104+
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 59ms)
logtail: upload: log upload of 252 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 12 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 252 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 58 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 252 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 121 msec
3.1M/93.0M tun module not loaded nor found on disk
3.1M/93.0M CreateTUN: can't create TUN device; /dev/net/tun does not exist
wgengine.New: can't create TUN device; /dev/net/tun does not exist

I read #504 and created /dev/net/tun:

!mkdir /dev/net
!mknod /dev/net/tun c 10 200
!chmod 0666 /dev/net/tun

Then I tried to run tailscaled again:
(modprobe tun failed because Google colab runtime uses kernel 4.19 but there is no /lib/modules/4.19.104 directory but /lib/modules/4.15.0-112-generic)

!tailscaled --state=tailscaled.state

logtail started
Program starting: v0.100.0-153-gc6c838785, Go 1.14.4-ts56db765: []string{"tailscaled", "--state=tailscaled.state"}
LogID: 3f96acd67e70b830a5a2b0482eb3e276917c58fc9774164c4c7cd1e3f83250a5
5.2M/93.0M Starting userspace wireguard engine with tun device "tailscale0"
3.1M/93.0M Linux kernel version: 4.19.104+
3.1M/93.0M is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: FATAL: Module tun not found in directory /lib/modules/4.19.104+
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 56ms)
logtail: upload: log upload of 756 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 12 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 756 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 58 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 756 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 121 msec
3.3M/93.0M tun module not loaded nor found on disk
3.3M/93.0M CreateTUN: operation not permitted
wgengine.New: operation not permitted

That error message says tun module not loaded but there is tun module:

!lsmod

Module                  Size  Used by
tun                    49152  0
xt_nat                 16384  8
veth                   24576  0
cls_u32                24576  2
sch_htb                24576  1
ipt_MASQUERADE         16384  1
iptable_nat            16384  3
nf_nat_ipv4            16384  2 ipt_MASQUERADE,iptable_nat
xt_addrtype            16384  2
nf_nat                 53248  2 nf_nat_ipv4,xt_nat
br_netfilter           24576  0
ip6table_filter        16384  1
ip6_tables             28672  1 ip6table_filter
aesni_intel           200704  0
virtio_balloon         20480  0
aes_x86_64             20480  1 aesni_intel
crypto_simd            16384  1 aesni_intel
cryptd                 28672  2 crypto_simd,aesni_intel
glue_helper            16384  1 aesni_intel

This document says CAP_NET_ADMIN is required for creating network devices.
https://github.com/torvalds/linux/blob/master/Documentation/networking/tuntap.rst

It seems tailscaled failed to run because google colab runtime doesn't have CAP_NET_ADMIN capability.
tailscale cannot work without CAP_NET_ADMIN capability?

I have a python script that automatically setup ssh server on Google Colaboratory and allow login to it using ngrok.
https://github.com/demotomohiro/remocolab
And I tried to use tailscale on Google Colaboratory because tailscale looks faster and more secure alternative to ngrok.

[danderson]

There's multiple things missing here.

Tailscale does need CAP_NET_ADMIN, because it has to reconfigure the network stack of the machine to route traffic through the VPN.

This virtual environment also seems to be missing the tuntap device driver. Tailscale can't function without that since there's no way for us to receive network traffic from the system without tuntap.

If you can suggest changes to this sandbox, adding CAP_NET_ADMIN and bind-mounting the /dev/net/tun device into the Docker container should be enough to get Tailscale working. Unfortunately, there's not much we can do without those two things :(

[bradfitz]

tun module not loaded nor found on disk

That's a result of dpkg -S kernel/drivers/net/tun.ko not finding tun.ko in your container, even though it exists on the outer host environment.

[bradfitz]

Tailscale can't function without that since there's no way for us to receive network traffic from the system without tuntap.

This would be yet another use case for a pure-userspace tailscaled, using e.g. https://github.com/google/netstack

[danderson]

Such a tailscaled would have limited functionality compared to a regular system. It's interesting to explore at some point, assuming this environment can't be made more friendly, but it's not a small undertaking.

[apenwarr]

Once we get into hosting an http proxy inside tailscaled, there are actually tons of use cases for this that would work well, similarly to sshuttle. Another great use case would be running as non-admin on Windows machines. ᐧ

…

On Mon, Aug 3, 2020 at 4:57 PM Dave Anderson ***@***.***> wrote: Such a tailscaled would have limited functionality compared to a regular system. It's interesting to explore at some point, assuming this environment can't be made more friendly, but it's not a small undertaking. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#634 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFA4EMRAI2IF64NIMPVB3R64QB3ANCNFSM4PTWAZ7A> .

-- Avery Pennarun // CEO @ Tailscale

[DentonGentry]

#707 tracks the userspace proxy mentioned earlier.

Google Colab is unlikely to be supported any other way, there is little hope of getting TUN and/or CAP_NET_ADMIN.

[DentonGentry]

Note that the userspace tailscaled is working at HEAD and in the 1.6 release.

tailscaled --tun=userspace-networking --socks5-server=localhost:1055 &

[demotomohiro]

Note that the userspace tailscaled is working at HEAD and in the 1.6 release.

tailscaled --tun=userspace-networking --socks5-server=localhost:1055 &

@DentonGentry, sorry for the late reply.

I tried to run tailscale on Google Colaboratory again as you told, but I got error "magicsock: bind(udp4/0): listen udp4 :0: setting SO_MARK bypass: operation not permitted".
According to this man page, setting SO_MARK option requires the CAP_NET_ADMIN capability.

I installed tailscale with following command:

!curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/bionic.gpg | sudo apt-key add -
!curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/bionic.list | sudo tee /etc/apt/sources.list.d/tailscale.list
!sudo apt-get update
!sudo apt-get install tailscale

Then I ran the following command:

!tailscaled --tun=userspace-networking --socks5-server=localhost:1055

Here is output:

logtail started
Program starting: v1.6.0-tddc975fcb-g995460c32, Go 1.16.2-ts3b83fb3: []string{"tailscaled", "--tun=userspace-networking", "--socks5-server=localhost:1055"}
LogID: 6586cb9d9d65102cd03913609a81d502701f43777b82448238c07e1db028f2af
logpolicy: using system state directory "/var/lib/tailscale"
logpolicy.Read /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory
wgengine.NewUserspaceEngine(tun "userspace-networking") ...
link state: interfaces.State{defaultRoute=eth0 ifs={eth0:[172.28.0.2/16]} v4=true v6global=false}
magicsock: bind(udp4/0): listen udp4 :0: setting SO_MARK bypass: operation not permitted
wgengine.NewUserspaceEngine(tun "userspace-networking") error: wgengine: magicsock: bind: udp4/0: listen udp4 :0: setting SO_MARK bypass: operation not permitted
wgengine.New: wgengine: magicsock: bind: udp4/0: listen udp4 :0: setting SO_MARK bypass: operation not permitted
flushing log.
logger closing down
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 32ms)
logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 12 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 40 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 1ms)
logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 83 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 161 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 311 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 409 msec

[apenwarr]

Definitely should not be doing SO_MARK when in userspace mode. I'm a little surprised we didn't catch that before? Maybe everyone has been testing it as root. ᐧ

…

On Thu, Apr 8, 2021 at 3:05 AM Tomohiro ***@***.***> wrote: Note that the userspace tailscaled is working at HEAD and in the 1.6 release. tailscaled --tun=userspace-networking --socks5-server=localhost:1055 & @DentonGentry <https://github.com/DentonGentry>, sorry for the late reply. I tried to run tailscale on Google Colaboratory again as you told, but I got error "magicsock: bind(udp4/0): listen udp4 :0: setting SO_MARK bypass: operation not permitted". According to this man page <https://www.man7.org/linux/man-pages/man7/socket.7.html>, setting SO_MARK option requires the CAP_NET_ADMIN capability. I installed tailscale with following command: !curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/bionic.gpg | sudo apt-key add - !curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/bionic.list | sudo tee /etc/apt/sources.list.d/tailscale.list !sudo apt-get update !sudo apt-get install tailscale Then I ran the following command: !tailscaled --tun=userspace-networking --socks5-server=localhost:1055 Here is output: logtail started Program starting: v1.6.0-tddc975fcb-g995460c32, Go 1.16.2-ts3b83fb3: []string{"tailscaled", "--tun=userspace-networking", "--socks5-server=localhost:1055"} LogID: 6586cb9d9d65102cd03913609a81d502701f43777b82448238c07e1db028f2af logpolicy: using system state directory "/var/lib/tailscale" logpolicy.Read /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory wgengine.NewUserspaceEngine(tun "userspace-networking") ... link state: interfaces.State{defaultRoute=eth0 ifs={eth0:[172.28.0.2/16]} v4=true v6global=false} magicsock: bind(udp4/0): listen udp4 :0: setting SO_MARK bypass: operation not permitted wgengine.NewUserspaceEngine(tun "userspace-networking") error: wgengine: magicsock: bind: udp4/0: listen udp4 :0: setting SO_MARK bypass: operation not permitted wgengine.New: wgengine: magicsock: bind: udp4/0: listen udp4 :0: setting SO_MARK bypass: operation not permitted flushing log. logger closing down logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 32ms) logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted logtail: backoff: 12 msec logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms) logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted logtail: backoff: 40 msec logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 1ms) logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted logtail: backoff: 83 msec logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms) logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted logtail: backoff: 161 msec logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms) logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted logtail: backoff: 311 msec logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms) logtail: upload: log upload of 380 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/90530037a0ccb003fb752ed145da70566232437a1029295b00a498b3dc102e70": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted logtail: backoff: 409 msec — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#634 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFA4G2WTBH7HJVOIB3TRLTHVITRANCNFSM4PTWAZ7A> .

-- Avery Pennarun // CEO @ Tailscale

[DentonGentry]

The code which calls SO_MARK has:

func ignoreErrors() bool {
        // If we're in a test, ignore errors. Assume the test knows
        // what it's doing and will do its own skips or permission
        // checks if it's setting up a world that needs netns to work.
        // But by default, assume that tests don't need netns and it's
        // harmless to ignore the sockopts failing.
        if flag.CommandLine.Lookup("test.v") != nil {
                return true
        }
        if os.Getuid() != 0 {
                // only root can manipulate these socket flags
                return true
        }
        return false
}

This is either too clever or not clever enough: Colab runs the binary as root, but SO_MARK isn't allowed. So we exit tailscaled.

We can get around this even in the current binary by running it as something non-root. I chose IRC because, as everyone knows, IRC is all powerful. I was able to get a Colab notebook to join the Tailnet like so:

!curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/bionic.gpg | sudo apt-key add -
!curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/bionic.list | sudo tee /etc/apt/sources.list.d/tailscale.list
!sudo apt-get update
!sudo apt-get install tailscale
!rm -rf /tmp/tailscaled
!mkdir -p /tmp/tailscaled
!chown irc.irc /tmp/tailscaled
!rm -rf /var/run/tailscale
!mkdir -p /var/run/tailscale
!chown irc.irc /var/run/tailscale
!cp /var/lib/tailscaled/tailscaled.state /tmp/tailscaled/tailscaled.state
!chown irc.irc /tmp/tailscaled/tailscaled.state
!nohup sudo -u irc tailscaled --tun=userspace-networking --socks5-server=localhost:1055 --state=/tmp/tailscaled/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock --port 41641 &
!until tailscale up; do sleep 1; done

It printed a message:

To authenticate, visit:

	https://login.tailscale.com/a/a-long-code

which I clicked on to authorize it. I now have a machine named 79c8ee43a59f in my Tailnet, which I immediately renamed as "George." As one does.

I'll work on making this nicer by ignoring a SO_MARK failure in Colab, even when running as root. This will remove the need for most of that ugly scripting as it will be able run with its normal paths.

To set expectations: running userspace-networking as a SOCKS5 server means that regular shell commands run from Colab !script will not suddenly be able to reach addresses on the Tailnet. Binaries have to be told to connect to the SOCKS5 proxy being run by tailscaled, for example:

!curl --socks5-hostname localhost:1055 http://100.100.1.1/some/path/to/something/