Tailscale crashes on OpenWRT after ~24 hours of normal usage

[0xjams]

What is the issue?

After 24 hours of usage, the binary of the architecture mipsle fails like this:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x6fe72000 pc=0x2889c]

goroutine 1 [running, locked to thread]:
runtime.throw({0x8ae889, 0x2a})
	runtime/panic.go:1198 +0x60 fp=0x14689c8 sp=0x14689b4 pc=0x5499c
runtime.sigpanic()
	runtime/signal_unix.go:719 +0x308 fp=0x14689e0 sp=0x14689c8 pc=0x73080
runtime.heapBitsSetType(0x1474180, 0x18, 0x18, 0x804800)
	runtime/mbitmap.go:1278 +0x8b4 fp=0x1468a44 sp=0x14689e4 pc=0x2889c
runtime.mallocgc(0x18, 0x804800, 0x1)
	runtime/malloc.go:1100 +0x8ac fp=0x1468a88 sp=0x1468a44 pc=0x1e314
runtime.convT2E(0x804800, 0x1468b1c)
	runtime/iface.go:325 +0x4c fp=0x1468aa0 sp=0x1468a88 pc=0x1b0fc
encoding/xml.init()
	encoding/xml/read.go:301 +0x1f4 fp=0x1468b34 sp=0x1468aa0 pc=0x3cb01c
runtime.doInit(0xcf5520)
	runtime/proc.go:6501 +0x1a0 fp=0x1468c58 sp=0x1468b34 pc=0x69bb4
runtime.doInit(0xcf5820)
	runtime/proc.go:6478 +0x94 fp=0x1468d7c sp=0x1468c58 pc=0x69aa8
runtime.doInit(0xcf7ba0)
	runtime/proc.go:6478 +0x94 fp=0x1468ea0 sp=0x1468d7c pc=0x69aa8
runtime.doInit(0xcf9f00)
	runtime/proc.go:6478 +0x94 fp=0x1468fc4 sp=0x1468ea0 pc=0x69aa8
runtime.main()
	runtime/proc.go:238 +0x268 fp=0x1468fec sp=0x1468fc4 pc=0x58440
runtime.goexit()
	runtime/asm_mipsx.s:625 +0x4 fp=0x1468fec sp=0x1468fec pc=0x974b8

Steps to reproduce

Run Tailscale version tailscale_1.18.2_mipsle on GL-MT1300 running OpenWRT

To fix the issue, the workaround is to reboot the router, delete old binaries, copy them again. And it starts working correctly.

Are there any recent changes that introduced the issue?

No response

OS

Other

OS version

OpenWrt 19.07.7

Tailscale version

1.18.2

Bug report

No response

[bradfitz]

To fix the issue, the workaround is to reboot the router, delete old binaries, copy them again. And it starts working correctly.

You have to recopy the binaries !? 😬

If you're seeing that level of corruption, I'm not sure Tailscale's involved here.

That's 256M total memory on that thing, right? How much is free for Tailscale before it starts?

[0xjams]

Hi!!!

Here's the memory before Tailscale:

root@GL-MT1300:/mnt/sda1/opt/tailscale/archive# free
              total        used        free      shared  buff/cache   available
Mem:         253132       56692      139788         416       56652      153196
Swap:             0           0           0

[0xjams]

The weird thing (I should've mentioned this before), but the binary's integrity is not compromised. The original file and the one that fails, have the same md5sum:

root@GL-MT1300:/etc/init.d# md5sum /opt/tailscale/archive/tailscale_1.18.2_mipsle/tailscale (original)
c6f6b9768410ade247ef6960462fbba6  /opt/tailscale/archive/tailscale_1.18.2_mipsle/tailscale
root@GL-MT1300:/etc/init.d# md5sum /usr/sbin/tailscale (running version)
c6f6b9768410ade247ef6960462fbba6  /usr/sbin/tailscale

But still, if I delete the old one and replace it -with apparently the same file-, everything works again.

Never mind, the client is unaffected, but apparently tailscaled gets modified (I just checked)

root@GL-MT1300:/etc/init.d# md5sum /usr/sbin/tailscaled (running version)
81d6f2bad684261fd85269f09b5b80ea  /usr/sbin/tailscaled
root@GL-MT1300:/etc/init.d# md5sum /opt/tailscale/archive/tailscale_1.18.2_mipsle/tailscaled  (original)
aa6fbaa3d32f53e1e3f6cd17d7f5210e  /opt/tailscale/archive/tailscale_1.18.2_mipsle/tailscaled

Is there anything I can do to monitor the file and see what causes the corruption?

[bradfitz]

Seems like your flash is bad or your kernel's corrupting things. I don't know how I or Tailscale can help here.

[0xjams]

Sorry, I'll keep debugging this. I'll just close the issue, and post back if I solve this.