netstack: unaligned 64-bit atomic operation on MIPS

[DentonGentry]

From #2129 (comment):

Different version, different arch

panic: unaligned 64-bit atomic operation

goroutine 193 [running]:
runtime/internal/atomic.panicUnaligned(...)
        runtime/internal/atomic/unaligned.go:8
runtime/internal/atomic.lockAndCheck(...)
        runtime/internal/atomic/atomic_mipsx.go:38
runtime/internal/atomic.Store64(0x214a56c, 0x8000)
        runtime/internal/atomic/atomic_mipsx.go:104 +0x94
inet.af/netstack/atomicbitops.(*AlignedAtomicInt64).Store(...)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/atomicbitops/aligned_64bit.go:39
inet.af/netstack/tcpip.(*SocketOptions).SetReceiveBufferSize(0x214a4f4, 0x8000, 0x0)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/socketops.go:663 +0x1c8
inet.af/netstack/tcpip/transport/udp.newEndpoint(0x2098000, 0x800, 0x20daba0)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/transport/udp/endpoint.go:187 +0x270
inet.af/netstack/tcpip/transport/udp.(*ForwarderRequest).CreateEndpoint(0x20dab80, 0x20daba0)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/transport/udp/forwarder.go:79 +0x11c
tailscale.com/wgengine/netstack.(*Impl).acceptUDP(0x20d3b00, 0x20dab80)
        tailscale.com@v1.16.2/wgengine/netstack/netstack.go:597 +0xb4
inet.af/netstack/tcpip/transport/udp.(*Forwarder).HandlePacket(...)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/transport/udp/forwarder.go:47
tailscale.com/wgengine/netstack.(*Impl).wrapProtoHandler.func1({0x35, {0x236fbcc, 0x4}, 0x9cfa, {0x236fbc8, 0x4}}, 0x206b950)
        tailscale.com@v1.16.2/wgengine/netstack/netstack.go:153 +0x1b8
inet.af/netstack/tcpip/stack.(*nic).DeliverTransportPacket(0x20641e0, 0x11, 0x206b950)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/stack/nic.go:826 +0x370
inet.af/netstack/tcpip/network/ipv4.(*endpoint).handleValidatedPacket(0x20c0800, {0x23bb580, 0x14, 0x40}, 0x206b950, {0x0, 0x0})
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/network/ipv4/ipv4.go:1046 +0x102c
inet.af/netstack/tcpip/network/ipv4.(*endpoint).HandlePacket(0x20c0800, 0x206b950)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/network/ipv4/ipv4.go:836 +0x394
inet.af/netstack/tcpip/stack.(*nic).DeliverNetworkPacket(0x20641e0, {0x0, 0x0}, {0x0, 0x0}, 0x800, 0x206b950)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/stack/nic.go:749 +0x2dc
inet.af/netstack/tcpip/link/channel.(*Endpoint).InjectLinkAddr(...)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/link/channel/channel.go:193
inet.af/netstack/tcpip/link/channel.(*Endpoint).InjectInbound(...)
        inet.af/netstack@v0.0.0-20210622165351-29b14ebc044e/tcpip/link/channel/channel.go:188
tailscale.com/wgengine/netstack.(*Impl).injectInbound(0x20d3b00, 0x2390e60, 0x20ea000)
        tailscale.com@v1.16.2/wgengine/netstack/netstack.go:470 +0x260
tailscale.com/net/tstun.(*Wrapper).filterIn(0x20ea000, {0x251e010, 0x3e, 0xffef})
        tailscale.com@v1.16.2/net/tstun/wrap.go:587 +0xa24
tailscale.com/net/tstun.(*Wrapper).Write(0x20ea000, {0x251e000, 0x4e, 0xffff}, 0x10)
        tailscale.com@v1.16.2/net/tstun/wrap.go:599 +0x94
golang.zx2c4.com/wireguard/device.(*Peer).RoutineSequentialReceiver(0x217b080)
        golang.zx2c4.com/wireguard@v0.0.0-20210905140043-2ef39d47540c/device/receive.go:477 +0x644
created by golang.zx2c4.com/wireguard/device.(*Peer).Start
        golang.zx2c4.com/wireguard@v0.0.0-20210905140043-2ef39d47540c/device/peer.go:202 +0x3b8

1.16.2
  tailscale commit: f449111792f662c8552514cf3691c3c1bd7f38e4
  other commit: 3853a3b920c7ec33da2b8ba8454e9ca8b3772774
  go version: go1.17.1-ts097d1284f9

and

1.17.169
  tailscale commit: 8d14bc32d182f8951ff080357a4838aff5b2bdef
  other commit: 6536d93f9d38d1aa0a74e015b5c2f9566f56dbac
  go version: go1.17.2-ts7037d3ea51

MIPS. I was just testing doesn't actually affect anyone.

Originally posted by @compliment in #2129 (comment)

[bradfitz]

https://github.com/inetaf/netstack/blob/main/atomicbitops/aligned_64bit.go (from that stack trace) has build tags:

// +build !arm,!mips,!386

So I guess @compliment is using mipsle,

$ go tool dist list | grep ^linux/mips
linux/mips
linux/mips64
linux/mips64le
linux/mipsle

... which it looks like the netstack folk forgot in their build tags.

/cc @josharian

[bradfitz]

(And we do have a fair number of linux/mipsle users from Unifi equipment etc)

@compliment, which hardware device was this out of curiosity? And were you explicitly trying to run it in netstack mode or did it just fall back to that after TUN failed?

[compliment]

@bradfitz It's this one. https://wikidevi.wi-cat.ru/Xiaomi_MiWiFi_Mini
and yes, I was trying to run in userspace-networking mode, I didn't try connecting to it with tun (it doesn't segfaults until you try connecting to it).

[bradfitz]

Filed google/gvisor#6804

[bradfitz]

Easy to repro with:

// tstest/cmd/atomictest/atomictest.go ...

package main

import (
        "fmt"
        "runtime"

        "inet.af/netstack/atomicbitops"
)

type T struct {
        A atomicbitops.AlignedAtomicInt64
        x int32
        B atomicbitops.AlignedAtomicInt64
}

func main() {
        fmt.Printf("I am %v/%v\n", runtime.GOOS, runtime.GOARCH)
        var t T
        t.A.Store(1)
        t.B.Store(2)
        fmt.Printf("got %v, %v\n", t.A.Load(), t.B.Load())
}

Then:

bradfitz@tsdev:~/src/tailscale.com$ GOARCH=mipsle go run --exec=qemu-mipsel ./tstest/cmd/atomictest/
I am linux/mipsle
panic: unaligned 64-bit atomic operation

goroutine 1 [running]:
runtime/internal/atomic.panicUnaligned(...)
        /home/bradfitz/sdk/go1.17/src/runtime/internal/atomic/unaligned.go:8
runtime/internal/atomic.lockAndCheck(...)
        /home/bradfitz/sdk/go1.17/src/runtime/internal/atomic/atomic_mipsx.go:38
runtime/internal/atomic.Store64(0x53e00c, 0x2)
        /home/bradfitz/sdk/go1.17/src/runtime/internal/atomic/atomic_mipsx.go:104 +0x94
inet.af/netstack/atomicbitops.(*AlignedAtomicInt64).Store(...)
        /home/bradfitz/go/pkg/mod/inet.af/netstack@v0.0.0-20211027215559-ec21145de76b/atomicbitops/aligned_64bit.go:39
main.main()
        /home/bradfitz/src/tailscale.com/tstest/cmd/atomictest/atomictest.go:20 +0x10c
exit status 2

I'll add some basic qemu-user tests in the tailscale.com repo & add the right apt-get deps on our CI so they're able to run (with all the qemu-* binaries) to catch regressions. Then I confidently just fix it in our netstack fork and trust that we won't regress if we pull from upstream gvisor in the future and google/gvisor#6804 hasn't yet been fixed.