gvisor panic: Invalid MmapLayout

[InhexSTER]

Description

panic: Invalid MmapLayout: {MinAddr:32768 MaxAddr:545460846592 BottomUpBase:11351179231232 TopDownBase:18446733448603492352 DefaultDirection:1 MaxStackRand:35184372088832}

goroutine 1 [running]:
panic({0xb242e0?, 0x4000199f30?})
GOROOT/src/runtime/panic.go:1017 +0x3cc fp=0x4000129a90 sp=0x40001299e0 pc=0x49d0c
gvisor.dev/gvisor/pkg/sentry/arch.(*Context64).NewMmapLayout(0x150?, 0xc0e100?, 0x0?, 0x16000000100e718?)
pkg/sentry/arch/arch_arm64.go:249 +0x2cc fp=0x4000129ba0 sp=0x4000129a90 pc=0x2c5d0c
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).SetMmapLayout(0x400037e000, 0xb26460?, 0xf73620?)
pkg/sentry/mm/lifecycle.go:49 +0x78 fp=0x4000129c80 sp=0x4000129ba0 pc=0x3f3228

Steps to reproduce

No response

runsc version

runsc version release-20231030.0
spec: 1.1.0-rc.1

docker version (if using docker)

No response

uname

aarch64

kubectl (if using Kubernetes)

No response

repo state (if built from source)

No response

runsc debug logs (if available)

No response

[milantracy]

Can you list how to reproduce it?

[ayushr2]

cc @avagin

Looking at Context64.NewMmapLayout(), it seems like this can happen when NewMmapLayout() is called with a max such that (max/3)*2 < maxMmapRand64 where maxMmapRand64 = (1 << 28) * pagesize. Something like that. We need to rescale maxMmapRand64 if max is not large enough?

You can repro it with something like:

	ctx := Context64{}
	l := limits.NewLimitSet()
	ctx.NewMmapLayout(32768, 545460846592, l)

[InhexSTER]

1. Installed on Orange Pi 5 Plus 16GB, running Ubuntu and k3s (worker)

  set -e
  ARCH=$(uname -m)
  URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}
  wget ${URL}/runsc ${URL}/runsc.sha512 \
    ${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512
  sha512sum -c runsc.sha512 \
    -c containerd-shim-runsc-v1.sha512
  rm -f *.sha512
  chmod a+rx runsc containerd-shim-runsc-v1
  sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
)

2. Added following to /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

Restarted the machine.

3. Tried running in k3s using RuntimeClass. Got deadline exceeded error during create sandbox in the k8s events.

4. Tried running the following sample, similar error

 SANDBOX_ID=$(sudo crictl runp --runtime runsc sandbox.json) 

Finally found /tmp/gvisor-panic.log with MmapLyaout panic

[milantracy]

it looks similiar to #9303 iiuc

you could try #9303 (comment) or #9303 (comment)

it will be helpful if you could share your strace log and debug log

[avagin]

You have to recompile a kernel with CONFIG_ARM64_VA_BITS_48. We don't support CONFIG_ARM64_VA_BITS_39.

[InhexSTER]

Thanks for suggestion. Looks like that is it most likely

zgrep -E "CONFIG_ARM64_VA_BITS_" /proc/config.gz
CONFIG_ARM64_VA_BITS_39=y
# CONFIG_ARM64_VA_BITS_48 is not set

I might give re-compiling the kernel with the 48 bits a try. In the meantime, amd64 version works fine for me.
Would be good to have better error handling in this scenario.

[milantracy]

to wrap up, gVisor doesn't support < 3 layer paging table on kvm platform when running on ARM64.

The commit at a1be003 adds the debug info that would require users to recompile their kernel with CONFIG_ARM64_VA_BITS_48 or more.