Sandbox failed (error starting container: exit status 125)

[tbiens]

Hello, I saw the blog post about this project. I'm excited to give it a try. Completely fresh ubuntu 22 lts server install vm. Docker.io is installed and functioning. golang and libpcap-dev installed.

$ mkdir /tmp/results
$ docker run --privileged -ti
-v /tmp/results:/results
gcr.io/ossf-malware-analysis/analysis analyze
-package Django -ecosystem pypi
-upload file:///results/

https://imgur.com/a/Y578M0W

running with sudo vs root doesn't seem to make any significant difference. Thanks for the help!

[sycophantic]

I have the same issue with Ubuntu 20.04 and Docker version 20.10.14, build a224086.

docker run --privileged -ti -v /tmp/results:/results gcr.io/ossf-malware-analysis/analysis analyze -package Django -ecosystem pypi -upload file:///results/ 2022-05-03T16:12:47.560Z INFO analyze/main.go:55 Got request {"ecosystem": "pypi", "name": "Django", "localPath": "", "version": ""} 2022-05-03T16:12:47.648Z INFO analyze/main.go:79 Got request {"ecosystem": "pypi", "name": "Django", "version": ""} 2022-05-03T16:12:47.648Z INFO analysis/analysis.go:90 Running analysis {"args": ["/usr/local/bin/analyze.py", "--version", "4.0.4", "install", "django"]} 2022-05-03T16:12:47.648Z DEBUG analysis/analysis.go:93 Preparing packet capture 2022-05-03T16:12:47.670Z DEBUG analysis/analysis.go:104 Running the command {"args": ["/usr/local/bin/analyze.py", "--version", "4.0.4", "install", "django"]} 2022-05-03T16:12:47.670Z DEBUG sandbox/sandbox.go:205 podman {"args": ["--cgroup-manager=cgroupfs", "--events-backend=file", "network", "inspect", "podman", "-f", "{{range .plugins}}{{with .ipam.subnet}}{{.}}{{end}}{{end}}"]} 2022-05-03T16:12:47.781Z DEBUG sandbox/sandbox.go:205 podman {"args": ["--cgroup-manager=cgroupfs", "--events-backend=file", "pull", "gcr.io/ossf-malware-analysis/python:latest"]} 2022-05-03T16:12:54.193Z DEBUG sandbox/sandbox.go:205 podman {"args": ["--cgroup-manager=cgroupfs", "--events-backend=file", "image", "prune", "-f"]} 2022-05-03T16:12:54.237Z DEBUG sandbox/sandbox.go:205 podman {"args": ["--cgroup-manager=cgroupfs", "--events-backend=file", "create", "--runtime=/usr/local/bin/runsc_compat.sh", "--init", "--dns=8.8.8.8", "--dns=8.8.4.4", "--dns-search=.", "gcr.io/ossf-malware-analysis/python:latest"]} 2022-05-03T16:12:54.322Z DEBUG sandbox/sandbox.go:205 podman {"args": ["--cgroup-manager=cgroupfs", "--events-backend=file", "start", "--runtime=/usr/local/bin/runsc_compat.sh", "--runtime-flag=root=/var/run/runsc", "--runtime-flag=debug-log=/tmp/sandbox_logs_2222537600/runsc.log.%COMMAND%", "--runtime-flag=net-raw", "--runtime-flag=strace", "--runtime-flag=log-packets", "d672a65893df44cb7d31ab35ae7db60bda78521c004a89a195b26c9428410d85"]} 2022-05-03T16:12:54.373Z WARN log/writer.go:63 time="2022-05-03T16:12:54Z" level=warning msg="Couldn't run auplink before unmount /var/lib/containers/storage/aufs/mnt/b826187222f6e8b7437ef3e4faae1b79699283827ae3ed89fa391be4740347c9: exec: \"auplink\": executable file not found in $PATH" {"args": ["/usr/local/bin/analyze.py", "--version", "4.0.4", "install", "django"]} github.com/ossf/package-analysis/internal/log.WriteTo /src/internal/log/writer.go:63 github.com/ossf/package-analysis/internal/log.Writer.func1 /src/internal/log/writer.go:40 2022-05-03T16:12:54.660Z WARN log/writer.go:63 Error: unable to start container "d672a65893df44cb7d31ab35ae7db60bda78521c004a89a195b26c9428410d85": error mounting storage for container d672a65893df44cb7d31ab35ae7db60bda78521c004a89a195b26c9428410d85: error creating aufs mount to /var/lib/containers/storage/aufs/mnt/b826187222f6e8b7437ef3e4faae1b79699283827ae3ed89fa391be4740347c9: invalid argument {"args": ["/usr/local/bin/analyze.py", "--version", "4.0.4", "install", "django"]} github.com/ossf/package-analysis/internal/log.WriteTo /src/internal/log/writer.go:63 github.com/ossf/package-analysis/internal/log.Writer.func1 /src/internal/log/writer.go:40 2022-05-03T16:12:55.761Z FATAL analyze/main.go:104 Analysis phase failed {"phase": "install", "error": "sandbox failed (error starting container: exit status 125)"} main.main /src/cmd/analyze/main.go:104 runtime.main /usr/local/go/src/runtime/proc.go:225

[calebbrown]

Thanks for the report and the error message. I'll take a look.

[calebbrown]

The issue is that running a container inside a container requires the right filesystem.

[calebbrown]

I got it running by adding: -v /var/lib/containers:/var/lib/containers to map the location Podman stores it's containers to a filesystem outside of the Docker overlayfs.

Internet searches also suggest changing Docker's storage filesystem can also work around the issue.

I will update the docs.

[calebbrown]

Can you please try again with the updated command.

Feel free to re-open this issue if it doesn't work.

[tbiens]

I gave it a try adding var lib containers. No change in result.

[calebbrown]

I suspect the issue may be GVisor currently doesn't support cgroups v2 and has to use v1 instead (see: google/gvisor#3481)

This is set during boot.

I'm not sure what boot loader your Linux system uses, but for Grub you can do this:

$ echo 'GRUB_CMDLINE_LINUX+=" systemd.unified_cgroup_hierarchy=0"' | sudo tee /etc/default/grub.d/cgroup.cfg
$ sudo update-grub
$ sudo reboot

[calebbrown]

I'll also check if a more recent version of GVisor is available

[tbiens]

I am not familiar with gvisor or cgroups. update-grub doesn't like that line.

I installed Ubuntu 22.04 LTS Server:
apt install golang libpcap-dev docker.io
systemctl start docker
systemctl enable docker

Then heading right into the command that fails.

[calebbrown]

Please try again! I've updated the GVisor version and refreshed the container images.

You might need to rm the analysis image from docker to get the new one (or add :latest to it)

[tbiens]

Did a docker system prune -a

No improvement.

[oliverchang]

Thanks for your patience @tbiens. I can reproduce this with cgroups v2 enabled. We'll take a closer look at this next week and hopefully resolve this.

[tbiens]

No problem, have a great weekend!

[oliverchang]

@tbiens I've managed to get this working locally with #276.

Could you please try this? (Just add --cgroupns=host to your docker run command).

[tbiens]

That did the job, output is scrolling well.

Great job Oliver!