Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/playground: use gVisor on linux/amd64 binaries instead of NaCl for sandboxing #25224

Open
andybons opened this issue May 2, 2018 · 8 comments

Comments

@andybons
Copy link
Member

commented May 2, 2018

Native Client deprecation has been announced in favor of Web Assembly. It's unclear what that means in terms of NaCl's future development and support.

I'd like us to investigate using gVisor as our sandbox mechanism for the playground. This is not a statement that we should definitively switch over to gVisor.

https://github.com/google/gvisor

@ysmolsky

@gopherbot gopherbot added this to the Unreleased milestone May 2, 2018
@andybons

This comment has been minimized.

Copy link
Member Author

commented Aug 25, 2018

This has come up again as attempting to upgrade the playground to 1.11 results in the following breakage during the NaCl time patching phase:

 ---> ed8ffc3e1836
Step 20/63 : RUN patch -p1 -d /usr/local/go </usr/local/playground/strict-time.patch
 ---> Running in 1f97414872b7
patching file src/runtime/os_nacl.go
Reversed (or previously applied) patch detected!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/runtime/os_nacl.go.rej
patching file src/runtime/sys_nacl_amd64p32.s
Reversed (or previously applied) patch detected!  Assume -R? [n] 
Apply anyway? [n] 
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file src/runtime/sys_nacl_amd64p32.s.rej
The command '/bin/sh -c patch -p1 -d /usr/local/go </usr/local/playground/strict-time.patch' returned a non-zero code: 1

/cc @bradfitz

@bradfitz

This comment has been minimized.

Copy link
Member

commented Aug 25, 2018

That's @bcmills's patch. The old patch (enable-fake-time.patch) was designed to basically never bit rot but the new patch seems to have more context that makes it fragile. We should really upstream this and have it behind a bool or build tag so it's less likely to rot.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 13, 2019

I've been playing around with this. I'm deciding between:

  1. continuing to use App Engine Flex and launching gvisor's runsc "directly" (if one could consider the OCI interface direct). Unfortunately runsc still requires root today (for now: google/gvisor#311) and App Engine Flex doesn't give you root at serving time (only for debugging). So doing this would require modifying runsc to not require root.

  2. migrating off App Engine Flex and just using Docker with vanilla runsc. We'd need to change how we serve a bit. Probably an HTTP load balancer to a GCE VM instance group (perhaps running Container-Optimized OS, which already runs docker, and supports declarative yaml configs of which containers to run, including I believe privileged ones)

I think I'm leaning towards (2).

As part of this, we'd migrate to (2) well before the Go 1.14 release (which drops nacl), so the release process doesn't involve changing playground machinery during a release. We'd still support Go 1.13 nacl mode with the new infrastructure, and then the Go 1.14 release would just be changing a single line in its Dockerfile.

/cc @dmitshur @andybons @toothrot @prattmic

@toothrot

This comment has been minimized.

Copy link
Contributor

commented Sep 16, 2019

@bradfitz Have you investigated using Cloud Run? The documentation seems to suggest that the runtime user is root: https://cloud.google.com/run/docs/tips#container-security

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 16, 2019

@toothrot, Cloud Run is already under gVisor itself. (and it doesn't let us tweak its sandbox config to be more restrictive, as we'd need). Cloud Run gives you root, but fake-ish constrained root. We definitely couldn't use gVisor's KVM mode ("platform") under it. We could probably just the ptrace mode (ptrace syscall is at least partially supported: https://gvisor.dev/docs/user_guide/compatibility/linux/amd64/). But once runsc went to do the other root-requiring stuff (setting up namespaces, etc), I suspect we'd trip over unimplemented system calls. e.g. I see on that system call list:

Mount namespace (CLONE_NEWNS) not supported. Options CLONE_PARENT, CLONE_SYSVSEM not supported.

So basically, to use Cloud Run we'd need to do the same work hacking up runsc to run rootless that we'd have to do to run it on its current App Engine Flex environment.

That said, we should move many of our things to Cloud Run regardless, but I don't think it'd help us here.

@prattmic can correct me if indeed the ptrace mode of runsc would work unmodified under gvisor. That would be a happy surprise.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 16, 2019

@prattmic confirms that nested gVisor doesn't work yet.

@bradfitz bradfitz changed the title x/playground: investigate using gVisor instead of NaCl for sandboxing x/playground: use gVisor on linux/amd64 binaries instead of NaCl for sandboxing Sep 16, 2019
@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 17, 2019

In case we do end up going the path of using some modified root-less runsc directly, without docker, this syzkaller code drives runsc directly:

https://github.com/google/syzkaller/blob/master/vm/gvisor/gvisor.go

@gopherbot

This comment has been minimized.

Copy link

commented Sep 17, 2019

Change https://golang.org/cl/195983 mentions this issue: sandbox: add gvisor runsc-based sandbox

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.