Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: use Cloud Run #31834

Open
bradfitz opened this issue May 4, 2019 · 2 comments
Open

x/build: use Cloud Run #31834

bradfitz opened this issue May 4, 2019 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

bradfitz commented May 4, 2019

There are a lot of interesting things we could do with Cloud Run for the Go build system.

The make.bash step would be great, but at Cloud Run's 1 vCPU, that takes 4.5 minutes. If they add support for cranking that up, we plateau pretty flatly around 4 vCPUs (1m45s). It doesn't get much better than 1m15s even with 96 CPUs. So 4 vCPUs would be great. But that's not available.

But we could also wrap each make.bash tool invocation (compile, link, etc) with a toolexec wrapper, and run each step remotely as its own Cloud Run call. We'd need to inject a VFS into the remote call with some private syscall and/or os package hooks, as Cloud Run doesn't permit FUSE or NFS or anything in the kernel. (It's gVisor)

So Cloud Run for make.bash naively is too slow today with its 1 vCPU (4.5 minutes) and doing make.bash quickly on Cloud Run today requires some time consuming (but straight forward) work for the VFS and wrappers.

But tests today are easy...

We can do make.bash elsewhere, and then use Cloud Run to massively shard the test execution.

With a make.bash snapshot,

  • first run go tool dist test --list to find all the tests
  • then run each of those tests as its own Cloud Run call
  • but for standard package "testing" tests, shard further... go test -list . unicode/utf8 to get all the TestFoo names, then:
  • separate call for each utf8.test -short -vet=off -run=^TestEncodeRune$, etc.

Cloud Run's default quota (before asking for it to be raised) permits 1,000 instances. We'd need to lock down concurrency to 1 call per service (probably), but 1,000 test executions in parallel SGTM.

I hacked up a super rough prototype today that's promising.

/cc @dmitshur
@bradfitz bradfitz self-assigned this May 4, 2019
@gopherbot gopherbot added this to the Unreleased milestone May 4, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label May 4, 2019
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/175143 mentions this issue: all: add start of experimental support for using Cloud Run

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/175297 mentions this issue: cmd/dist: allow builders to control granularity of test/ directory sharding

gopherbot pushed a commit that referenced this issue May 6, 2019
@bradfitz
cmd/dist: allow builders to control granularity of test/ directory sh…
6b1ac82 
…arding

Even with 10 shards on builders, it still takes about ~2.5 minutes per
shard (and getting slower all the time as the test/ directory grows).
I'm currently experimenting with massively sharding out testing on
Cloud Run (each dist test & normal TestFoo func all running in
parallel), and in such a setup, 2.5 minutes is an eternity. I'd like
to increase that dist test's sharding from 10 to more like 1,000.

Updates #31834

Change-Id: I8b02989727793b5b5b2013d67e1eb01ef4786e28
Reviewed-on: https://go-review.googlesource.com/c/go/+/175297
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@bradfitz bradfitz removed their assignment May 3, 2020
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @dmitshur @gopherbot