x/build: sharded iOS builders #23824
Comments
|
This issue is about Android, but in theory the same could be done for the iOS simulator. |
|
This https://paulemtz.blogspot.dk/2013/05/android-testing-in-headless-emulator.html seems to indicate that the standard emulator supports headless mode. |
|
Sure, prepare a Docker image that lets us do the traditional make.bash, then snapshot, then clone to several mirrors, then run tests one-at-a-time. It'll require some tweaking of the build system probably, but I'll do those parts if you do the bulk of the work in preparing the Dockerfile.
headless helps but isn't required. Worst case we could run a headless X server so it thinks it has a window. |
|
The following Dockerfile And supporting run-emulator.sh (as well as a Go checkout in Can be started with:
Where I'm able to complete androidtest.sh with android/386 and android/amd64: It can even complete the full test suite of gomobile: Let me know how to proceed. |
|
Ping @bradfitz |
|
Sorry, I realistically won't be able to get to start on this until April 1st. I'm preparing for an upcoming Gophercon talk this weekend, traveling next week for the talk, and then on vacation the last week of March. I might be able to work on this next week from Moscow, but no promises. @andybons has been wanting to learn how this whole system works more, so this might be a good "starter project" for adding a builder. BTW, I forgot to ask earlier: how long does this all take? I imagine sharding is necessary for being a trybot, but is your goal for it to be a trybot right away, or just to have it run on GCE by us in Kubernetes? If it's easier, we could skip sharding for the first version. But I'm also fine with sharding for the first version, but curious about timing. I imagine make.bash is still fast like always (1 minute or less), but then how long are the various tests? |
No problem at all. My ping was just as much to see if the Docker image was enough.
My wishlist ordered by importance:
(And in the future: 4. Do 1-3 for a darwin builder running an 386/amd64 iOS emulator.) In other words, "just" a regular non-sharded builder on GCE would make me very grateful.
On my i7-6700K@4GHz with 32GB memory with a running Android 8 x86_64 emulator, ./androidtest.bash takes ~2min and ./all.bash on the same machine takes ~3min. BTW, I'm not even sure trybots can be stable enough before #23795 is fixed. You wouldn't happen to know someone to ping for the corresponding adb bug, https://issuetracker.google.com/issues/73230216 ? |
|
I just stumpled on #9579. Will the Android emulator run on GCE? |
|
@bcmills, this is another builder request. |
|
Gentle ping. If I can do anything to help this, please let me know. |
|
I've been focused on vgo reviews and bugs (since that's fairly urgent for keeping 1.11 on schedule). I plan to get back to this after the 1.11 release, but if anybody else would like to take it before then you're welcome to it. |
|
Change https://golang.org/cl/162959 mentions this issue: |
…ting This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
|
Change https://golang.org/cl/163057 mentions this issue: |
The COS image I'd forked from earlier didn't have CONFIG_KVM or CONFIG_KVM_INTEL enabled in its kernel, so even though I'd enabled the VMX license bit for the VM, the kernel was unable to use it. Now I've instead rebuilt the ChromiumOS "lakitu" board with a modified kernel config: https://cloud.google.com/container-optimized-os/docs/how-to/building-from-open-source More docs later. Still tinkering. Nothing uses this yet. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Id2839066e67d9ddda939d96c5f4287af3267a769 Reviewed-on: https://go-review.googlesource.com/c/163057 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
|
Change https://golang.org/cl/163301 mentions this issue: |
…d OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
|
@eliasnaur, this is ready from my side if you want to give me a Docker image that uses the Android SDK + emulator (which can now use KVM on our buildlets) to run tests. |
|
Is the one from #23824 (comment) good enough? |
|
Oh right. Forgot about that. I'll try. |
|
|
I updated the Dockerfile and run-emulator.sh script, please try again. Changes:
I've created CL 163377 to fix a gomobile test failure with jdk8, and CL 163378 so the Edit: The two CLs are in, all gomobile tests pass and the gomobile init step is no longer necessary. Thank you for working on this. |
|
I'm running it locally as a test. I notice that when androidtest.bash is still doing the make.bash phase, the Android qemu process is successfully using KVM (great) but is also using 200% CPU and I haven't even started sending it tests yet because make.bash is still running. What is the emulator doing early on? Animating some live wallpaper or something? Anything we can do to reduce unnecessary CPU usage to make the build & tests faster? |
|
I'm not sure. There is an emulator flag, |
Are we loopback mounting anything, though? In any case, I can get it down to 36 seconds with: Haven't yet tried to docker commit the first step, though. |
The androidtest.bash script encodes the additional steps to build Go and run tests on Android. In order to add sharded builders and trybots, Android needs to fit into the usual make.bash + cmd/dist test pattern. This change moves building the exec wrapper into cmd/dist bootstrap. Do the same for iOS while we're here. Updates #23824 Change-Id: I58a1b0679c3a6c92fdc7fff464b469641f1fee74 Reviewed-on: https://go-review.googlesource.com/c/163618 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
To run the standard library tests on Android, the androidtest.bash script copies GOROOT to the device. Move that logic to the android exec wrapper, thereby making androidtest.bash obsolete. Apart from making Android less special, the sharded builder infrastructure should now be able to run (emulated) Android builders and trybots without special treatment. Updates #23824 Change-Id: I41591fea9a15b38c6dcf84046ea57f1e9165eaa5 Reviewed-on: https://go-review.googlesource.com/c/163619 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Android emulator builders are soon to join the trybot set. To avoid flaky runs, work around a longstanding adb bug where concurrent adb commands sometimes fail. I haven't seen the problem on actual devices until recently. It seems that the recently added "adb wait-for-device" can introduce flakyness with errors such as: adb: error: failed to get feature set: protocol fault (couldn't read status): Connection reset by peer Instead of working around that, give up and serialize use of adb everywhere. Fixes #23795 Updates #23824 Change-Id: If347c9981fa32ff8a1e14b7454f122ef682450a6 Reviewed-on: https://go-review.googlesource.com/c/163625 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
Getting closer. I almost have all the pieces done. Testing it all out now, Few little config things missing it appears. |
|
Works now. |
Now that we can do nested virtualization on GCE, that means we can run the Android emulator (which requires KVM) on GCE and at least get fast trybots and such for android-amd64. Updates golang/go#23824 Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d Reviewed-on: https://go-review.googlesource.com/c/163738 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
Change https://golang.org/cl/164380 mentions this issue: |
Updates golang/go#23824 Change-Id: I24b590dab6719d014472da3d7850fc9a1b3ef4cc Reviewed-on: https://go-review.googlesource.com/c/164380 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
Change https://golang.org/cl/164478 mentions this issue: |
Updates golang/go#23824 Change-Id: Iee70bcbfef6a271fdd26694d848a703a200dc0c0 Reviewed-on: https://go-review.googlesource.com/c/164478 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
|
Now that Android builders are on GCE and soon joining the trybot set (thank you Brad!), I wonder how far we can get with iOS. @bradfitz can we have Darwin hosts in the cloud with Xcode installed running iOS tests on the iPhone simulator? Even without sharding and trybots, having cloud builders is a major step up from the very slow (and impossible to gomote) device builders we have now. |
|
Change https://golang.org/cl/165797 mentions this issue: |
Android tests are built on the host and run on the device. To do that, the exec wrapper copies the test binary and testdata to the device. To enable many more tests, make the copied environment more like the host: - Copy all of pkg from GOROOT, not just the android pkg directory. - Copy any parent testdata directories as well as the package's own. - Copy *.go files from the package directory. This enables misc/cgo/stdio and misc/cgo/life tests that were invisible before so disable them explicitly. - Always copy the GOROOT, even for tests outside GOROOT. This is expensive but only done once per make.bash. - Build the go tool for the device and put it in PATH. Set GOCACHE to a writable directory and disable cgo. While here, use a single directory for all the exec wrapper files and delete that once per make.bash as well. In total, this CL enables many tests in the subrepos that would need skips without it, in particular the x/tools tests. Fixes #11452 Updates #23824 Updates #11811 Change-Id: I2e50d8b57db9bc4637f25272a5360c8b2cf4e627 Reviewed-on: https://go-review.googlesource.com/c/go/+/165797 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
Change https://golang.org/cl/166397 mentions this issue: |
The android emulator builders is running out of space after CL 165797 copied most of GOROOT to the device. The pkg directory is by far the largest, so only include what seems necessary to build the x/ repositories: pkg/android_$GOARCH and pkg/tool/android_$GOARCH. While here, rename the device root directory to match the exec wrapper name and make sure the deferred cleanups actually run before os.Exit. Hopefully fixes the emulator builders. Updates #23824 Change-Id: I4d1e3ab2c89fd1e5818503d323ddb87f073094da Reviewed-on: https://go-review.googlesource.com/c/go/+/166397 Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
Change https://golang.org/cl/172798 mentions this issue: |
Updates golang/go#23824 Change-Id: I6386bc267c35856858b3e46d0998fb60e3406e02 Reviewed-on: https://go-review.googlesource.com/c/build/+/172798 Reviewed-by: Elias Naur <mail@eliasnaur.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
Change https://golang.org/cl/174079 mentions this issue: |
Missing from CL 172798 Updates golang/go#23824 Change-Id: I8f1b09ce8f52b555162b344ceddfbfe88e19ad4e Reviewed-on: https://go-review.googlesource.com/c/build/+/174079 Reviewed-by: Elias Naur <mail@eliasnaur.com> Run-TryBot: Elias Naur <mail@eliasnaur.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
|
@eliasnaur, I don't know what remains here (is it done? iOS stuff maybe?) so reassigning to you for now. |
eliasnaur
changed the title
|
iOS builders, yes. You mentioned that your macstadium VMs might be improving enough for me to use them for iOS simulator builds. Now that we have Corellium virtual arm64 devices, iOS simulators are probably only interesting if we can have enough simulators to run trybots. |
|
@bradfitz @eliasnaur Do you know if there are there known limitations on the I can't seem to ssh into a gomote instance, but maybe it's just not supported: Asking because there was a trybot failure that happens only on this trybot configuration, not others in CL 196321. /cc @ianthehat |
Currently, the android/amd64 and android/386 builders run on an Android emulator with a amd64 system image. This is taxing the heavyly loaded mobile builder Mac Mini. It is also inefficient: the emulator builds competes with Android device builds and any concurrent iOS builds. Because builds are slow, android is not in the trybot set, leaving me to often pester CL authors with "this CL broke android".
Is it possible to add (and run) the Android emulator inside the existing docker images used for regular builders and then run android sharded tests as any other GOOS? If so, android/amd64 and android/386 builds would complete much faster, and take some pressure off the mobile device builder.
If the emulator builds are stable enough, android/amd64 (and perhaps android/386) could even join the trybot set, avoiding most (if not all) android specific followup work after a CL is submitted.
The text was updated successfully, but these errors were encountered: