x/build: sharded iOS builders

[eliasnaur]

Currently, the android/amd64 and android/386 builders run on an Android emulator with a amd64 system image. This is taxing the heavyly loaded mobile builder Mac Mini. It is also inefficient: the emulator builds competes with Android device builds and any concurrent iOS builds. Because builds are slow, android is not in the trybot set, leaving me to often pester CL authors with "this CL broke android".

Is it possible to add (and run) the Android emulator inside the existing docker images used for regular builders and then run android sharded tests as any other GOOS? If so, android/amd64 and android/386 builds would complete much faster, and take some pressure off the mobile device builder.

If the emulator builds are stable enough, android/amd64 (and perhaps android/386) could even join the trybot set, avoiding most (if not all) android specific followup work after a CL is submitted.

[eliasnaur]

This issue is about Android, but in theory the same could be done for the iOS simulator.

[ianlancetaylor]

CC @bradfitz @andybons

[eliasnaur]

This

https://paulemtz.blogspot.dk/2013/05/android-testing-in-headless-emulator.html

seems to indicate that the standard emulator supports headless mode.

[bradfitz]

Sure, prepare a Docker image that lets us do the traditional make.bash, then snapshot, then clone to several mirrors, then run tests one-at-a-time.

It'll require some tweaking of the build system probably, but I'll do those parts if you do the bulk of the work in preparing the Dockerfile.

seems to indicate that the standard emulator supports headless mode.

headless helps but isn't required. Worst case we could run a headless X server so it thinks it has a window.

[eliasnaur]

The following Dockerfile

# Copyright 2014 The Go Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.

FROM debian:sid
MAINTAINER golang-dev <golang-dev@googlegroups.com>

ENV DEBIAN_FRONTEND noninteractive

# gdb: optionally used by runtime tests for gdb
# strace: optionally used by some net/http tests
# gcc libc6-dev: for building Go's bootstrap 'dist' prog
# libc6-dev-i386 gcc-multilib: for 32-bit builds
# procps lsof psmisc: misc basic tools
# libgles2-mesa-dev libopenal-dev fonts-noto: required by x/mobile repo
# unzip openjdk-8-jdk python lib32z1: required by the Android SDK
RUN apt-get update && apt-get install -y \
	--no-install-recommends \
	ca-certificates \
	curl \
	gdb \
	strace \
	gcc \
	libc6-dev \
	libc6-dev-i386 \
	gcc-multilib \
	procps \
	lsof \
	psmisc \
	libgles2-mesa-dev \
	libopenal-dev \
	fonts-noto \
	fonts-noto-mono \
	openssh-server \
	unzip \
	openjdk-8-jdk \
	python \
	lib32z1 \
	&& rm -rf /var/lib/apt/lists/*

RUN mkdir -p /go1.4-amd64 \
	&& ( \
		curl --silent https://storage.googleapis.com/golang/go1.4.linux-amd64.tar.gz | tar -C /go1.4-amd64 -zxv \
	) \
	&& mv /go1.4-amd64/go /go1.4 \
	&& rm -rf /go1.4-amd64 \
	&& rm -rf /go1.4/pkg/linux_amd64_race \
		/go1.4/api \
		/go1.4/blog \
		/go1.4/doc \
		/go1.4/misc \
		/go1.4/test \
	&& find /go1.4 -type d -name testdata | xargs rm -rf

RUN curl -o /usr/local/bin/stage0 https://storage.googleapis.com/go-builder-data/buildlet-stage0.linux-amd64-kube \
	&& chmod +x /usr/local/bin/stage0

RUN mkdir -p /android/sdk \
	&& curl -o /android/sdk/sdk-tools-linux.zip https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip \
	&& unzip -d /android/sdk /android/sdk/sdk-tools-linux.zip \
	&& rm -rf /android/sdk/sdk-tools-linux.zip

RUN yes | /android/sdk/tools/bin/sdkmanager --licenses \
	&& /android/sdk/tools/bin/sdkmanager ndk-bundle "system-images;android-26;google_apis;x86_64" \
	&& /android/sdk/tools/bin/sdkmanager "build-tools;21.1.2" "platforms;android-26" \
	&& /android/sdk/tools/bin/sdkmanager --update

# Gradle for gomobile
RUN curl -L -o /android/gradle-5.2.1-bin.zip https://services.gradle.org/distributions/gradle-5.2.1-bin.zip \
	&& unzip -d /android /android/gradle-5.2.1-bin.zip \
	&& rm /android/gradle-5.2.1-bin.zip

# Cleanup
RUN rm -rf /android/sdk/sdk-tools-linux.zip \
	&& apt-get -y remove unzip \
	&& apt-get -y autoremove

COPY run-emulator.sh /android/run-emulator.sh
RUN chmod +x /android/run-emulator.sh

# Include a checkout of Go
COPY go-tip /go-tip

CMD ["/usr/local/bin/stage0"]

And supporting run-emulator.sh (as well as a Go checkout in go-tip)

#!/bin/sh

set -e

# create emulator
echo no | /android/sdk/tools/bin/avdmanager create avd --force --name android-avd --package "system-images;android-26;google_apis;x86_64"

# run the emulator.
/android/sdk/emulator/emulator @android-avd -no-window &

# wait for it.
/android/sdk/platform-tools/adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;'

Can be started with:

sudo docker run --privileged -it <image-name> /bin/bash

Where I'm able to complete androidtest.sh with android/386 and android/amd64:

/android/run-emulator.sh
cd /go-tip/src
apt-get update && apt-get install git-core
export PATH=$PATH:/android/sdk/platform-tools; GOROOT_BOOTSTRAP=/go1.4 CC_FOR_TARGET=/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/i686-linux-android26-clang GOARCH=386 ./androidtest.bash
export PATH=$PATH:/android/sdk/platform-tools; GOROOT_BOOTSTRAP=/go1.4 CC_FOR_TARGET=/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android26-clang GOARCH=amd64 ./androidtest.bash

It can even complete the full test suite of gomobile:

GOROOT_BOOTSTRAP=/go1.4 ./make.bash
export PATH=$PATH:/go-tip/bin:$HOME/go/bin:/android/sdk/platform-tools:/android/gradle-5.2.1/bin
export GOPATH=$HOME/go                                                                                                                  
export ANDROID_HOME=/android/sdk                                                                                                        
go get -u golang.org/x/mobile/...                                                                                                       
go test -short golang.org/x/mobile/...

Let me know how to proceed.

[eliasnaur]

Ping @bradfitz

[bradfitz]

Sorry, I realistically won't be able to get to start on this until April 1st. I'm preparing for an upcoming Gophercon talk this weekend, traveling next week for the talk, and then on vacation the last week of March.

I might be able to work on this next week from Moscow, but no promises.

@andybons has been wanting to learn how this whole system works more, so this might be a good "starter project" for adding a builder.

BTW, I forgot to ask earlier: how long does this all take? I imagine sharding is necessary for being a trybot, but is your goal for it to be a trybot right away, or just to have it run on GCE by us in Kubernetes? If it's easier, we could skip sharding for the first version.

But I'm also fine with sharding for the first version, but curious about timing. I imagine make.bash is still fast like always (1 minute or less), but then how long are the various tests?

[eliasnaur]

Sorry, I realistically won't be able to get to start on this until April 1st. I'm preparing for an upcoming Gophercon talk this weekend, traveling next week for the talk, and then on vacation the last week of March.

No problem at all. My ping was just as much to see if the Docker image was enough.

BTW, I forgot to ask earlier: how long does this all take? I imagine sharding is necessary for being a trybot, but is your goal for it to be a trybot right away, or just to have it run on GCE by us in Kubernetes? If it's easier, we could skip sharding for the first version.

My wishlist ordered by importance:

1. Run emulated android/386 and android/amd64 builder on GCE (non-sharded or sharded, whatever is convenient for you). That will take a good amount of load off the strained mac mini that runs all mobile builders today. Also, it will ensure a builder is running if the mini is down which happens from time to time.
2. Use the GCE builder to run the gomobile subrepository tests. There is no meaningful automatic testing for gomobile today, so this would be a great way to get some coverage there.
3. Android trybot builders, to catch any Android specific issues with CLs before they're submitted.

(And in the future: 4. Do 1-3 for a darwin builder running an 386/amd64 iOS emulator.)

In other words, "just" a regular non-sharded builder on GCE would make me very grateful.

But I'm also fine with sharding for the first version, but curious about timing. I imagine make.bash is still fast like always (1 minute or less), but then how long are the various tests?

On my i7-6700K@4GHz with 32GB memory with a running Android 8 x86_64 emulator, ./androidtest.bash takes ~2min and ./all.bash on the same machine takes ~3min.

BTW, I'm not even sure trybots can be stable enough before #23795 is fixed. You wouldn't happen to know someone to ping for the corresponding adb bug, https://issuetracker.google.com/issues/73230216 ?

[eliasnaur]

I just stumpled on #9579. Will the Android emulator run on GCE?

[bradfitz]

@bcmills, this is another builder request.

[eliasnaur]

Gentle ping. If I can do anything to help this, please let me know.

[bcmills]

I've been focused on vgo reviews and bugs (since that's fairly urgent for keeping 1.11 on schedule). I plan to get back to this after the 1.11 release, but if anybody else would like to take it before then you're welcome to it.

[gopherbot]

Change https://golang.org/cl/162959 mentions this issue: dashboard, buildlet: add a disabled builder with nested virt, for testing

[gopherbot]

Change https://golang.org/cl/163057 mentions this issue: buildlet: change image name for COS-with-vmx buildlet

[gopherbot]

Change https://golang.org/cl/163301 mentions this issue: env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx

[bradfitz]

@eliasnaur, this is ready from my side if you want to give me a Docker image that uses the Android SDK + emulator (which can now use KVM on our buildlets) to run tests.

[eliasnaur]

Is the one from #23824 (comment) good enough?

[bradfitz]

Oh right. Forgot about that. I'll try.

[bradfitz]

@eliasnaur,

Step 8/14 : RUN yes | /android/sdk/tools/bin/sdkmanager --licenses      && /android/sdk/tools/bin/sdkmanager ndk-bundle "system-images;android-26;google_apis;x86_64"       && /android/sdk/tools/bin/sdkmanager "build-tools;21.1.2" "platforms;android-19"        && /android/sdk/tools/bin/sdkmanager --update
 ---> Running in 1288386ab097
Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlSchema
        at com.android.repository.api.SchemaModule$SchemaModuleVersion.<init>(SchemaModule.java:156)
        at com.android.repository.api.SchemaModule.<init>(SchemaModule.java:75)
        at com.android.sdklib.repository.AndroidSdkHandler.<clinit>(AndroidSdkHandler.java:81)
        at com.android.sdklib.tool.SdkManagerCli.main(SdkManagerCli.java:117)
        at com.android.sdklib.tool.SdkManagerCli.main(SdkManagerCli.java:93)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlSchema
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 5 more
The command '/bin/sh -c yes | /android/sdk/tools/bin/sdkmanager --licenses      && /android/sdk/tools/bin/sdkmanager ndk-bundle "system-images;android-26;google_apis;x86_64"       && /android/sdk/tools/bin/sdkmanager "build-tools;21.1.2" "platforms;android-19"        && /android/sdk/tools/bin/sdkmanager --update' returned a non-zero code: 1

[eliasnaur]

I updated the Dockerfile and run-emulator.sh script, please try again. Changes:

• Replaced default-jdk with openjdk-8-jdk in Dockerfile to make sdkmanager run.
• Removed the prebuilding of the NDK toolchains from run-emulator.sh and updated the androidtest.bash instructions to use the builtin prebuilt toolchains introduced in NDK r19b.
• Updated Gradle to 5.2.1 in the Dockerfile since the old one was no longer supported by gomobile.

I've created CL 163377 to fix a gomobile test failure with jdk8, and CL 163378 so the gomobile init step can go away.

Edit: The two CLs are in, all gomobile tests pass and the gomobile init step is no longer necessary.

Thank you for working on this.

[bradfitz]

I'm running it locally as a test.

I notice that when androidtest.bash is still doing the make.bash phase, the Android qemu process is successfully using KVM (great) but is also using 200% CPU and I haven't even started sending it tests yet because make.bash is still running. What is the emulator doing early on? Animating some live wallpaper or something? Anything we can do to reduce unnecessary CPU usage to make the build & tests faster?

[eliasnaur]

I'm not sure. There is an emulator flag, -no-boot-anim that "disable animation for faster boot", but I assumed -no-window already effectively did that.

[bradfitz]

Getting closer. I almost have all the pieces done.

Testing it all out now,

FAIL    crypto/tls      0.002s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH
FAIL    crypto/x509     0.002s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH
FAIL    database/sql    0.006s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH
FAIL    database/sql/driver     0.002s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH
FAIL    debug/dwarf     0.002s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH

Few little config things missing it appears.

[bradfitz]

Works now.

bradfitz@gdev:~/src/golang.org/x/build$ debugnewvm --host=host-android-amd64-emu -run-build=android-amd64-emu
...
....
##### ../misc/cgo/testasan
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb shell mkdir -p /data/local/tmp/main-1465
go_android_exec: adb shell mkdir -p /data/local/tmp/goroot/misc/cgo/testasan
go_android_exec: adb push /workdir/tmp/go-build746649494/b001/exe/main /data/local/tmp/main-1465/main-1465
[  5%] /data/local/tmp/main-1465/main-1465
[ 10%] /data/local/tmp/main-1465/main-1465
[ 15%] /data/local/tmp/main-1465/main-1465
[ 21%] /data/local/tmp/main-1465/main-1465
[ 26%] /data/local/tmp/main-1465/main-1465
[ 31%] /data/local/tmp/main-1465/main-1465
[ 37%] /data/local/tmp/main-1465/main-1465
[ 42%] /data/local/tmp/main-1465/main-1465
[ 47%] /data/local/tmp/main-1465/main-1465
[ 53%] /data/local/tmp/main-1465/main-1465
[ 58%] /data/local/tmp/main-1465/main-1465
[ 63%] /data/local/tmp/main-1465/main-1465
[ 69%] /data/local/tmp/main-1465/main-1465
[ 74%] /data/local/tmp/main-1465/main-1465
[ 79%] /data/local/tmp/main-1465/main-1465
[ 85%] /data/local/tmp/main-1465/main-1465
[ 90%] /data/local/tmp/main-1465/main-1465
[ 95%] /data/local/tmp/main-1465/main-1465
[100%] /data/local/tmp/main-1465/main-1465
/workdir/tmp/go-build746649494/b001/exe/main: 1 file pushed. 77.5 MB/s (1232413 bytes in 0.015s)
go_android_exec: adb shell export TMPDIR="/data/local/tmp/main-1465"; export GOROOT="/data/local/tmp/goroot"; export GOPATH="/data/local/tmp/gopath"; cd "/data/local/tmp/goroot/misc/cgo/testasan"; '/data/local/tmp/main-1465
/main-1465' ; echo -n exitcode=$?
exitcode=0go_android_exec: adb shell rm -rf /data/local/tmp/main-1465

##### ../misc/cgo/testsigfwd
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb shell mkdir -p /data/local/tmp/main-1653
go_android_exec: adb shell mkdir -p /data/local/tmp/goroot/misc/cgo/testsigfwd
go_android_exec: adb push /workdir/tmp/go-build400298215/b001/exe/main /data/local/tmp/main-1653/main-1653
[  3%] /data/local/tmp/main-1653/main-1653
[  6%] /data/local/tmp/main-1653/main-1653
[  9%] /data/local/tmp/main-1653/main-1653
[ 12%] /data/local/tmp/main-1653/main-1653
[ 15%] /data/local/tmp/main-1653/main-1653
[ 18%] /data/local/tmp/main-1653/main-1653
[ 22%] /data/local/tmp/main-1653/main-1653
[ 25%] /data/local/tmp/main-1653/main-1653
[ 28%] /data/local/tmp/main-1653/main-1653
[ 31%] /data/local/tmp/main-1653/main-1653
[ 34%] /data/local/tmp/main-1653/main-1653
[ 37%] /data/local/tmp/main-1653/main-1653
[ 41%] /data/local/tmp/main-1653/main-1653
[ 44%] /data/local/tmp/main-1653/main-1653
[ 47%] /data/local/tmp/main-1653/main-1653
[ 50%] /data/local/tmp/main-1653/main-1653
[ 53%] /data/local/tmp/main-1653/main-1653
[ 56%] /data/local/tmp/main-1653/main-1653
[ 60%] /data/local/tmp/main-1653/main-1653
[ 63%] /data/local/tmp/main-1653/main-1653
[ 66%] /data/local/tmp/main-1653/main-1653
[ 69%] /data/local/tmp/main-1653/main-1653
[ 72%] /data/local/tmp/main-1653/main-1653
[ 75%] /data/local/tmp/main-1653/main-1653
[ 79%] /data/local/tmp/main-1653/main-1653
[ 82%] /data/local/tmp/main-1653/main-1653
[ 85%] /data/local/tmp/main-1653/main-1653
[ 88%] /data/local/tmp/main-1653/main-1653
[ 91%] /data/local/tmp/main-1653/main-1653
[ 94%] /data/local/tmp/main-1653/main-1653
[ 98%] /data/local/tmp/main-1653/main-1653
[100%] /data/local/tmp/main-1653/main-1653
/workdir/tmp/go-build400298215/b001/exe/main: 1 file pushed. 114.1 MB/s (2069533 bytes in 0.017s) 
go_android_exec: adb shell export TMPDIR="/data/local/tmp/main-1653"; export GOROOT="/data/local/tmp/goroot"; export GOPATH="/data/local/tmp/gopath"; cd "/data/local/tmp/goroot/misc/cgo/testsigfwd"; '/data/local/tmp/main-1653/main-1653' ; echo -n exitcode=$?
exitcode=0go_android_exec: adb shell rm -rf /data/local/tmp/main-1653

ALL TESTS PASSED
---
Installed Go for android/amd64 in /workdir/go
Installed commands in /workdir/go/bin
2019/02/27 22:42:23 ran src/androidtest.bash in 8m18s
2019/02/27 22:42:23 done.
2019/02/27 22:42:25 SERIAL: Feb 27 22:42:24 debug-temp-1551306784 containerd[677]: time="2019-02-27T22:42:24.295592684Z" level=info msg="shim reaped" id=aa181c26866f80366f38706e4509922d7db9e24af15ff9cbe64ff01e4bf954eb 
                            Feb 27 22:42:24 debug-temp-1551306784 dockerd[675]: time="2019-02-27T22:42:24.305745697Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"                     
                            Feb 27 22:42:24 debug-temp-1551306784 containerd[677]: time="2019-02-27T22:42:24.442179713Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/aa181c26866f80366f38706e4509922d7db9e24af15ff9cbe64ff01e4bf954eb/shim.sock" debug=false pid=3208
bradfitz@gdev:~/src/golang.org/x/build$
bradfitz@gdev:~/src/golang.org/x/build$

[gopherbot]

Change https://golang.org/cl/164380 mentions this issue: dashboard: enable android-amd64-emu and android-386-emu builders

[gopherbot]

Change https://golang.org/cl/164478 mentions this issue: cmd/coordinator: test x/mobile on android emu builders; and exp on some

[eliasnaur]

Now that Android builders are on GCE and soon joining the trybot set (thank you Brad!), I wonder how far we can get with iOS. @bradfitz can we have Darwin hosts in the cloud with Xcode installed running iOS tests on the iPhone simulator? Even without sharding and trybots, having cloud builders is a major step up from the very slow (and impossible to gomote) device builders we have now.

[gopherbot]

Change https://golang.org/cl/165797 mentions this issue: misc/android: enable many more tests on GOOS=android

[gopherbot]

Change https://golang.org/cl/166397 mentions this issue: misc/android: copy less from GOROOT to the device

[gopherbot]

Change https://golang.org/cl/172798 mentions this issue: dashboard: add Android trybots

[gopherbot]

Change https://golang.org/cl/174079 mentions this issue: dashboard: add android subrepo trybots too

[bradfitz]

@eliasnaur, I don't know what remains here (is it done? iOS stuff maybe?) so reassigning to you for now.

[eliasnaur]

iOS builders, yes. You mentioned that your macstadium VMs might be improving enough for me to use them for iOS simulator builds. Now that we have Corellium virtual arm64 devices, iOS simulators are probably only interesting if we can have enough simulators to run trybots.

[dmitshur]

@bradfitz @eliasnaur Do you know if there are there known limitations on the android-amd64-emu builder type? I checked its notes in dashboard and it just says "Android emulator on GCE". Does it support filesystem reads/writes? Is gomote ssh supported?

I can't seem to ssh into a gomote instance, but maybe it's just not supported:

~ $ gomote create android-amd64-emu
user-dmitshur-android-amd64-emu-0
~ $ gomote ssh user-dmitshur-android-amd64-emu-0
$ ssh -p 2222 user-dmitshur-android-amd64-emu-0@farmer.golang.org # auth using https://github.com/dmitshur.keys
# Welcome to the gomote ssh proxy, dmitshur.
# Connecting to/starting remote ssh...
#
failed to connect to ssh on user-dmitshur-android-amd64-emu-0: unexpected /connect-ssh response: 502 Bad Gateway, dial tcp [::1]:2200: connect: connection refused

Connection to farmer.golang.org closed.

Asking because there was a trybot failure that happens only on this trybot configuration, not others in CL 196321. /cc @ianthehat