Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: sharded mobile builders #23824

Open
eliasnaur opened this Issue Feb 14, 2018 · 64 comments

Comments

Projects
None yet
6 participants
@eliasnaur
Copy link
Contributor

eliasnaur commented Feb 14, 2018

Currently, the android/amd64 and android/386 builders run on an Android emulator with a amd64 system image. This is taxing the heavyly loaded mobile builder Mac Mini. It is also inefficient: the emulator builds competes with Android device builds and any concurrent iOS builds. Because builds are slow, android is not in the trybot set, leaving me to often pester CL authors with "this CL broke android".

Is it possible to add (and run) the Android emulator inside the existing docker images used for regular builders and then run android sharded tests as any other GOOS? If so, android/amd64 and android/386 builds would complete much faster, and take some pressure off the mobile device builder.

If the emulator builds are stable enough, android/amd64 (and perhaps android/386) could even join the trybot set, avoiding most (if not all) android specific followup work after a CL is submitted.

@gopherbot gopherbot added this to the Unreleased milestone Feb 14, 2018

@gopherbot gopherbot added the Builders label Feb 14, 2018

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 14, 2018

This issue is about Android, but in theory the same could be done for the iOS simulator.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 14, 2018

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 14, 2018

This

https://paulemtz.blogspot.dk/2013/05/android-testing-in-headless-emulator.html

seems to indicate that the standard emulator supports headless mode.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 14, 2018

Sure, prepare a Docker image that lets us do the traditional make.bash, then snapshot, then clone to several mirrors, then run tests one-at-a-time.

It'll require some tweaking of the build system probably, but I'll do those parts if you do the bulk of the work in preparing the Dockerfile.

seems to indicate that the standard emulator supports headless mode.

headless helps but isn't required. Worst case we could run a headless X server so it thinks it has a window.

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 15, 2018

The following Dockerfile

# Copyright 2014 The Go Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.

FROM debian:sid
MAINTAINER golang-dev <golang-dev@googlegroups.com>

ENV DEBIAN_FRONTEND noninteractive

# gdb: optionally used by runtime tests for gdb
# strace: optionally used by some net/http tests
# gcc libc6-dev: for building Go's bootstrap 'dist' prog
# libc6-dev-i386 gcc-multilib: for 32-bit builds
# procps lsof psmisc: misc basic tools
# libgles2-mesa-dev libopenal-dev fonts-noto: required by x/mobile repo
# unzip openjdk-8-jdk python lib32z1: required by the Android SDK
RUN apt-get update && apt-get install -y \
	--no-install-recommends \
	ca-certificates \
	curl \
	gdb \
	strace \
	gcc \
	libc6-dev \
	libc6-dev-i386 \
	gcc-multilib \
	procps \
	lsof \
	psmisc \
	libgles2-mesa-dev \
	libopenal-dev \
	fonts-noto \
	fonts-noto-mono \
	openssh-server \
	unzip \
	openjdk-8-jdk \
	python \
	lib32z1 \
	&& rm -rf /var/lib/apt/lists/*

RUN mkdir -p /go1.4-amd64 \
	&& ( \
		curl --silent https://storage.googleapis.com/golang/go1.4.linux-amd64.tar.gz | tar -C /go1.4-amd64 -zxv \
	) \
	&& mv /go1.4-amd64/go /go1.4 \
	&& rm -rf /go1.4-amd64 \
	&& rm -rf /go1.4/pkg/linux_amd64_race \
		/go1.4/api \
		/go1.4/blog \
		/go1.4/doc \
		/go1.4/misc \
		/go1.4/test \
	&& find /go1.4 -type d -name testdata | xargs rm -rf

RUN curl -o /usr/local/bin/stage0 https://storage.googleapis.com/go-builder-data/buildlet-stage0.linux-amd64-kube \
	&& chmod +x /usr/local/bin/stage0

RUN mkdir -p /android/sdk \
	&& curl -o /android/sdk/sdk-tools-linux.zip https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip \
	&& unzip -d /android/sdk /android/sdk/sdk-tools-linux.zip \
	&& rm -rf /android/sdk/sdk-tools-linux.zip

RUN yes | /android/sdk/tools/bin/sdkmanager --licenses \
	&& /android/sdk/tools/bin/sdkmanager ndk-bundle "system-images;android-26;google_apis;x86_64" \
	&& /android/sdk/tools/bin/sdkmanager "build-tools;21.1.2" "platforms;android-26" \
	&& /android/sdk/tools/bin/sdkmanager --update

# Gradle for gomobile
RUN curl -L -o /android/gradle-5.2.1-bin.zip https://services.gradle.org/distributions/gradle-5.2.1-bin.zip \
	&& unzip -d /android /android/gradle-5.2.1-bin.zip \
	&& rm /android/gradle-5.2.1-bin.zip

# Cleanup
RUN rm -rf /android/sdk/sdk-tools-linux.zip \
	&& apt-get -y remove unzip \
	&& apt-get -y autoremove

COPY run-emulator.sh /android/run-emulator.sh
RUN chmod +x /android/run-emulator.sh

# Include a checkout of Go
COPY go-tip /go-tip

CMD ["/usr/local/bin/stage0"]

And supporting run-emulator.sh (as well as a Go checkout in go-tip)

#!/bin/sh

set -e

# create emulator
echo no | /android/sdk/tools/bin/avdmanager create avd --force --name android-avd --package "system-images;android-26;google_apis;x86_64"

# run the emulator.
/android/sdk/emulator/emulator @android-avd -no-window &

# wait for it.
/android/sdk/platform-tools/adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;'

Can be started with:

sudo docker run --privileged -it <image-name> /bin/bash

Where I'm able to complete androidtest.sh with android/386 and android/amd64:

/android/run-emulator.sh
cd /go-tip/src
apt-get update && apt-get install git-core
export PATH=$PATH:/android/sdk/platform-tools; GOROOT_BOOTSTRAP=/go1.4 CC_FOR_TARGET=/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/i686-linux-android26-clang GOARCH=386 ./androidtest.bash
export PATH=$PATH:/android/sdk/platform-tools; GOROOT_BOOTSTRAP=/go1.4 CC_FOR_TARGET=/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android26-clang GOARCH=amd64 ./androidtest.bash

It can even complete the full test suite of gomobile:

GOROOT_BOOTSTRAP=/go1.4 ./make.bash
export PATH=$PATH:/go-tip/bin:$HOME/go/bin:/android/sdk/platform-tools:/android/gradle-5.2.1/bin
export GOPATH=$HOME/go                                                                                                                  
export ANDROID_HOME=/android/sdk                                                                                                        
go get -u golang.org/x/mobile/...                                                                                                       
go test -short golang.org/x/mobile/...

Let me know how to proceed.

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Mar 5, 2018

Ping @bradfitz

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Mar 10, 2018

Sorry, I realistically won't be able to get to start on this until April 1st. I'm preparing for an upcoming Gophercon talk this weekend, traveling next week for the talk, and then on vacation the last week of March.

I might be able to work on this next week from Moscow, but no promises.

@andybons has been wanting to learn how this whole system works more, so this might be a good "starter project" for adding a builder.

BTW, I forgot to ask earlier: how long does this all take? I imagine sharding is necessary for being a trybot, but is your goal for it to be a trybot right away, or just to have it run on GCE by us in Kubernetes? If it's easier, we could skip sharding for the first version.

But I'm also fine with sharding for the first version, but curious about timing. I imagine make.bash is still fast like always (1 minute or less), but then how long are the various tests?

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Mar 11, 2018

Sorry, I realistically won't be able to get to start on this until April 1st. I'm preparing for an upcoming Gophercon talk this weekend, traveling next week for the talk, and then on vacation the last week of March.

No problem at all. My ping was just as much to see if the Docker image was enough.

BTW, I forgot to ask earlier: how long does this all take? I imagine sharding is necessary for being a trybot, but is your goal for it to be a trybot right away, or just to have it run on GCE by us in Kubernetes? If it's easier, we could skip sharding for the first version.

My wishlist ordered by importance:

  1. Run emulated android/386 and android/amd64 builder on GCE (non-sharded or sharded, whatever is convenient for you). That will take a good amount of load off the strained mac mini that runs all mobile builders today. Also, it will ensure a builder is running if the mini is down which happens from time to time.
  2. Use the GCE builder to run the gomobile subrepository tests. There is no meaningful automatic testing for gomobile today, so this would be a great way to get some coverage there.
  3. Android trybot builders, to catch any Android specific issues with CLs before they're submitted.

(And in the future: 4. Do 1-3 for a darwin builder running an 386/amd64 iOS emulator.)

In other words, "just" a regular non-sharded builder on GCE would make me very grateful.

But I'm also fine with sharding for the first version, but curious about timing. I imagine make.bash is still fast like always (1 minute or less), but then how long are the various tests?

On my i7-6700K@4GHz with 32GB memory with a running Android 8 x86_64 emulator, ./androidtest.bash takes ~2min and ./all.bash on the same machine takes ~3min.

BTW, I'm not even sure trybots can be stable enough before #23795 is fixed. You wouldn't happen to know someone to ping for the corresponding adb bug, https://issuetracker.google.com/issues/73230216 ?

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Apr 26, 2018

I just stumpled on #9579. Will the Android emulator run on GCE?

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Apr 26, 2018

@bcmills, this is another builder request.

@bradfitz bradfitz added the OS-Android label Apr 26, 2018

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Jun 19, 2018

Gentle ping. If I can do anything to help this, please let me know.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jun 19, 2018

I've been focused on vgo reviews and bugs (since that's fairly urgent for keeping 1.11 on schedule). I plan to get back to this after the 1.11 release, but if anybody else would like to take it before then you're welcome to it.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 15, 2019

Change https://golang.org/cl/162959 mentions this issue: dashboard, buildlet: add a disabled builder with nested virt, for testing

gopherbot pushed a commit to golang/build that referenced this issue Feb 15, 2019

dashboard, buildlet: add a disabled builder with nested virt, for tes…
…ting

This adds a linux-amd64 COS builder that should be just like our
existing linux-amd64 COS builder except that it's using a forked image
that has the VMX license bit enabled for nested virtualization. (GCE
appears to be using the license mechanism as some sort of opt-in
mechanism for features that aren't yet GA; might go away?)

Once this is in, it won't do any new builds as regular+trybot builders
are disabled. But it means I can then use gomote + debugnewvm to work
on preparing the other four image types.

Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)

Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015
Reviewed-on: https://go-review.googlesource.com/c/162959
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 19, 2019

Change https://golang.org/cl/163057 mentions this issue: buildlet: change image name for COS-with-vmx buildlet

gopherbot pushed a commit to golang/build that referenced this issue Feb 19, 2019

buildlet: change image name for COS-with-vmx buildlet
The COS image I'd forked from earlier didn't have CONFIG_KVM or
CONFIG_KVM_INTEL enabled in its kernel, so even though I'd enabled the
VMX license bit for the VM, the kernel was unable to use it.

Now I've instead rebuilt the ChromiumOS "lakitu" board with a modified
kernel config:

   https://cloud.google.com/container-optimized-os/docs/how-to/building-from-open-source

More docs later. Still tinkering. Nothing uses this yet.

Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)

Change-Id: Id2839066e67d9ddda939d96c5f4287af3267a769
Reviewed-on: https://go-review.googlesource.com/c/163057
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 21, 2019

Change https://golang.org/cl/163301 mentions this issue: env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx

gopherbot pushed a commit to golang/build that referenced this issue Feb 21, 2019

env/linux-x86-vmx: add new Debian host that's like Container-Optimize…
…d OS + vmx

This adds scripts to create a new builder host image that acts like
Container-Optimized OS (has docker, runs konlet on startup) but with a
Debian 9 kernel + userspace that permits KVM for nested
virtualization.

Updates golang/go#15581 (solaris)
Updates golang/go#23060 (dragonfly)
Updates golang/go#30262 (riscv)
Updates golang/go#30267 (fuchsia)
Updates golang/go#23824 (android)

Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884
Reviewed-on: https://go-review.googlesource.com/c/163301
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 21, 2019

@eliasnaur, this is ready from my side if you want to give me a Docker image that uses the Android SDK + emulator (which can now use KVM on our buildlets) to run tests.

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 21, 2019

Is the one from #23824 (comment) good enough?

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 21, 2019

Oh right. Forgot about that. I'll try.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 22, 2019

@eliasnaur,

Step 8/14 : RUN yes | /android/sdk/tools/bin/sdkmanager --licenses      && /android/sdk/tools/bin/sdkmanager ndk-bundle "system-images;android-26;google_apis;x86_64"       && /android/sdk/tools/bin/sdkmanager "build-tools;21.1.2" "platforms;android-19"        && /android/sdk/tools/bin/sdkmanager --update
 ---> Running in 1288386ab097
Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlSchema
        at com.android.repository.api.SchemaModule$SchemaModuleVersion.<init>(SchemaModule.java:156)
        at com.android.repository.api.SchemaModule.<init>(SchemaModule.java:75)
        at com.android.sdklib.repository.AndroidSdkHandler.<clinit>(AndroidSdkHandler.java:81)
        at com.android.sdklib.tool.SdkManagerCli.main(SdkManagerCli.java:117)
        at com.android.sdklib.tool.SdkManagerCli.main(SdkManagerCli.java:93)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlSchema
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 5 more
The command '/bin/sh -c yes | /android/sdk/tools/bin/sdkmanager --licenses      && /android/sdk/tools/bin/sdkmanager ndk-bundle "system-images;android-26;google_apis;x86_64"       && /android/sdk/tools/bin/sdkmanager "build-tools;21.1.2" "platforms;android-19"        && /android/sdk/tools/bin/sdkmanager --update' returned a non-zero code: 1
@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 22, 2019

I updated the Dockerfile and run-emulator.sh script, please try again. Changes:

  • Replaced default-jdk with openjdk-8-jdk in Dockerfile to make sdkmanager run.
  • Removed the prebuilding of the NDK toolchains from run-emulator.sh and updated the androidtest.bash instructions to use the builtin prebuilt toolchains introduced in NDK r19b.
  • Updated Gradle to 5.2.1 in the Dockerfile since the old one was no longer supported by gomobile.

I've created CL 163377 to fix a gomobile test failure with jdk8, and CL 163378 so the gomobile init step can go away.

Edit: The two CLs are in, all gomobile tests pass and the gomobile init step is no longer necessary.

Thank you for working on this.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 22, 2019

I'm running it locally as a test.

I notice that when androidtest.bash is still doing the make.bash phase, the Android qemu process is successfully using KVM (great) but is also using 200% CPU and I haven't even started sending it tests yet because make.bash is still running. What is the emulator doing early on? Animating some live wallpaper or something? Anything we can do to reduce unnecessary CPU usage to make the build & tests faster?

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 22, 2019

I'm not sure. There is an emulator flag, -no-boot-anim that "disable animation for faster boot", but I assumed -no-window already effectively did that.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 25, 2019

Change https://golang.org/cl/163625 mentions this issue: misc/android: serialize adb commands on android emulators

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 25, 2019

I did the docker commit thing, but I'm seeing it take 20 seconds later to use the warm image rather than the base pre-commit image:

Here I build golang/android, then run the emulator in it with docker run --privileged, commit it as golang/android-warm, and then time re-starting the emulator in both the original image vs the warm image. The warm image seems to take 1m40s and the original is 1m20s-ish.

Am I missing a step?

bradfitz@gdev:~/src/golang.org/x/build/env/android-amd64$ docker build --tag=golang/android .
...
Successfully built f06b8bc59234
Successfully tagged golang/android:latest


bradfitz@gdev:~/src/golang.org/x/build/env/android-amd64$ docker run --privileged -it golang/android /android/sdk/emulator/emulator @android-avd -no-window -no-boot-anim -quit-after-boot 100
Couldn't statvfs() path: No such file or directory
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
Your emulator is out of date, please update by launching Android Studio:
 - Start Android Studio
 - Select menu "Tools > Android > SDK Manager"
 - Click "SDK Tools" tab
 - Check "Android Emulator" checkbox
 - Click "OK"

emulator: INFO: boot completed
emulator: Saving state on exit with session uptime 75431 ms


bradfitz@gdev:~/src/golang.org/x/build/env/android-amd64$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                      PORTS               NAMES
2a03a697a8ff        golang/android      "/android/sdk/emulat…"   About a minute ago   Exited (0) 30 seconds ago                       unruffled_neumann
....

bradfitz@gdev:~/src/golang.org/x/build/env/android-amd64$ docker commit 2a03a697a8ff golang/android-warm
sha256:314bb914743d041d6366415b2078264c53ed1718d5ae880e795059e600ac8da5


bradfitz@gdev:~/src/golang.org/x/build/env/android-amd64$ time docker run --privileged -it golang/android-warm /android/sdk/emulator/emulator @android-avd -no-window -no-boot-anim -quit-after-boot 100
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
emulator: ERROR: fail to boot after 100 seconds, quit
emulator: Saving state on exit with session uptime 97567 ms

real    1m41.658s
user    0m0.084s
sys     0m0.016s
bradfitz@gdev:~/src/golang.org/x/build/env/android-amd64$ time docker run --privileged -it golang/android /android/sdk/emulator/emulator @android-avd -no-window -no-boot-anim -quit-after-boot 100
Couldn't statvfs() path: No such file or directory
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
Your emulator is out of date, please update by launching Android Studio:
 - Start Android Studio
 - Select menu "Tools > Android > SDK Manager"
 - Click "SDK Tools" tab
 - Check "Android Emulator" checkbox
 - Click "OK"

emulator: INFO: boot completed
emulator: Saving state on exit with session uptime 74576 ms

real    1m19.830s
user    0m0.092s
sys     0m0.020s


bradfitz@gdev:~/src/golang.org/x/build/env/android-amd64$ time docker run --privileged -it golang/android-warm /android/sdk/emulator/emulator @android-avd -no-window -no-boot-anim -quit-after-boot 100
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
emulator: ERROR: fail to boot after 100 seconds, quit
emulator: Saving state on exit with session uptime 98278 ms

real    1m42.308s
user    0m0.088s
sys     0m0.024s
@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 25, 2019

That's essentially the same thing I did so I don't think you missed a step. I did get some strange timings the first few times I ran the warm image; it took longer for the emulator to start but it booted faster. Perhaps docker is doing something slow with the image data?

Also your warm image timing is actually worse than what you see. The -quit-after-boot flag takes a number of seconds to wait for boot (100 in your case), and the emulator failed to boot in that time: emulator: ERROR: fail to boot after 100 seconds, quit

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 25, 2019

Perhaps we should start with the simpler cold booted emulator? Even if your timings eventually match mine, they might not apply to GCE.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 25, 2019

My dev instance is also on GCE, on the same VM types.

So the warm boot isn't even working it seems. Can you investigate that while I finish up the buildlet & Dockerfile & dashboard/builders.go configuration? (I'll be sending that to you soon here.)

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 25, 2019

I'm on my own machine, not on GCE :) So I think that's a pretty good indication that warm booting is not a net win. The extra time probably goes to fetching the extra ~3GB of Docker image across the network.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 25, 2019

Why should warm booting be slower, though?

Presumably it's faster on a developer's desktop, so why isn't it faster for us? In this case I have all the images pulled locally, so network isn't a concern. And we'll have these Docker files pre-baked into the image whose block device is lazily read over the super fast network anyway, so Dockerfile size isn't really a concern here.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 25, 2019

Change https://golang.org/cl/163738 mentions this issue: env/android-amd64-emu: add new Android emulator image

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 25, 2019

Why should warm booting be slower, though?

Presumably it's faster on a developer's desktop, so why isn't it faster for us? In this case I have all the images pulled locally, so network isn't a concern. And we'll have these Docker files pre-baked into the image whose block device is lazily read over the super fast network anyway, so Dockerfile size isn't really a concern here.

I don't know. And now I can't even reproduce my numbers above.

Warm boot attempts:
1m17,427s
1m15,680s
1m12,846s
1m15,234s

Cold boot attempts:
1m12,414s
1m36,328s
1m29,003s
1m10,898s

Cold boot times vary quite a bit and is even faster than warm boot sometimes.

In an attempt to optimize the image I tried to docker export and then docker import the warm boot container, but that made it much worse and similar to your warm boot case.

$ docker run --privileged -it warmboot /android/sdk/emulator/emulator @android-avd -no-window -no-boot-anim -quit-after-boot 200
$ docker container ls --all
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS               NAMES
8d026f942a82        warmboot            "/android/sdk/emul..."   3 minutes ago       Exited (0) 2 minutes ago                       hopeful_liskov
$ docker export 8d026f942a82 > warmboot.tar
$ docker import warmboot.tar :warmboot-import
$ time docker run --rm --privileged -it warmboot-import /android/sdk/emulator/emulator @android-avd -no-window -no-boot-anim -quit-after-boot 200
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
emulator: ERROR: fail to boot after 200 seconds, quit
emulator: Saving state on exit with session uptime 181916 ms
0.02user 0.00system 3:22.36elapsed 0%CPU (0avgtext+0avgdata 18580maxresident)k
0inputs+0outputs (0major+1808minor)pagefaults 0swaps
```
@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 25, 2019

Compare that to emulator runs from my host machine:

$ time emulator @android27 -no-window -no-boot-anim -quit-after-boot 200
Couldn't statvfs() path: No such file or directory
Your emulator is out of date, please update by launching Android Studio:
 - Start Android Studio
 - Select menu "Tools > Android > SDK Manager"
 - Click "SDK Tools" tab
 - Check "Android Emulator" checkbox
 - Click "OK"

emulator: INFO: boot completed
emulator: Saving state on exit with session uptime 12187 ms

real	0m13,183s
user	0m19,371s
sys	0m2,603s
@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 25, 2019

13 seconds is much better. Weird. I wonder what's going wrong within Docker.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 25, 2019

Can we get more verbose logs out of it?

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 25, 2019

FWIW, I can get closer to my host timing by warming up the Docker container:

$ docker run --rm --privileged -it cold /bin/bash
[sudo] password for elias: 
root@7775accc64eb:/# time /android/sdk/emulator/emulator @android-avd -no-window -no-boot-anim -quit-after-boot 200
Couldn't statvfs() path: No such file or directory
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
Your emulator is out of date, please update by launching Android Studio:
 - Start Android Studio
 - Select menu "Tools > Android > SDK Manager"
 - Click "SDK Tools" tab
 - Check "Android Emulator" checkbox
 - Click "OK"

emulator: INFO: boot completed
emulator: Saving state on exit with session uptime 17154 ms

real	1m4.910s
user	0m14.057s
sys	0m4.621s
root@7775accc64eb:/# time /android/sdk/emulator/emulator @android-avd -no-window -no-boot-anim -quit-after-boot 200
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
^Cemulator: Saving state on exit with session uptime 51646 ms

real	0m51.995s
user	0m4.143s
sys	0m1.630s
root@7775accc64eb:/# time /android/sdk/emulator/emulator @android-avd -wipe-data -no-window -no-boot-anim -quit-after-boot 200
Couldn't statvfs() path: No such file or directory
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
emulator: INFO: boot completed
emulator: Saving state on exit with session uptime 17304 ms

real	0m31.853s
user	0m13.885s
sys	0m4.415s
root@7775accc64eb:/# time /android/sdk/emulator/emulator @android-avd -wipe-data -no-window -no-boot-anim -quit-after-boot 200
Couldn't statvfs() path: No such file or directory
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
emulator: INFO: boot completed
emulator: Saving state on exit with session uptime 17136 ms

real	0m32.429s
user	0m15.675s
sys	0m4.477s

Note that -wipe-datawas needed to avoid a hang on the second window-less run.

@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Feb 25, 2019

I found this on the slowness of Docker's loopback mounted disks: moby/moby#21485

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 26, 2019

I found this on the slowness of Docker's loopback mounted disks: moby/moby#21485

Are we loopback mounting anything, though?

In any case, I can get it down to 36 seconds with:

root@3a11990d0b49:/# time /android/sdk/emulator/emulator @android-avd -no-audio -no-window -no-boot-anim -no-sim -quit-after-boot 200
...
(builds snapshot)
...

Then:

root@3a11990d0b49:/# time /android/sdk/emulator/emulator @android-avd -no-audio -no-window -no-boot-anim -quit-after-boot 200 -no-snapshot-save -show-kernel -no-sim 
...
emulator: INFO: boot completed
emulator: WARNING: Not saving state: RAM not mapped as shared

real    0m36.144s
user    0m44.888s
sys     0m23.976s

Haven't yet tried to docker commit the first step, though.

gopherbot pushed a commit that referenced this issue Feb 26, 2019

cmd/dist: build exec wrappers during bootstrap
The androidtest.bash script encodes the additional steps to build
Go and run tests on Android. In order to add sharded builders and
trybots, Android needs to fit into the usual make.bash + cmd/dist test
pattern.

This change moves building the exec wrapper into cmd/dist bootstrap.

Do the same for iOS while we're here.

Updates #23824

Change-Id: I58a1b0679c3a6c92fdc7fff464b469641f1fee74
Reviewed-on: https://go-review.googlesource.com/c/163618
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

gopherbot pushed a commit that referenced this issue Feb 26, 2019

misc/android,cmd/dist: move $GOROOT copying to the exec wrapper
To run the standard library tests on Android, the androidtest.bash
script copies GOROOT to the device. Move that logic to the android
exec wrapper, thereby making androidtest.bash obsolete.

Apart from making Android less special, the sharded builder
infrastructure should now be able to run (emulated) Android builders
and trybots without special treatment.

Updates #23824

Change-Id: I41591fea9a15b38c6dcf84046ea57f1e9165eaa5
Reviewed-on: https://go-review.googlesource.com/c/163619
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

gopherbot pushed a commit that referenced this issue Feb 26, 2019

misc/android: serialize adb commands on android emulators
Android emulator builders are soon to join the trybot set. To avoid
flaky runs, work around a longstanding adb bug where concurrent adb
commands sometimes fail.

I haven't seen the problem on actual devices until recently. It seems
that the recently added "adb wait-for-device" can introduce flakyness
with errors such as:

adb: error: failed to get feature set: protocol fault (couldn't read status): Connection reset by peer

Instead of working around that, give up and serialize use of adb
everywhere.

Fixes #23795
Updates #23824

Change-Id: If347c9981fa32ff8a1e14b7454f122ef682450a6
Reviewed-on: https://go-review.googlesource.com/c/163625
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 27, 2019

Getting closer. I almost have all the pieces done.

Testing it all out now,

FAIL    crypto/tls      0.002s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH
FAIL    crypto/x509     0.002s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH
FAIL    database/sql    0.006s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH
FAIL    database/sql/driver     0.002s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH
FAIL    debug/dwarf     0.002s
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;: exec: "adb": executable file not found in $PATH

Few little config things missing it appears.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 27, 2019

Works now.

bradfitz@gdev:~/src/golang.org/x/build$ debugnewvm --host=host-android-amd64-emu -run-build=android-amd64-emu
...
....
##### ../misc/cgo/testasan
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb shell mkdir -p /data/local/tmp/main-1465
go_android_exec: adb shell mkdir -p /data/local/tmp/goroot/misc/cgo/testasan
go_android_exec: adb push /workdir/tmp/go-build746649494/b001/exe/main /data/local/tmp/main-1465/main-1465
[  5%] /data/local/tmp/main-1465/main-1465
[ 10%] /data/local/tmp/main-1465/main-1465
[ 15%] /data/local/tmp/main-1465/main-1465
[ 21%] /data/local/tmp/main-1465/main-1465
[ 26%] /data/local/tmp/main-1465/main-1465
[ 31%] /data/local/tmp/main-1465/main-1465
[ 37%] /data/local/tmp/main-1465/main-1465
[ 42%] /data/local/tmp/main-1465/main-1465
[ 47%] /data/local/tmp/main-1465/main-1465
[ 53%] /data/local/tmp/main-1465/main-1465
[ 58%] /data/local/tmp/main-1465/main-1465
[ 63%] /data/local/tmp/main-1465/main-1465
[ 69%] /data/local/tmp/main-1465/main-1465
[ 74%] /data/local/tmp/main-1465/main-1465
[ 79%] /data/local/tmp/main-1465/main-1465
[ 85%] /data/local/tmp/main-1465/main-1465
[ 90%] /data/local/tmp/main-1465/main-1465
[ 95%] /data/local/tmp/main-1465/main-1465
[100%] /data/local/tmp/main-1465/main-1465
/workdir/tmp/go-build746649494/b001/exe/main: 1 file pushed. 77.5 MB/s (1232413 bytes in 0.015s)
go_android_exec: adb shell export TMPDIR="/data/local/tmp/main-1465"; export GOROOT="/data/local/tmp/goroot"; export GOPATH="/data/local/tmp/gopath"; cd "/data/local/tmp/goroot/misc/cgo/testasan"; '/data/local/tmp/main-1465
/main-1465' ; echo -n exitcode=$?
exitcode=0go_android_exec: adb shell rm -rf /data/local/tmp/main-1465

##### ../misc/cgo/testsigfwd
go_android_exec: adb wait-for-device shell while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;
go_android_exec: adb shell mkdir -p /data/local/tmp/main-1653
go_android_exec: adb shell mkdir -p /data/local/tmp/goroot/misc/cgo/testsigfwd
go_android_exec: adb push /workdir/tmp/go-build400298215/b001/exe/main /data/local/tmp/main-1653/main-1653
[  3%] /data/local/tmp/main-1653/main-1653
[  6%] /data/local/tmp/main-1653/main-1653
[  9%] /data/local/tmp/main-1653/main-1653
[ 12%] /data/local/tmp/main-1653/main-1653
[ 15%] /data/local/tmp/main-1653/main-1653
[ 18%] /data/local/tmp/main-1653/main-1653
[ 22%] /data/local/tmp/main-1653/main-1653
[ 25%] /data/local/tmp/main-1653/main-1653
[ 28%] /data/local/tmp/main-1653/main-1653
[ 31%] /data/local/tmp/main-1653/main-1653
[ 34%] /data/local/tmp/main-1653/main-1653
[ 37%] /data/local/tmp/main-1653/main-1653
[ 41%] /data/local/tmp/main-1653/main-1653
[ 44%] /data/local/tmp/main-1653/main-1653
[ 47%] /data/local/tmp/main-1653/main-1653
[ 50%] /data/local/tmp/main-1653/main-1653
[ 53%] /data/local/tmp/main-1653/main-1653
[ 56%] /data/local/tmp/main-1653/main-1653
[ 60%] /data/local/tmp/main-1653/main-1653
[ 63%] /data/local/tmp/main-1653/main-1653
[ 66%] /data/local/tmp/main-1653/main-1653
[ 69%] /data/local/tmp/main-1653/main-1653
[ 72%] /data/local/tmp/main-1653/main-1653
[ 75%] /data/local/tmp/main-1653/main-1653
[ 79%] /data/local/tmp/main-1653/main-1653
[ 82%] /data/local/tmp/main-1653/main-1653
[ 85%] /data/local/tmp/main-1653/main-1653
[ 88%] /data/local/tmp/main-1653/main-1653
[ 91%] /data/local/tmp/main-1653/main-1653
[ 94%] /data/local/tmp/main-1653/main-1653
[ 98%] /data/local/tmp/main-1653/main-1653
[100%] /data/local/tmp/main-1653/main-1653
/workdir/tmp/go-build400298215/b001/exe/main: 1 file pushed. 114.1 MB/s (2069533 bytes in 0.017s) 
go_android_exec: adb shell export TMPDIR="/data/local/tmp/main-1653"; export GOROOT="/data/local/tmp/goroot"; export GOPATH="/data/local/tmp/gopath"; cd "/data/local/tmp/goroot/misc/cgo/testsigfwd"; '/data/local/tmp/main-1653/main-1653' ; echo -n exitcode=$?
exitcode=0go_android_exec: adb shell rm -rf /data/local/tmp/main-1653

ALL TESTS PASSED
---
Installed Go for android/amd64 in /workdir/go
Installed commands in /workdir/go/bin
2019/02/27 22:42:23 ran src/androidtest.bash in 8m18s
2019/02/27 22:42:23 done.
2019/02/27 22:42:25 SERIAL: Feb 27 22:42:24 debug-temp-1551306784 containerd[677]: time="2019-02-27T22:42:24.295592684Z" level=info msg="shim reaped" id=aa181c26866f80366f38706e4509922d7db9e24af15ff9cbe64ff01e4bf954eb 
                            Feb 27 22:42:24 debug-temp-1551306784 dockerd[675]: time="2019-02-27T22:42:24.305745697Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"                     
                            Feb 27 22:42:24 debug-temp-1551306784 containerd[677]: time="2019-02-27T22:42:24.442179713Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/aa181c26866f80366f38706e4509922d7db9e24af15ff9cbe64ff01e4bf954eb/shim.sock" debug=false pid=3208
bradfitz@gdev:~/src/golang.org/x/build$
bradfitz@gdev:~/src/golang.org/x/build$

gopherbot pushed a commit to golang/build that referenced this issue Feb 28, 2019

env/android-amd64-emu: add new Android emulator container + VM image
Now that we can do nested virtualization on GCE, that means we can run
the Android emulator (which requires KVM) on GCE and at least get
fast trybots and such for android-amd64.

Updates golang/go#23824

Change-Id: I0da38c7fa0f15492230a31291d2921ba72f2151d
Reviewed-on: https://go-review.googlesource.com/c/163738
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 28, 2019

Change https://golang.org/cl/164380 mentions this issue: dashboard: enable android-amd64-emu and android-386-emu builders

gopherbot pushed a commit to golang/build that referenced this issue Feb 28, 2019

dashboard: enable android-amd64-emu and android-386-emu builders
Updates golang/go#23824

Change-Id: I24b590dab6719d014472da3d7850fc9a1b3ef4cc
Reviewed-on: https://go-review.googlesource.com/c/164380
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 28, 2019

Change https://golang.org/cl/164478 mentions this issue: cmd/coordinator: test x/mobile on android emu builders; and exp on some

gopherbot pushed a commit to golang/build that referenced this issue Feb 28, 2019

cmd/coordinator: test x/mobile on android emu builders; and exp on some
Updates golang/go#23824

Change-Id: Iee70bcbfef6a271fdd26694d848a703a200dc0c0
Reviewed-on: https://go-review.googlesource.com/c/164478
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
@eliasnaur

This comment has been minimized.

Copy link
Contributor Author

eliasnaur commented Mar 8, 2019

Now that Android builders are on GCE and soon joining the trybot set (thank you Brad!), I wonder how far we can get with iOS. @bradfitz can we have Darwin hosts in the cloud with Xcode installed running iOS tests on the iPhone simulator? Even without sharding and trybots, having cloud builders is a major step up from the very slow (and impossible to gomote) device builders we have now.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Mar 8, 2019

Change https://golang.org/cl/165797 mentions this issue: misc/android: enable many more tests on GOOS=android

gopherbot pushed a commit that referenced this issue Mar 8, 2019

misc/android: enable many more tests on GOOS=android
Android tests are built on the host and run on the device. To do
that, the exec wrapper copies the test binary and testdata to the device.
To enable many more tests, make the copied environment more like the host:

- Copy all of pkg from GOROOT, not just the android pkg directory.
- Copy any parent testdata directories as well as the package's own.
- Copy *.go files from the package directory. This enables misc/cgo/stdio
and misc/cgo/life tests that were invisible before so disable them explicitly.
- Always copy the GOROOT, even for tests outside GOROOT. This is expensive
 but only done once per make.bash.
- Build the go tool for the device and put it in PATH. Set GOCACHE
to a writable directory and disable cgo.

While here, use a single directory for all the exec wrapper files and
delete that once per make.bash as well.

In total, this CL enables many tests in the subrepos that would need skips
without it, in particular the x/tools tests.

Fixes #11452
Updates #23824
Updates #11811

Change-Id: I2e50d8b57db9bc4637f25272a5360c8b2cf4e627
Reviewed-on: https://go-review.googlesource.com/c/go/+/165797
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Mar 9, 2019

Change https://golang.org/cl/166397 mentions this issue: misc/android: copy less from GOROOT to the device

gopherbot pushed a commit that referenced this issue Mar 11, 2019

misc/android: copy less from GOROOT to the device
The android emulator builders is running out of space after CL 165797
copied most of GOROOT to the device.
The pkg directory is by far the largest, so only include what seems
necessary to build the x/ repositories: pkg/android_$GOARCH and
pkg/tool/android_$GOARCH.

While here, rename the device root directory to match the exec
wrapper name and make sure the deferred cleanups actually run before
os.Exit.

Hopefully fixes the emulator builders.

Updates #23824

Change-Id: I4d1e3ab2c89fd1e5818503d323ddb87f073094da
Reviewed-on: https://go-review.googlesource.com/c/go/+/166397
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.