New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: sharded mobile builders #23824

Open
eliasnaur opened this Issue Feb 14, 2018 · 13 comments

Comments

Projects
None yet
5 participants
@eliasnaur
Contributor

eliasnaur commented Feb 14, 2018

Currently, the android/amd64 and android/386 builders run on an Android emulator with a amd64 system image. This is taxing the heavyly loaded mobile builder Mac Mini. It is also inefficient: the emulator builds competes with Android device builds and any concurrent iOS builds. Because builds are slow, android is not in the trybot set, leaving me to often pester CL authors with "this CL broke android".

Is it possible to add (and run) the Android emulator inside the existing docker images used for regular builders and then run android sharded tests as any other GOOS? If so, android/amd64 and android/386 builds would complete much faster, and take some pressure off the mobile device builder.

If the emulator builds are stable enough, android/amd64 (and perhaps android/386) could even join the trybot set, avoiding most (if not all) android specific followup work after a CL is submitted.

@gopherbot gopherbot added this to the Unreleased milestone Feb 14, 2018

@gopherbot gopherbot added the Builders label Feb 14, 2018

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Feb 14, 2018

This issue is about Android, but in theory the same could be done for the iOS simulator.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 14, 2018

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Feb 14, 2018

This

https://paulemtz.blogspot.dk/2013/05/android-testing-in-headless-emulator.html

seems to indicate that the standard emulator supports headless mode.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Feb 14, 2018

Sure, prepare a Docker image that lets us do the traditional make.bash, then snapshot, then clone to several mirrors, then run tests one-at-a-time.

It'll require some tweaking of the build system probably, but I'll do those parts if you do the bulk of the work in preparing the Dockerfile.

seems to indicate that the standard emulator supports headless mode.

headless helps but isn't required. Worst case we could run a headless X server so it thinks it has a window.

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Feb 15, 2018

The following Dockerfile

# Copyright 2014 The Go Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.

FROM debian:sid
MAINTAINER golang-dev <golang-dev@googlegroups.com>

ENV DEBIAN_FRONTEND noninteractive

# gdb: optionally used by runtime tests for gdb
# strace: optionally used by some net/http tests
# gcc libc6-dev: for building Go's bootstrap 'dist' prog
# libc6-dev-i386 gcc-multilib: for 32-bit builds
# procps lsof psmisc: misc basic tools
# libgles2-mesa-dev libopenal-dev fonts-noto: required by x/mobile repo
# unzip default-jdk python lib32z1: required by the Android SDK
RUN apt-get update && apt-get install -y \
	--no-install-recommends \
	ca-certificates \
	curl \
	gdb \
	strace \
	gcc \
	libc6-dev \
	libc6-dev-i386 \
	gcc-multilib \
	procps \
	lsof \
	psmisc \
	libgles2-mesa-dev \
	libopenal-dev \
	fonts-noto \
	fonts-noto-mono \
	openssh-server \
	unzip \
	default-jdk \
	python \
	lib32z1 \
	&& rm -rf /var/lib/apt/lists/*

RUN mkdir -p /go1.4-amd64 \
	&& ( \
		curl --silent https://storage.googleapis.com/golang/go1.4.linux-amd64.tar.gz | tar -C /go1.4-amd64 -zxv \
	) \
	&& mv /go1.4-amd64/go /go1.4 \
	&& rm -rf /go1.4-amd64 \
	&& rm -rf /go1.4/pkg/linux_amd64_race \
		/go1.4/api \
		/go1.4/blog \
		/go1.4/doc \
		/go1.4/misc \
		/go1.4/test \
	&& find /go1.4 -type d -name testdata | xargs rm -rf

RUN curl -o /usr/local/bin/stage0 https://storage.googleapis.com/go-builder-data/buildlet-stage0.linux-amd64-kube \
	&& chmod +x /usr/local/bin/stage0

RUN mkdir -p /android/sdk \
	&& curl -o /android/sdk/sdk-tools-linux.zip https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip \
	&& unzip -d /android/sdk /android/sdk/sdk-tools-linux.zip \
	&& rm -rf /android/sdk/sdk-tools-linux.zip

RUN yes | /android/sdk/tools/bin/sdkmanager --licenses \
	&& /android/sdk/tools/bin/sdkmanager ndk-bundle "system-images;android-26;google_apis;x86_64" \
	&& /android/sdk/tools/bin/sdkmanager "build-tools;21.1.2" "platforms;android-19" \
	&& /android/sdk/tools/bin/sdkmanager --update

# Gradle for gomobile
RUN curl -L -o /android/gradle-2.13-bin.zip https://services.gradle.org/distributions/gradle-2.13-bin.zip \
	&& unzip -d /android /android/gradle-2.13-bin.zip \
	&& rm /android/gradle-2.13-bin.zip

# Cleanup
RUN rm -rf /android/sdk/sdk-tools-linux.zip \
	&& apt-get -y remove unzip \
	&& apt-get -y autoremove

COPY run-emulator.sh /android/run-emulator.sh
RUN chmod +x /android/run-emulator.sh

# Include a checkout of Go
COPY go-tip /go-tip

CMD ["/usr/local/bin/stage0"]

And supporting run-emulator.sh (as well as a Go checkout in go-tip)

#!/bin/sh

set -e

/android/sdk/ndk-bundle/build/tools/make_standalone_toolchain.py --arch x86 --install-dir /android/ndk-standalone-386
/android/sdk/ndk-bundle/build/tools/make_standalone_toolchain.py --arch x86_64 --install-dir /android/ndk-standalone-amd64

echo no | /android/sdk/tools/bin/avdmanager create avd --force --name android-avd --package "system-images;android-26;google_apis;x86_64"

# run the emulator.
/android/sdk/emulator/emulator @android-avd -no-window &

# wait for it.
/android/sdk/platform-tools/adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;'

Can be started with:

sudo docker run --privileged -it <container-name> /bin/bash

Where I'm able to complete androidtest.sh with android/386 and android/amd64:

/android/run-emulator.sh
cd /go-tip/src
apt-get update && apt-get install git-core
export PATH=$PATH:/android/sdk/platform-tools; GOROOT_BOOTSTRAP=/go1.4 CC_FOR_TARGET=/android/ndk-standalone-386/bin/clang GOARCH=386 ./
androidtest.bash
export PATH=$PATH:/android/sdk/platform-tools; GOROOT_BOOTSTRAP=/go1.4 CC_FOR_TARGET=/android/ndk-standalone-amd64/bin/clang GOARCH=amd6
4 ./androidtest.bash

With these 3 CLs applied:
https://go-review.googlesource.com/c/mobile/+/94195
https://go-review.googlesource.com/c/mobile/+/93998
https://go-review.googlesource.com/c/mobile/+/93997

I can even complete a full test of gomobile:

GOROOT_BOOTSTRAP=/go1.4 ./make.bash
export PATH=$PATH:/go-tip/bin:$HOME/go/bin:/android/sdk/platform-tools:/android/gradle-2.13/bin
export GOPATH=$HOME/go                                                                                                                  
export ANDROID_HOME=/android/sdk                                                                                                        
go get -u golang.org/x/mobile/...                                                                                                       
gomobile init                                                                                                                           
go test -short golang.org/x/mobile/...

Let me know how to proceed.

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Feb 21, 2018

The 3 CLs are now submitted.

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Mar 5, 2018

Ping @bradfitz

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 10, 2018

Sorry, I realistically won't be able to get to start on this until April 1st. I'm preparing for an upcoming Gophercon talk this weekend, traveling next week for the talk, and then on vacation the last week of March.

I might be able to work on this next week from Moscow, but no promises.

@andybons has been wanting to learn how this whole system works more, so this might be a good "starter project" for adding a builder.

BTW, I forgot to ask earlier: how long does this all take? I imagine sharding is necessary for being a trybot, but is your goal for it to be a trybot right away, or just to have it run on GCE by us in Kubernetes? If it's easier, we could skip sharding for the first version.

But I'm also fine with sharding for the first version, but curious about timing. I imagine make.bash is still fast like always (1 minute or less), but then how long are the various tests?

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Mar 11, 2018

Sorry, I realistically won't be able to get to start on this until April 1st. I'm preparing for an upcoming Gophercon talk this weekend, traveling next week for the talk, and then on vacation the last week of March.

No problem at all. My ping was just as much to see if the Docker image was enough.

BTW, I forgot to ask earlier: how long does this all take? I imagine sharding is necessary for being a trybot, but is your goal for it to be a trybot right away, or just to have it run on GCE by us in Kubernetes? If it's easier, we could skip sharding for the first version.

My wishlist ordered by importance:

  1. Run emulated android/386 and android/amd64 builder on GCE (non-sharded or sharded, whatever is convenient for you). That will take a good amount of load off the strained mac mini that runs all mobile builders today. Also, it will ensure a builder is running if the mini is down which happens from time to time.
  2. Use the GCE builder to run the gomobile subrepository tests. There is no meaningful automatic testing for gomobile today, so this would be a great way to get some coverage there.
  3. Android trybot builders, to catch any Android specific issues with CLs before they're submitted.

(And in the future: 4. Do 1-3 for a darwin builder running an 386/amd64 iOS emulator.)

In other words, "just" a regular non-sharded builder on GCE would make me very grateful.

But I'm also fine with sharding for the first version, but curious about timing. I imagine make.bash is still fast like always (1 minute or less), but then how long are the various tests?

On my i7-6700K@4GHz with 32GB memory with a running Android 8 x86_64 emulator, ./androidtest.bash takes ~2min and ./all.bash on the same machine takes ~3min.

BTW, I'm not even sure trybots can be stable enough before #23795 is fixed. You wouldn't happen to know someone to ping for the corresponding adb bug, https://issuetracker.google.com/issues/73230216 ?

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Apr 26, 2018

I just stumpled on #9579. Will the Android emulator run on GCE?

@bradfitz

This comment has been minimized.

Member

bradfitz commented Apr 26, 2018

@bcmills, this is another builder request.

@bradfitz bradfitz added the OS-Android label Apr 26, 2018

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Jun 19, 2018

Gentle ping. If I can do anything to help this, please let me know.

@bcmills

This comment has been minimized.

Member

bcmills commented Jun 19, 2018

I've been focused on vgo reviews and bugs (since that's fairly urgent for keeping 1.11 on schedule). I plan to get back to this after the 1.11 release, but if anybody else would like to take it before then you're welcome to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment