x/build: set up AIX builder

[bradfitz]

Tony Reix (tony.reix@atos.net) requests an AIX builder. He needs a key, builder.go configuration, etc.

(He's been waiting since mid-June, but it got lost in my email.)

Dmitri, can you own this?

[trex58]

I have a AIX 7.2 machine available on OSU and I need some more information for finalizing its configuration in order to make it available to the golang buildfarm. Thx.

[bradfitz]

@dmitshur, what you'll want to do is email him the host key for:

host-aix-ppc64-osuosl

(the same was you made a gomote user token earlier, but with host-* instead of user-*)

And then add that same host config name into x/build/dashboard/builders.go, and then add a aix-ppc64 builder entry referencing that host config host-aix-ppc64-osuosl.

[gopherbot]

Change https://golang.org/cl/131335 mentions this issue: dashboard: add aix-ppc64 builder

[dmitshur]

@trex58 I've sent CL 131335 as a starting point. /cc @bradfitz

We can continue the discussion there. Once the CL is ready and submitted, I suspect I'll need to redeploy cmd/coordinator, while you get your builder to run with the latest cmd/buildlet binary. It should then be able to connect to the coordinator (https://farmer.golang.org/).

[trex58]

Hi Dmitri,

• I've updated the buildlet binary (go get -v -insecure golang.org/x/build/cmd/buildlet).
  It is : /go/bin/buildlet
• I have put the host key in: /.gobuildkey & /.gobuildkey-host-foo-bar
• I have:
  env | grep GO
  GOPATH=/go
  GOCACHE=/tmp/.cache/go-build
  GO_BUILD_KEY_PATH=/.gobuildkey
• And I get:
  buildlet -coordinator=farmer.golang.org -reverse-type=host-foo-bar -reboot=false
  2018/08/28 08:40:39 buildlet starting.
  2018/08/28 08:40:39 Dialing coordinator farmer.golang.org:443 ...
  2018/08/28 08:40:40 Doing TLS handshake with coordinator (verifying hostname "farmer.golang.org")...
  2018/08/28 08:40:40 Registering reverse mode with coordinator...
  2018/08/28 08:40:40 Error dialing coordinator: coordinator registration failed; want HTTP status 101; got 412 Precondition Failed:
  bad key for mode "host-foo-bar"

ping farmer.golang.org
PING farmer.golang.org: (107.178.219.46): 56 data bytes
64 bytes from 107.178.219.46: icmp_seq=0 ttl=45 time=44 ms

What am I doing wrong?

For now, I'm using our RPM golang go version:

go version
go version go1.11rc1 aix/ppc64

rpm -qa | grep golang
golang-bin-1.11-1beta.ppc
golang-misc-1.11-1beta.noarch
golang-src-1.11-1beta.noarch
golang-1.11-1beta.ppc

go env
GOARCH="ppc64"
GOBIN=""
GOCACHE="/tmp/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="ppc64"
GOHOSTOS="aix"
GOOS="aix"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/opt/freeware/lib/golang"
GOTMPDIR=""
GOTOOLDIR="/opt/freeware/lib/golang/pkg/tool/aix_ppc64"
GCCGO="/opt/freeware/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -maix64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build395950374=/tmp/go-build -gno-record-gcc-switches"

[bradfitz]

@trex58, you pasted the private builder key above. I deleted it. Please don't share it publicly.

Your mistake is here -reverse-type=host-foo-bar. "foo bar" is an example placeholder. That's where you put your host config type.

[trex58]

Ooops ! Sorry about showing the key.

Now using:
cat /go/bin/buildletloop
while true
do
buildlet -coordinator=farmer.golang.org -reverse-type=host-aix-ppc64-osuosl -reboot=false
done

Now, I have:
sh buildletloop
2018/08/28 14:29:05 buildlet starting.
2018/08/28 14:29:05 Dialing coordinator farmer.golang.org:443 ...
2018/08/28 14:29:06 Doing TLS handshake with coordinator (verifying hostname "farmer.golang.org")...
2018/08/28 14:29:06 Registering reverse mode with coordinator...
2018/08/28 14:29:06 Connected to coordinator; reverse dialing active
------------- NOTHING MORE
Do you see something on your side?

Since AIX does not have "systemd", I have to figure out how to manage the loop script so that it runs forever and also when the system is rebooted.

[trex58]

I have changed: GOROOT_BOOTSTRAP=/opt/freeware/lib/golang/ . This is what we already use for bootstrapping from the previously installed RPMs of golang build by our .spec file.

I have created a SRC subsystem for /go/bin/buildletloop which will run continuously and will restart at AIX reboot.

[bradfitz]

Great. And https://farmer.golang.org/ now shows your machine connected:

Reverse pool machine detail
power8-aix-host1 (10.240.0.34:36799) version 17, host-aix-ppc64-osuosl: connected 12m22.5s, idle for 6.6s

So now we just need to configure our side to give it work.

[trex58]

Hi Brad.
I guess that, each time some change in golang has to be compiled/checked on my machine, the go buildlet will download code from golang and then put it somewhere on my machine and build/test it. However: where will the files be stored? I need to allocate enough disk place.
I also guess that, once the CL 131335 is merged and file dashboard/builders.go has been updated with knowledge of AIX, I'll have to update it by "go get golang.org/x/build/cmd/buildlet". Correct?
Thx

[trex58]

Hello.
Brad ? Dmitri ?
What is the next step?

[bradfitz]

I'm on vacation for several weeks. Dmitri replied on the CL, IIRC.

[trex58]

Oh ! yes! Sorry. Thx. Enjoy your vacations!

[dmitshur]

@trex58 I hope I've answered your questions in my latest reply in the CL (https://go-review.googlesource.com/c/build/+/131335#message-b4ef0e3ec637c8fb3dd5603bd1397d5b7029aec7).

I'll be updating the coordinator so it's ready to give out work to the buildlet. If there's anything unexpected on your side, or you want to stop things temporarily, just stop the buildlet binary from running and let me know.

[dmitshur]

I've redeployed coordinator with CL 131335 merged. @trex58 Your reverse builder has successfully connected, and the coordinator is giving it work now:

Since this is a new builder, the coordinator will be giving it many commits to test one after another. Let's see keep an eye on it and see how it goes (whether they pass or not). Also let me know if you have any concerns from your side (the builder machine).

[dmitshur]

The first 2 builds failed so far:

We can learn from the build log to see what the problem was:

https://farmer.golang.org/temporarylogs?name=aix-ppc64&rev=24e51bbe64b4a534b096a3b0c6bfae6a732eea59&st=0xc421274fc0

  builder: aix-ppc64
      rev: 24e51bbe64b4a534b096a3b0c6bfae6a732eea59
 buildlet: http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  started: 2018-09-06 19:24:48.939230358 +0000 UTC m=+6.428601975
    ended: 2018-09-06 19:26:29.106765759 +0000 UTC m=+106.596137369
  success: false

Events:
  2018-09-06T19:24:49Z checking_for_snapshot 
  2018-09-06T19:24:49Z finish_checking_for_snapshot after 183ms
  2018-09-06T19:24:49Z get_buildlet 
  2018-09-06T19:24:49Z wait_static_builder host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_wait_static_builder after 0s; host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z clean_buildlet http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_clean_buildlet after 63ms; http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_get_buildlet after 257.9ms
  2018-09-06T19:24:49Z using_buildlet power8-aix-host1
  2018-09-06T19:24:49Z write_version_tar 
  2018-09-06T19:24:49Z get_source 
  2018-09-06T19:24:49Z get_source_from_gitmirror 
  2018-09-06T19:24:53Z finish_get_source_from_gitmirror after 3.52s
  2018-09-06T19:24:53Z finish_get_source after 3.65s
  2018-09-06T19:24:53Z write_go_src_tar 
  2018-09-06T19:26:23Z finish_write_go_src_tar after 1m29.4s
  2018-09-06T19:26:23Z make_and_test 
  2018-09-06T19:26:23Z make src/make.bash
  2018-09-06T19:26:28Z finish_make after 5.38s; err=exit status 1 (core dumped); src/make.bash
  2018-09-06T19:26:28Z finish_make_and_test after 5.45s
  2018-09-06T19:26:28Z done with test failures

Build log:
aix-ppc64 at 24e51bbe64b4a534b096a3b0c6bfae6a732eea59

:: Running /tmp/workdir-host-aix-ppc64-osuosl/go/src/make.bash with args ["/tmp/workdir-host-aix-ppc64-osuosl/go/src/make.bash"] and env ["_=/go/bin/buildlet" "LANG=en_US" "LOGIN=root" "CLCMD_PASSTHRU=1" "PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin" "LC__FASTMSG=true" "LOGNAME=root" "LOCPATH=/usr/lib/nls/loc" "USER=root" "AUTHSTATE=compat" "SHELL=/usr/bin/ksh" "ODMDIR=/etc/objrepos" "HOME=/" "TERM=dumb" "PWD=/" "TZ=CST6CDT" "NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat:/usr/lib/nls/msg/%l.%c/%N:/usr/lib/nls/msg/%l.%c/%N.cat" "WORKDIR=/tmp/workdir-host-aix-ppc64-osuosl" "GOROOT_BOOTSTRAP=/opt/freeware/lib/golang" "GO_BUILDER_NAME=aix-ppc64" "GOBIN="] in dir /tmp/workdir-host-aix-ppc64-osuosl/go/src

Building Go cmd/dist using /opt/freeware/lib/golang.
go tool dist: unknown architecture: 00F9C1964C00



Error: build failed: make script failed: exit status 1 (core dumped)

It looks like it's failing at building Go using the given bootstrap Go (in /opt/freeware/lib/golang). Can you see if you can figure out why that's happening inside the builder machine? Is there a working version of Go 1.4 in /opt/freeware/lib/golang?

[bradfitz]

uname -m returns 00F9C1964C00 on AIX? That's unexpected (to me, at least).

In any case, until any AIX is in-tree, you should expect all builds to fail, and that's okay. That's exactly what https://golang.org/wiki/PortingPolicy says the path should be.

[dmitshur]

Oh, I didn't realize this was a new (not yet implemented) port. In that case, build failures in the short term are expected, but please let us know if it's failing in an unexpected point and you need help getting it past that point.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)