Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: set up AIX builder #27160

Closed
bradfitz opened this issue Aug 22, 2018 · 19 comments

Comments

Projects
None yet
4 participants
@bradfitz
Copy link
Member

commented Aug 22, 2018

Tony Reix (tony.reix@atos.net) requests an AIX builder. He needs a key, builder.go configuration, etc.

(He's been waiting since mid-June, but it got lost in my email.)

Dmitri, can you own this?

@bradfitz bradfitz added the NeedsFix label Aug 22, 2018

@gopherbot gopherbot added this to the Unreleased milestone Aug 22, 2018

@gopherbot gopherbot added the Builders label Aug 22, 2018

@bradfitz bradfitz changed the title x/build: setup AIX builder x/build: set up AIX builder Aug 22, 2018

@trex58

This comment has been minimized.

Copy link

commented Aug 23, 2018

I have a AIX 7.2 machine available on OSU and I need some more information for finalizing its configuration in order to make it available to the golang buildfarm. Thx.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Aug 23, 2018

@dmitshur, what you'll want to do is email him the host key for:

host-aix-ppc64-osuosl

(the same was you made a gomote user token earlier, but with host-* instead of user-*)

And then add that same host config name into x/build/dashboard/builders.go, and then add a aix-ppc64 builder entry referencing that host config host-aix-ppc64-osuosl.

@gopherbot

This comment has been minimized.

Copy link

commented Aug 24, 2018

Change https://golang.org/cl/131335 mentions this issue: dashboard: add aix-ppc64 builder

@dmitshur

This comment has been minimized.

Copy link
Member

commented Aug 24, 2018

@trex58 I've sent CL 131335 as a starting point. /cc @bradfitz

We can continue the discussion there. Once the CL is ready and submitted, I suspect I'll need to redeploy cmd/coordinator, while you get your builder to run with the latest cmd/buildlet binary. It should then be able to connect to the coordinator (https://farmer.golang.org/).

@trex58

This comment has been minimized.

Copy link

commented Aug 28, 2018

Hi Dmitri,

  • I've updated the buildlet binary (go get -v -insecure golang.org/x/build/cmd/buildlet).
    It is : /go/bin/buildlet
  • I have put the host key in: /.gobuildkey & /.gobuildkey-host-foo-bar
  • I have:
    env | grep GO
    GOPATH=/go
    GOCACHE=/tmp/.cache/go-build
    GO_BUILD_KEY_PATH=/.gobuildkey
  • And I get:
    buildlet -coordinator=farmer.golang.org -reverse-type=host-foo-bar -reboot=false
    2018/08/28 08:40:39 buildlet starting.
    2018/08/28 08:40:39 Dialing coordinator farmer.golang.org:443 ...
    2018/08/28 08:40:40 Doing TLS handshake with coordinator (verifying hostname "farmer.golang.org")...
    2018/08/28 08:40:40 Registering reverse mode with coordinator...
    2018/08/28 08:40:40 Error dialing coordinator: coordinator registration failed; want HTTP status 101; got 412 Precondition Failed:
    bad key for mode "host-foo-bar"

ping farmer.golang.org
PING farmer.golang.org: (107.178.219.46): 56 data bytes
64 bytes from 107.178.219.46: icmp_seq=0 ttl=45 time=44 ms

What am I doing wrong?

For now, I'm using our RPM golang go version:

go version
go version go1.11rc1 aix/ppc64

rpm -qa | grep golang
golang-bin-1.11-1beta.ppc
golang-misc-1.11-1beta.noarch
golang-src-1.11-1beta.noarch
golang-1.11-1beta.ppc

go env
GOARCH="ppc64"
GOBIN=""
GOCACHE="/tmp/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="ppc64"
GOHOSTOS="aix"
GOOS="aix"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/opt/freeware/lib/golang"
GOTMPDIR=""
GOTOOLDIR="/opt/freeware/lib/golang/pkg/tool/aix_ppc64"
GCCGO="/opt/freeware/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -maix64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build395950374=/tmp/go-build -gno-record-gcc-switches"

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Aug 28, 2018

@trex58, you pasted the private builder key above. I deleted it. Please don't share it publicly.

Your mistake is here -reverse-type=host-foo-bar. "foo bar" is an example placeholder. That's where you put your host config type.

@trex58

This comment has been minimized.

Copy link

commented Aug 28, 2018

Ooops ! Sorry about showing the key.

Now using:
cat /go/bin/buildletloop
while true
do
buildlet -coordinator=farmer.golang.org -reverse-type=host-aix-ppc64-osuosl -reboot=false
done

Now, I have:
sh buildletloop
2018/08/28 14:29:05 buildlet starting.
2018/08/28 14:29:05 Dialing coordinator farmer.golang.org:443 ...
2018/08/28 14:29:06 Doing TLS handshake with coordinator (verifying hostname "farmer.golang.org")...
2018/08/28 14:29:06 Registering reverse mode with coordinator...
2018/08/28 14:29:06 Connected to coordinator; reverse dialing active
------------- NOTHING MORE
Do you see something on your side?

Since AIX does not have "systemd", I have to figure out how to manage the loop script so that it runs forever and also when the system is rebooted.

@trex58

This comment has been minimized.

Copy link

commented Aug 28, 2018

I have changed: GOROOT_BOOTSTRAP=/opt/freeware/lib/golang/ . This is what we already use for bootstrapping from the previously installed RPMs of golang build by our .spec file.

I have created a SRC subsystem for /go/bin/buildletloop which will run continuously and will restart at AIX reboot.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Aug 28, 2018

Great. And https://farmer.golang.org/ now shows your machine connected:

Reverse pool machine detail
power8-aix-host1 (10.240.0.34:36799) version 17, host-aix-ppc64-osuosl: connected 12m22.5s, idle for 6.6s

So now we just need to configure our side to give it work.

@trex58

This comment has been minimized.

Copy link

commented Aug 29, 2018

Hi Brad.
I guess that, each time some change in golang has to be compiled/checked on my machine, the go buildlet will download code from golang and then put it somewhere on my machine and build/test it. However: where will the files be stored? I need to allocate enough disk place.
I also guess that, once the CL 131335 is merged and file dashboard/builders.go has been updated with knowledge of AIX, I'll have to update it by "go get golang.org/x/build/cmd/buildlet". Correct?
Thx

@trex58

This comment has been minimized.

Copy link

commented Sep 6, 2018

Hello.
Brad ? Dmitri ?
What is the next step?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Sep 6, 2018

I'm on vacation for several weeks. Dmitri replied on the CL, IIRC.

@trex58

This comment has been minimized.

Copy link

commented Sep 6, 2018

Oh ! yes! Sorry. Thx. Enjoy your vacations!

@dmitshur

This comment has been minimized.

Copy link
Member

commented Sep 6, 2018

@trex58 I hope I've answered your questions in my latest reply in the CL (https://go-review.googlesource.com/c/build/+/131335#message-b4ef0e3ec637c8fb3dd5603bd1397d5b7029aec7).

I'll be updating the coordinator so it's ready to give out work to the buildlet. If there's anything unexpected on your side, or you want to stop things temporarily, just stop the buildlet binary from running and let me know.

gopherbot pushed a commit to golang/build that referenced this issue Sep 6, 2018

dashboard: add aix-ppc64 builder
This is a starting point for adding an aix-ppc64 reverse builder.
There can be further adjustments made to it as needed.

Updates golang/go#27160.

Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508
Reviewed-on: https://go-review.googlesource.com/131335
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@dmitshur

This comment has been minimized.

Copy link
Member

commented Sep 6, 2018

I've redeployed coordinator with CL 131335 merged. @trex58 Your reverse builder has successfully connected, and the coordinator is giving it work now:

image

Since this is a new builder, the coordinator will be giving it many commits to test one after another. Let's see keep an eye on it and see how it goes (whether they pass or not). Also let me know if you have any concerns from your side (the builder machine).

@dmitshur

This comment has been minimized.

Copy link
Member

commented Sep 6, 2018

The first 2 builds failed so far:

image

We can learn from the build log to see what the problem was:

https://farmer.golang.org/temporarylogs?name=aix-ppc64&rev=24e51bbe64b4a534b096a3b0c6bfae6a732eea59&st=0xc421274fc0

  builder: aix-ppc64
      rev: 24e51bbe64b4a534b096a3b0c6bfae6a732eea59
 buildlet: http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  started: 2018-09-06 19:24:48.939230358 +0000 UTC m=+6.428601975
    ended: 2018-09-06 19:26:29.106765759 +0000 UTC m=+106.596137369
  success: false

Events:
  2018-09-06T19:24:49Z checking_for_snapshot 
  2018-09-06T19:24:49Z finish_checking_for_snapshot after 183ms
  2018-09-06T19:24:49Z get_buildlet 
  2018-09-06T19:24:49Z wait_static_builder host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_wait_static_builder after 0s; host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z clean_buildlet http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_clean_buildlet after 63ms; http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_get_buildlet after 257.9ms
  2018-09-06T19:24:49Z using_buildlet power8-aix-host1
  2018-09-06T19:24:49Z write_version_tar 
  2018-09-06T19:24:49Z get_source 
  2018-09-06T19:24:49Z get_source_from_gitmirror 
  2018-09-06T19:24:53Z finish_get_source_from_gitmirror after 3.52s
  2018-09-06T19:24:53Z finish_get_source after 3.65s
  2018-09-06T19:24:53Z write_go_src_tar 
  2018-09-06T19:26:23Z finish_write_go_src_tar after 1m29.4s
  2018-09-06T19:26:23Z make_and_test 
  2018-09-06T19:26:23Z make src/make.bash
  2018-09-06T19:26:28Z finish_make after 5.38s; err=exit status 1 (core dumped); src/make.bash
  2018-09-06T19:26:28Z finish_make_and_test after 5.45s
  2018-09-06T19:26:28Z done with test failures

Build log:
aix-ppc64 at 24e51bbe64b4a534b096a3b0c6bfae6a732eea59

:: Running /tmp/workdir-host-aix-ppc64-osuosl/go/src/make.bash with args ["/tmp/workdir-host-aix-ppc64-osuosl/go/src/make.bash"] and env ["_=/go/bin/buildlet" "LANG=en_US" "LOGIN=root" "CLCMD_PASSTHRU=1" "PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin" "LC__FASTMSG=true" "LOGNAME=root" "LOCPATH=/usr/lib/nls/loc" "USER=root" "AUTHSTATE=compat" "SHELL=/usr/bin/ksh" "ODMDIR=/etc/objrepos" "HOME=/" "TERM=dumb" "PWD=/" "TZ=CST6CDT" "NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat:/usr/lib/nls/msg/%l.%c/%N:/usr/lib/nls/msg/%l.%c/%N.cat" "WORKDIR=/tmp/workdir-host-aix-ppc64-osuosl" "GOROOT_BOOTSTRAP=/opt/freeware/lib/golang" "GO_BUILDER_NAME=aix-ppc64" "GOBIN="] in dir /tmp/workdir-host-aix-ppc64-osuosl/go/src

Building Go cmd/dist using /opt/freeware/lib/golang.
go tool dist: unknown architecture: 00F9C1964C00



Error: build failed: make script failed: exit status 1 (core dumped)

It looks like it's failing at building Go using the given bootstrap Go (in /opt/freeware/lib/golang). Can you see if you can figure out why that's happening inside the builder machine? Is there a working version of Go 1.4 in /opt/freeware/lib/golang?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Sep 6, 2018

uname -m returns 00F9C1964C00 on AIX? That's unexpected (to me, at least).

In any case, until any AIX is in-tree, you should expect all builds to fail, and that's okay. That's exactly what https://golang.org/wiki/PortingPolicy says the path should be.

@dmitshur

This comment has been minimized.

Copy link
Member

commented Sep 6, 2018

Oh, I didn't realize this was a new (not yet implemented) port. In that case, build failures in the short term are expected, but please let us know if it's failing in an unexpected point and you need help getting it past that point.

@dmitshur dmitshur added WaitingForInfo and removed NeedsFix labels Sep 19, 2018

@gopherbot

This comment has been minimized.

Copy link

commented Oct 19, 2018

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this Oct 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.