Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: set up AIX builder #27160

Closed
bradfitz opened this issue Aug 22, 2018 · 19 comments
Closed

x/build: set up AIX builder #27160

bradfitz opened this issue Aug 22, 2018 · 19 comments

Comments

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Aug 22, 2018

Tony Reix (tony.reix@atos.net) requests an AIX builder. He needs a key, builder.go configuration, etc.

(He's been waiting since mid-June, but it got lost in my email.)

Dmitri, can you own this?

@bradfitz bradfitz added the NeedsFix label Aug 22, 2018
@gopherbot gopherbot added this to the Unreleased milestone Aug 22, 2018
@gopherbot gopherbot added the Builders label Aug 22, 2018
@bradfitz bradfitz changed the title x/build: setup AIX builder x/build: set up AIX builder Aug 22, 2018
@trex58
Copy link

@trex58 trex58 commented Aug 23, 2018

I have a AIX 7.2 machine available on OSU and I need some more information for finalizing its configuration in order to make it available to the golang buildfarm. Thx.

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Aug 23, 2018

@dmitshur, what you'll want to do is email him the host key for:

host-aix-ppc64-osuosl

(the same was you made a gomote user token earlier, but with host-* instead of user-*)

And then add that same host config name into x/build/dashboard/builders.go, and then add a aix-ppc64 builder entry referencing that host config host-aix-ppc64-osuosl.

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 24, 2018

Change https://golang.org/cl/131335 mentions this issue: dashboard: add aix-ppc64 builder

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Aug 24, 2018

@trex58 I've sent CL 131335 as a starting point. /cc @bradfitz

We can continue the discussion there. Once the CL is ready and submitted, I suspect I'll need to redeploy cmd/coordinator, while you get your builder to run with the latest cmd/buildlet binary. It should then be able to connect to the coordinator (https://farmer.golang.org/).

@trex58
Copy link

@trex58 trex58 commented Aug 28, 2018

Hi Dmitri,

  • I've updated the buildlet binary (go get -v -insecure golang.org/x/build/cmd/buildlet).
    It is : /go/bin/buildlet
  • I have put the host key in: /.gobuildkey & /.gobuildkey-host-foo-bar
  • I have:
    env | grep GO
    GOPATH=/go
    GOCACHE=/tmp/.cache/go-build
    GO_BUILD_KEY_PATH=/.gobuildkey
  • And I get:
    buildlet -coordinator=farmer.golang.org -reverse-type=host-foo-bar -reboot=false
    2018/08/28 08:40:39 buildlet starting.
    2018/08/28 08:40:39 Dialing coordinator farmer.golang.org:443 ...
    2018/08/28 08:40:40 Doing TLS handshake with coordinator (verifying hostname "farmer.golang.org")...
    2018/08/28 08:40:40 Registering reverse mode with coordinator...
    2018/08/28 08:40:40 Error dialing coordinator: coordinator registration failed; want HTTP status 101; got 412 Precondition Failed:
    bad key for mode "host-foo-bar"

ping farmer.golang.org
PING farmer.golang.org: (107.178.219.46): 56 data bytes
64 bytes from 107.178.219.46: icmp_seq=0 ttl=45 time=44 ms

What am I doing wrong?

For now, I'm using our RPM golang go version:

go version
go version go1.11rc1 aix/ppc64

rpm -qa | grep golang
golang-bin-1.11-1beta.ppc
golang-misc-1.11-1beta.noarch
golang-src-1.11-1beta.noarch
golang-1.11-1beta.ppc

go env
GOARCH="ppc64"
GOBIN=""
GOCACHE="/tmp/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="ppc64"
GOHOSTOS="aix"
GOOS="aix"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/opt/freeware/lib/golang"
GOTMPDIR=""
GOTOOLDIR="/opt/freeware/lib/golang/pkg/tool/aix_ppc64"
GCCGO="/opt/freeware/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -maix64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build395950374=/tmp/go-build -gno-record-gcc-switches"

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Aug 28, 2018

@trex58, you pasted the private builder key above. I deleted it. Please don't share it publicly.

Your mistake is here -reverse-type=host-foo-bar. "foo bar" is an example placeholder. That's where you put your host config type.

@trex58
Copy link

@trex58 trex58 commented Aug 28, 2018

Ooops ! Sorry about showing the key.

Now using:
cat /go/bin/buildletloop
while true
do
buildlet -coordinator=farmer.golang.org -reverse-type=host-aix-ppc64-osuosl -reboot=false
done

Now, I have:
sh buildletloop
2018/08/28 14:29:05 buildlet starting.
2018/08/28 14:29:05 Dialing coordinator farmer.golang.org:443 ...
2018/08/28 14:29:06 Doing TLS handshake with coordinator (verifying hostname "farmer.golang.org")...
2018/08/28 14:29:06 Registering reverse mode with coordinator...
2018/08/28 14:29:06 Connected to coordinator; reverse dialing active
------------- NOTHING MORE
Do you see something on your side?

Since AIX does not have "systemd", I have to figure out how to manage the loop script so that it runs forever and also when the system is rebooted.

@trex58
Copy link

@trex58 trex58 commented Aug 28, 2018

I have changed: GOROOT_BOOTSTRAP=/opt/freeware/lib/golang/ . This is what we already use for bootstrapping from the previously installed RPMs of golang build by our .spec file.

I have created a SRC subsystem for /go/bin/buildletloop which will run continuously and will restart at AIX reboot.

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Aug 28, 2018

Great. And https://farmer.golang.org/ now shows your machine connected:

Reverse pool machine detail
power8-aix-host1 (10.240.0.34:36799) version 17, host-aix-ppc64-osuosl: connected 12m22.5s, idle for 6.6s

So now we just need to configure our side to give it work.

@trex58
Copy link

@trex58 trex58 commented Aug 29, 2018

Hi Brad.
I guess that, each time some change in golang has to be compiled/checked on my machine, the go buildlet will download code from golang and then put it somewhere on my machine and build/test it. However: where will the files be stored? I need to allocate enough disk place.
I also guess that, once the CL 131335 is merged and file dashboard/builders.go has been updated with knowledge of AIX, I'll have to update it by "go get golang.org/x/build/cmd/buildlet". Correct?
Thx

@trex58
Copy link

@trex58 trex58 commented Sep 6, 2018

Hello.
Brad ? Dmitri ?
What is the next step?

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Sep 6, 2018

I'm on vacation for several weeks. Dmitri replied on the CL, IIRC.

@trex58
Copy link

@trex58 trex58 commented Sep 6, 2018

Oh ! yes! Sorry. Thx. Enjoy your vacations!

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Sep 6, 2018

@trex58 I hope I've answered your questions in my latest reply in the CL (https://go-review.googlesource.com/c/build/+/131335#message-b4ef0e3ec637c8fb3dd5603bd1397d5b7029aec7).

I'll be updating the coordinator so it's ready to give out work to the buildlet. If there's anything unexpected on your side, or you want to stop things temporarily, just stop the buildlet binary from running and let me know.

gopherbot pushed a commit to golang/build that referenced this issue Sep 6, 2018
This is a starting point for adding an aix-ppc64 reverse builder.
There can be further adjustments made to it as needed.

Updates golang/go#27160.

Change-Id: I8a61adbe21f1a08265df9a206d94bb04e880a508
Reviewed-on: https://go-review.googlesource.com/131335
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Sep 6, 2018

I've redeployed coordinator with CL 131335 merged. @trex58 Your reverse builder has successfully connected, and the coordinator is giving it work now:

image

Since this is a new builder, the coordinator will be giving it many commits to test one after another. Let's see keep an eye on it and see how it goes (whether they pass or not). Also let me know if you have any concerns from your side (the builder machine).

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Sep 6, 2018

The first 2 builds failed so far:

image

We can learn from the build log to see what the problem was:

https://farmer.golang.org/temporarylogs?name=aix-ppc64&rev=24e51bbe64b4a534b096a3b0c6bfae6a732eea59&st=0xc421274fc0

  builder: aix-ppc64
      rev: 24e51bbe64b4a534b096a3b0c6bfae6a732eea59
 buildlet: http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  started: 2018-09-06 19:24:48.939230358 +0000 UTC m=+6.428601975
    ended: 2018-09-06 19:26:29.106765759 +0000 UTC m=+106.596137369
  success: false

Events:
  2018-09-06T19:24:49Z checking_for_snapshot 
  2018-09-06T19:24:49Z finish_checking_for_snapshot after 183ms
  2018-09-06T19:24:49Z get_buildlet 
  2018-09-06T19:24:49Z wait_static_builder host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_wait_static_builder after 0s; host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z clean_buildlet http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_clean_buildlet after 63ms; http://power8-aix-host1 reverse peer power8-aix-host1/10.240.0.9:37881 for host type host-aix-ppc64-osuosl
  2018-09-06T19:24:49Z finish_get_buildlet after 257.9ms
  2018-09-06T19:24:49Z using_buildlet power8-aix-host1
  2018-09-06T19:24:49Z write_version_tar 
  2018-09-06T19:24:49Z get_source 
  2018-09-06T19:24:49Z get_source_from_gitmirror 
  2018-09-06T19:24:53Z finish_get_source_from_gitmirror after 3.52s
  2018-09-06T19:24:53Z finish_get_source after 3.65s
  2018-09-06T19:24:53Z write_go_src_tar 
  2018-09-06T19:26:23Z finish_write_go_src_tar after 1m29.4s
  2018-09-06T19:26:23Z make_and_test 
  2018-09-06T19:26:23Z make src/make.bash
  2018-09-06T19:26:28Z finish_make after 5.38s; err=exit status 1 (core dumped); src/make.bash
  2018-09-06T19:26:28Z finish_make_and_test after 5.45s
  2018-09-06T19:26:28Z done with test failures

Build log:
aix-ppc64 at 24e51bbe64b4a534b096a3b0c6bfae6a732eea59

:: Running /tmp/workdir-host-aix-ppc64-osuosl/go/src/make.bash with args ["/tmp/workdir-host-aix-ppc64-osuosl/go/src/make.bash"] and env ["_=/go/bin/buildlet" "LANG=en_US" "LOGIN=root" "CLCMD_PASSTHRU=1" "PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java7_64/jre/bin:/usr/java7_64/bin" "LC__FASTMSG=true" "LOGNAME=root" "LOCPATH=/usr/lib/nls/loc" "USER=root" "AUTHSTATE=compat" "SHELL=/usr/bin/ksh" "ODMDIR=/etc/objrepos" "HOME=/" "TERM=dumb" "PWD=/" "TZ=CST6CDT" "NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat:/usr/lib/nls/msg/%l.%c/%N:/usr/lib/nls/msg/%l.%c/%N.cat" "WORKDIR=/tmp/workdir-host-aix-ppc64-osuosl" "GOROOT_BOOTSTRAP=/opt/freeware/lib/golang" "GO_BUILDER_NAME=aix-ppc64" "GOBIN="] in dir /tmp/workdir-host-aix-ppc64-osuosl/go/src

Building Go cmd/dist using /opt/freeware/lib/golang.
go tool dist: unknown architecture: 00F9C1964C00



Error: build failed: make script failed: exit status 1 (core dumped)

It looks like it's failing at building Go using the given bootstrap Go (in /opt/freeware/lib/golang). Can you see if you can figure out why that's happening inside the builder machine? Is there a working version of Go 1.4 in /opt/freeware/lib/golang?

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Sep 6, 2018

uname -m returns 00F9C1964C00 on AIX? That's unexpected (to me, at least).

In any case, until any AIX is in-tree, you should expect all builds to fail, and that's okay. That's exactly what https://golang.org/wiki/PortingPolicy says the path should be.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Sep 6, 2018

Oh, I didn't realize this was a new (not yet implemented) port. In that case, build failures in the short term are expected, but please let us know if it's failing in an unexpected point and you need help getting it past that point.

@dmitshur dmitshur added WaitingForInfo and removed NeedsFix labels Sep 19, 2018
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 19, 2018

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this Oct 19, 2018
@golang golang locked and limited conversation to collaborators Oct 19, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.