Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: linux-arm-arm5spacemonkey persistently broken #29177

Closed
bcmills opened this issue Dec 11, 2018 · 8 comments

Comments

Projects
None yet
5 participants
@bcmills
Copy link
Member

commented Dec 11, 2018

The linux-arm-arm5spacemonkey builder has been broken for as far back as I dared to page through the logs. It seems to be the only all-red (#22689) builder in the “first-class ports” view (#27689).

As far as I can tell, it's a configuration issue: for some reason the builder sets the Go version to builder host-linux-armel-cross instead of a more realistic version string, and that somehow causes the API check to fail.

See previously #28041. (CC @bradfitz, @onionjake, @zeebo, @dmitshur )

Sample: https://build.golang.org/log/32dafc31f49b66f75902acd5f0d61cb8f6277de6

:: Running /bin/bash with args ["/bin/bash" "-c" "cd $WORKDIR/go/src && ./make.bash && cd .. && mv bin/*_*/* bin && rmdir bin/*_* && rm -rf pkg/linux_amd64 pkg/tool/linux_amd64 pkg/bootstrap pkg/obj"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-armel-cross-rn8aed9e5" "IN_KUBERNETES=1" "META_BUILDLET_BINARY_URL=http://storage.googleapis.com/go-builder-data/buildlet.linux-amd64" "META_BUILDLET_HOST_TYPE=host-linux-armel-cross" "KUBERNETES_PORT_443_TCP_PORT=443" "KUBERNETES_PORT_443_TCP_ADDR=10.3.240.1" "KUBERNETES_SERVICE_HOST=10.3.240.1" "KUBERNETES_SERVICE_PORT=443" "KUBERNETES_SERVICE_PORT_HTTPS=443" "KUBERNETES_PORT=tcp://10.3.240.1:443" "KUBERNETES_PORT_443_TCP=tcp://10.3.240.1:443" "KUBERNETES_PORT_443_TCP_PROTO=tcp" "DEBIAN_FRONTEND=noninteractive" "GOROOT_BOOTSTRAP=/go1.4" "GOOS=linux" "GOARCH=arm" "CC_FOR_TARGET=arm-linux-gnueabi-gcc" "HOME=/root" "WORKDIR=/tmp/workdir" "CGO_ENABLED=1" "GOARM=5"] in dir /tmp/workdir

Building Go cmd/dist using /go1.4.
warning: changing VERSION from "devel 552d7b918697e0a5ab1ff22f91abf3e1cfde2e1d" to "builder host-linux-armel-cross"
Building Go toolchain1 using /go1.4.
warning: changing VERSION from "devel 552d7b918697e0a5ab1ff22f91abf3e1cfde2e1d" to "builder host-linux-armel-cross"
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
warning: changing VERSION from "devel 552d7b918697e0a5ab1ff22f91abf3e1cfde2e1d" to "builder host-linux-armel-cross"
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, linux/amd64.
Building packages and commands for target, linux/arm.
[…]
##### API check
Error running API checker: exit status 1
Go version is "builder host-linux-armel-cross", ignoring -next /home/builder/stage0scratch/workdir/go/api/next.txt
+pkg bytes, func ReplaceAll([]uint8, []uint8, []uint8) []uint8
[…]
+pkg syscall, type RawSockaddrUnix struct
exit status 1
2018/11/26 19:20:01 Failed: exit status 1
2018/11/26 19:20:07 FAILED

@gopherbot gopherbot added this to the Unreleased milestone Dec 11, 2018

@gopherbot gopherbot added the Builders label Dec 11, 2018

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2018

For the record, if I remembered correctly, that builder is special in that it does cross compilation. The toolchain and the standard library is built on AMD64 machine(s) then shipped to the ARMv5 machine(s) to run the tests. When @rsc implemented the build cache, it would never work in this setting (failure was probably toolchain2 and toolchain3 were different). (And it is not possible to run everything on the ARMv5 machine because it is too slow.) So we decided to set up a toolchain version, to bypass certain checks.

@mvdan

This comment has been minimized.

Copy link
Member

commented Dec 12, 2018

Seems like a duplicate of #24754, initially titled cmd/api: failing on linux-arm builder due to non-devel Go version. Now it has a title for a proposed fix, as far as I understand.

@onionjake

This comment has been minimized.

Copy link

commented Dec 12, 2018

I'm happy to help if there is something we can do on the builders to make this easier.

@onionjake

This comment has been minimized.

Copy link

commented Feb 11, 2019

Is this resolved now? Or is this timeout issue masking it?

This is from one of the latest test runs (https://build.golang.org/log/5137b5ef408a5c5a56d13ae902fe833f501d1e12):

Test "cgo_test" ran over 20m0s limit (20m0.040245806s)
@gopherbot

This comment has been minimized.

Copy link

commented Feb 14, 2019

Change https://golang.org/cl/162540 mentions this issue: cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEOUT_SCALE

gopherbot pushed a commit to golang/build that referenced this issue Feb 14, 2019

cmd/coordinator, dashboard: scale test exec timeouts by GO_TEST_TIMEO…
…UT_SCALE

So slow builders can finish their cgo tests, etc.

Updates golang/go#29177 etc

Change-Id: If5ee8d2201dc53ab506d4f5e31406d6aae240352
Reviewed-on: https://go-review.googlesource.com/c/162540
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@bcmills

This comment has been minimized.

Copy link
Member Author

commented Feb 28, 2019

This builder is still persistently broken. Now it appears to be ignoring the next file in the API check.

From https://build.golang.org/log/4728fbb67eb490e9f3df8c5553e16c14fc7cfaf2:

##### API check
Error running API checker: exit status 1
Go version is "builder host-linux-armel-cross", ignoring -next /home/builder/stage0scratch/workdir/go/api/next.txt
+pkg errors, func As(error, interface{}) bool
+pkg errors, func Caller(int) Frame
+pkg errors, func Is(error, error) bool
+pkg errors, func Opaque(error) error
+pkg errors, func Unwrap(error) error
+pkg errors, method (Frame) Format(Printer)
+pkg errors, type Formatter interface { Error, FormatError }
+pkg errors, type Formatter interface, Error() string
+pkg errors, type Formatter interface, FormatError(Printer) error
+pkg errors, type Frame struct
+pkg errors, type Printer interface { Detail, Print, Printf }
+pkg errors, type Printer interface, Detail() bool
+pkg errors, type Printer interface, Print(...interface{})
+pkg errors, type Printer interface, Printf(string, ...interface{})
+pkg errors, type Wrapper interface { Unwrap }
+pkg errors, type Wrapper interface, Unwrap() error
+pkg text/scanner, const AllowNumberbars = 1024
+pkg text/scanner, const AllowNumberbars ideal-int
+pkg text/scanner, const GoTokens = 2036
exit status 1
@mvdan

This comment has been minimized.

Copy link
Member

commented Feb 28, 2019

@bcmills #24754 talks precisely about the API check and how the version name in that builder makes the test fail. So I still think this is a duplicate.

@bcmills

This comment has been minimized.

Copy link
Member Author

commented Feb 28, 2019

Agreed. Also #24789.

@bcmills bcmills closed this Feb 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.