Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: s390x builder not receiving/detecting new builds #27182

Closed
billotosyr opened this issue Aug 23, 2018 · 16 comments

Comments

Projects
None yet
4 participants
@billotosyr
Copy link

commented Aug 23, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

master tip (1.12)

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="s390x"
GOBIN=""
GOEXE=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/data/golang/go"
GOTOOLDIR="/data/golang/go/pkg/tool/linux_s390x"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build637051903=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

loaded the build dashboard

What did you expect to see?

s390x building with all the new CL's

What did you see instead?

Since yesterday (Aug. 22) it's been silent. The buildbot is still running normally (as far as I can tell) but no builds are happening. Suggestions?

@billotosyr

This comment has been minimized.

Copy link
Author

commented Aug 24, 2018

@bradfitz bradfitz changed the title build: s390x builder not receiving/detecting new builds x/build: s390x builder not receiving/detecting new builds Aug 24, 2018

@gopherbot gopherbot added this to the Unreleased milestone Aug 24, 2018

@gopherbot gopherbot added the Builders label Aug 24, 2018

@dmitshur

This comment has been minimized.

Copy link
Member

commented Aug 24, 2018

I see 23 entries of linux-s390x-ibm under active builds. They're all started anywhere between 51 and 46 hours ago, and stuck on this step:

waiting_machine_in_use 

That might (or might not) be related.

@dmitshur

This comment has been minimized.

Copy link
Member

commented Aug 24, 2018

The buildbot is still running normally (as far as I can tell) but no builds are happening. Suggestions?

@billotosyr @mundaym Can you try restarting it (the buildlet process)? Perhaps it's running but disconnected, and hasn't detected that.

You can also try updating to the latest version (i.e., go get -u golang.org/x/build/cmd/buildlet).

@billotosyr

This comment has been minimized.

Copy link
Author

commented Aug 24, 2018

restarted buildlet process - doesn't seem to have had any effect.

@billotosyr

This comment has been minimized.

Copy link
Author

commented Aug 24, 2018

I'm pretty sure its not a version thing -- as the buildlet was working up untl a benign unrelated CL yesterday ("skip TestGcSys on Windows").
Our machine had run out of space, we fixed it, and it started building again, but somehow the queue is too big perhaps? It just seems clogged. Any thoughts on getting it unclogged?

@dmitshur

This comment has been minimized.

Copy link
Member

commented Aug 24, 2018

Our machine had run out of space, we fixed it, and it started building again, but somehow the queue is too big perhaps? It just seems clogged. Any thoughts on getting it unclogged?

Great to hear that it's running again!

Do you know if the machine ran out of space due to the Go builder work, or was it for unrelated reasons?

About the queue, after talking to @bradfitz about it, I don't think there's an existing way to clear it out. Sorry about that. We have an open bug for a scheduler (#19178) and that would be helpful here (e.g., prioritize more recent commits). Let's keep an eye on how the builder runs, and I'll investigate what else can be done to help.

@billotosyr

This comment has been minimized.

Copy link
Author

commented Aug 27, 2018

Specifically, it's build cache that is filling up the drive. My current workaround is to use
'go clean -cache '
as root, frequently. Perhaps I will have to turn off the build cache.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Aug 27, 2018

I thought the buildlet wiped its work directory environment per run?

@billotosyr

This comment has been minimized.

Copy link
Author

commented Aug 29, 2018

Our buildlet doesn't do that. Maybe because it's an older version.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Aug 29, 2018

No, I was just misremembering. But if that helps your environment & wrapper process that's managing keeping it alive, we could add it to the buildlet.

@billotosyr

This comment has been minimized.

Copy link
Author

commented Aug 29, 2018

Well, I think it would help us, yes. The build cache is consuming about 2GB per day, and the drive it's on is small enough that I am now cleaning it daily. (This is a new thing, as I never had to clean it before.)

@gopherbot

This comment has been minimized.

Copy link

commented Oct 25, 2018

Change https://golang.org/cl/144637 mentions this issue: cmd/buildlet: set up & clean TMPDIR and GOCACHE for child processes

@bradfitz

This comment has been minimized.

Copy link
Member

commented Oct 25, 2018

Okay, this the new s390x buildlet binary is pushed.

I'll watch it for any new problems.

It'll keep itself cleaned for new stuff, but you might have to clean legacy messes.

@billotosyr

This comment has been minimized.

Copy link
Author

commented Oct 25, 2018

@bradfitz

This comment has been minimized.

Copy link
Member

commented Oct 25, 2018

Hmm, weird.

I'll debug in a few hours.

@gopherbot

This comment has been minimized.

Copy link

commented Oct 26, 2018

Change https://golang.org/cl/144858 mentions this issue: cmd/buildlet: restore tmp/gocache dirs after recursive delete of world

gopherbot pushed a commit to golang/build that referenced this issue Oct 26, 2018

cmd/buildlet: restore tmp/gocache dirs after recursive delete of world
Fixes regression from CL 144637 for s390x and arm5 (the only two deployed with that change).

Fixes golang/go#27182

Change-Id: I7a4a92c7c97f3f6c738d757a05b9dfc755547bf0
Reviewed-on: https://go-review.googlesource.com/c/144858
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.