Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: misc/reboot.TestRepeatBootstrap flaky during toolchain3 on linux-ppc64le-* #35233

Open
bcmills opened this issue Oct 29, 2019 · 4 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 29, 2019

The linux-ppc64le builders are frequently failing in TestRepeatBootstrap.

The failure mode seems to always be a compile error when building cmd/compile/internal/ssa during the Building Go toolchain3 step.

Sometimes the compile command fails with exit status 1 (https://build.golang.org/log/91e2d3f5fd454609faf1f004a756382ad5aa6e20).
Sometimes it instead fails with signal: killed (https://build.golang.org/log/00d4e0a84ece047eaa53251c7c4fc0955bba8eea).

Is it possible that this step is running out of RAM?

CC @randall77 @bradfitz @dmitshur @laboger

@laboger

This comment has been minimized.

Copy link
Contributor

@laboger laboger commented Oct 29, 2019

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 29, 2019

Change https://golang.org/cl/204038 mentions this issue: dashboard: skip reboot test on ppc64x builders

@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Oct 29, 2019

Can the test be modified to print out more information about the failure?
Maybe stderr has more information.

It's already redirecting stdout and stderr to its own stdout/stderr:

        cmd := exec.Command(filepath.Join(runtime.GOROOT(), "src", makeScript))
        cmd.Dir = gorootSrc
	cmd.Env = append(os.Environ(), "GOROOT=", "GOROOT_BOOTSTRAP="+runtime.GOROOT())
        cmd.Stderr = os.Stderr
        cmd.Stdout = os.Stdout
        if err := cmd.Run(); err != nil {
                t.Fatal(err)
        }

So right above the "Exit status 2" is how it failed, the lines that @bcmills quoted.

Not sure why reboot would take more memory, though. It already did a make.bash successfully if we made it this far, and dist itself doesn't take much memory.

Oh---- rundockerbuildlet gives it a tmpfs as its /workdir:

https://github.com/golang/build/blob/c53d4ba783d113c1e97ca07be915b7e2f6142ede/cmd/rundockerbuildlet/rundockerbuildlet.go#L157

And reboot runs at the end, when its /workdir is most full.

And because tmpfs, being in memory, contributes to its container memory limit, it has much less memory to work with in this second make.bash later.

gopherbot pushed a commit to golang/build that referenced this issue Oct 29, 2019
Updates golang/go#35233

Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc
Reviewed-on: https://go-review.googlesource.com/c/build/+/204038
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@laboger

This comment has been minimized.

Copy link
Contributor

@laboger laboger commented Oct 30, 2019

I thought there would be more information about the failure than just "exit status 2".

This test appears to be rebuilding the toolchain using the toolchain it just built as its bootstrap compiler. Doesn't that mean it will be changing/updating components of the toolchain as it is using those components for the new build?

go build cmd/compile/internal/ssa: /workdir/tmp/reboot-goroot202913779/pkg/tool/linux_ppc64le/compile: exit status 1
go tool dist: FAILED: /workdir/tmp/reboot-goroot202913779/pkg/tool/linux_ppc64le/go_bootstrap install -gcflags=all= -ldflags=all= -a -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1
--- FAIL: TestRepeatBootstrap (100.86s)
    reboot_test.go:50: exit status 2

Here it is using /workdir/tmp/reboot-goroot202913779/pkg/tool/linux_ppc64le/compile to compile with but the result is an install of cmd/compile.

@bradfitz bradfitz added the NeedsFix label Nov 4, 2019
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
Updates golang/go#35233

Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc
Reviewed-on: https://go-review.googlesource.com/c/build/+/204038
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.