Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: failures with "symbol type.[…] listed multiple times" on freebsd-amd64-12_2 since 2021-11-24 #49967

Closed
bcmills opened this issue Dec 3, 2021 · 16 comments
Labels
NeedsInvestigation OS-FreeBSD release-blocker
Milestone

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Dec 3, 2021

greplogs --dashboard -md -l -e 'symbol type.* listed multiple times'

2021-12-02T15:48:58-28ec0fd/freebsd-amd64-12_2
2021-12-01T18:47:44-3c63f30-08ecdf7/freebsd-amd64-12_2
2021-11-24T05:36:54-7cf1f38-47db3bb/freebsd-amd64-12_2

@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 3, 2021

Marking as release-blocker for Go 1.18 because this failure mode appears to be new as of November.
It occurs during compilation, so it is not trivial to resolve by adding a test skip.

(CC @jeremyfaller for routing)

@bcmills bcmills added NeedsInvestigation okay-after-beta1 OS-FreeBSD release-blocker labels Dec 3, 2021
@bcmills bcmills added this to the Go1.18 milestone Dec 3, 2021
@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 3, 2021

So far these failures have only been seen on the freebsd-amd64-12_2 builder, but with a sample size of N=3 it isn't obvious to me whether the underlying problem may affect other platforms.

@cherrymui
Copy link
Member

@cherrymui cherrymui commented Dec 3, 2021

The error is probably from the linker. May be the same cause s #49752.

@prattmic
Copy link
Member

@prattmic prattmic commented Dec 9, 2021

I haven't been trying to reproduce this (I'm going after #46272 (comment)), but this has reproduced in 2/5 of my crashes so far with:

$ goswarm -i 25 freebsd-amd64-12_2 go/src/all.bash

@cherrymui
Copy link
Member

@cherrymui cherrymui commented Dec 9, 2021

@prattmic do you think this could be a result from the memory corruption from the other bug, or this is a different issue? Thanks.

@prattmic
Copy link
Member

@prattmic prattmic commented Dec 9, 2021

It seems plausible, but I haven't seen enough details to have confidence one way or another.

@cherrymui cherrymui removed the okay-after-beta1 label Dec 14, 2021
@cherrymui
Copy link
Member

@cherrymui cherrymui commented Dec 15, 2021

Same as #49752 . We can revisit once we have FreeBSD builders with kernel fixed.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 5, 2022

https://storage.googleapis.com/go-build-log/301db3f5/freebsd-amd64-12_2_faaeee15.log (a TryBot).

I think @heschi pushed an updated FreeBSD 13 builder today — is there a fix available for 12.2 as well?

@prattmic
Copy link
Member

@prattmic prattmic commented Jan 5, 2022

#46272 (comment) noted that the fix is cherry-picked into FreeBSD 12 as well.

I believe https://lists.freebsd.org/archives/freebsd-snapshots/2021-December/000037.html is the latest 12 snapshot.

@heschi
Copy link
Contributor

@heschi heschi commented Jan 5, 2022

We're currently on 12.2, which doesn't have the fix. Would you like me to upgrade us to the latest 12.3 snapshot?

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 5, 2022

Since 12.2 has TryBot failures that we can't easily work around by adding test skips, I think we would add an updated 12.3 builder as soon as is feasible and turn off the 12.2 TryBots.

(Probably we should keep the 12.2 post-submit builders up until the 12.3 builders have run for enough CLs to be confident that there are no significant regressions from 12.2 to 12.3.)

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 5, 2022

Change https://golang.org/cl/375695 mentions this issue: dashboard,env/freebsd: add 12.3-STABLE

gopherbot pushed a commit to golang/build that referenced this issue Jan 5, 2022
With the XSAVE bug fix.

For golang/go#49967, golang/go#46272.

Change-Id: Ie0711ba83e4f15d26e6fe89e89c80b7c3ad2c23c
Reviewed-on: https://go-review.googlesource.com/c/build/+/375695
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jan 11, 2022

Change https://golang.org/cl/377474 mentions this issue: dashboard: clean up builders affected by memory corruption

@thanm
Copy link
Contributor

@thanm thanm commented Jan 11, 2022

I am pretty sure this falls into the same category as #49752, which is now marked fixed-- I am going to try one more goswarm test on this one to check that the problem is no longer reproducible.

@thanm
Copy link
Contributor

@thanm thanm commented Jan 12, 2022

OK, I ran some tests with goswarm overnight (10 motes running all.bash for roughly 20 hours) using the freebsd-amd64-12_3 builder, which incorporates the fix for the problem identified in #46272. No new failures with the "unreachable sym" mode. I am going to close out this bug.

For posterity, here are the failures that did turn up:

15 times:

--- FAIL: TestBlockProfile/debug=1 (0.02s)
    pprof_test.go:929: Bad chan send entry, expect:

7 times:

--- FAIL: TestStackGrowth (32.10s)
 .. stack_test.go:154: finalizer did not run

4 times:

 ##### Testing race detector
 --- FAIL: TestOutput (10.16s)
   output_test.go:83: failed test case test_fails_on_race, expect:

5 times:

# go run run.go -G=3 locklinear.go
exit status 2
panic: lockmany: too slow: 2000 ops: 16.141185ms; 4000 ops: 516.734873ms

then maybe a half dozen other singleton failures.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Jan 12, 2022

    pprof_test.go:929: Bad chan send entry, expect:

That's #6999, open since 2013 (and not FreeBSD-specific). 😩

    stack_test.go:154: finalizer did not run

That looks like #19381, but curiously it hasn't failed in the dashboard in the past few years. Maybe it's sensitive to machine load?

    output_test.go:83: failed test case test_fails_on_race, expect:

I don't see any issues filed at all for that one. 🤔

    panic: lockmany: too slow: 2000 ops: 16.141185ms; 4000 ops: 516.734873ms

#32986, open since 2019.

gopherbot pushed a commit to golang/build that referenced this issue Jan 12, 2022
Remove freebsd 12.2, which is replaced by 12.3 with the XSAVE fix.
Move freebsd 11.* to N2 machines, which are not affected.
Remove openbsd and netbsd e2/n1/n2/n2d-specific configurations now that
we have mostly understood the nature of that problem. Keep one around so
that the runtime team can create gomotes. Move the "official" builder to
the n2 cpu that works.

For golang/go#49967, golang/go#49209, golang/go#40561.
Fixes golang/go#50496.

Change-Id: If6989317f06cbec95d5addb19d9e968aecfa3f8a
Reviewed-on: https://go-review.googlesource.com/c/build/+/377474
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation OS-FreeBSD release-blocker
Projects
None yet
Development

No branches or pull requests

6 participants