Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: upgrade FreeBSD builders to avoid flakes #25289

Closed
ianlancetaylor opened this Issue May 8, 2018 · 17 comments

Comments

Projects
None yet
5 participants
@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented May 8, 2018

2018-05-01T14:50:33-7f6105f/freebsd-386-11_1
2018-05-01T18:22:06-a5b80d4/freebsd-amd64-11_1
2018-05-02T17:52:14-e8d417d/freebsd-amd64-11_1
2018-05-04T14:47:49-166c37a/freebsd-386-11_1
2018-05-05T00:53:45-2a7e19e/freebsd-amd64-11_1
2018-05-05T00:54:52-f7c767e/freebsd-amd64-11_1
2018-05-06T04:28:23-a8a60ac/freebsd-386-10_3
2018-05-08T13:29:22-b00f72e/freebsd-386-11_1

findflakes reports

First observed 7f6105f 01 May 14:50 2018 (92 commits ago)
Last observed  b00f72e 08 May 13:29 2018 (0 commits ago)
100% chance failure is still happening
7.6% failure probability (8 of 93 commits)
Likely culprits:
    8% 7f6105f net, syscall: make ECONNRESET/ECONNABORTED only temporary for Accept
    7% 1129ec2 database/sql: trivial documentation fixes
    6% 60c76f7 cmd/compile: optimize bvec routines
    6% d91e970 runtime: avoid unnecessary scanblock calls
    6% dda4591 runtime: add BenchmarkScanStack
    5% 9ecf899 cmd: remove some unnecessary gotos
    5% 030ac2c cmd/doc: skip directories like other go tools
    4% 8c67ca1 cmd/compile: apply TODO now that gcCompat is gone
    4% 16d1a8e runtime: move open/close/read/write from syscall to libc on Darwin
    4% ce5c387 runtime: move more syscalls to libc on Darwin
No known past failures
@ayang64

This comment has been minimized.

Copy link

ayang64 commented Jun 24, 2018

I just ran this test on 12-CURRENT and 11.1-RELEASE-p4 and it passed.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

ianlancetaylor commented Jun 25, 2018

@ayang64 Thanks. To be clear, the test normally passes. As the message above suggests, it is only failing about 7.6% of the time. How many times did you run the test?

@ayang64

This comment has been minimized.

Copy link

ayang64 commented Jun 27, 2018

I just tried 10000 times under 12-CURRNET (02bd6c4) and 12.0-CURRENT (58e53e526dc) and 500 under 11.1-RELEASE-p4 with no failures.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

ianlancetaylor commented Jun 27, 2018

Great, thanks!

Changing this issue to upgrade our builders.

@ianlancetaylor ianlancetaylor changed the title net: flakes in TestNotTemporaryRead on FreeBSD builders: upgrade FreeBSD builders to avoid flakes Jun 27, 2018

@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Unplanned Jun 27, 2018

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor Author

ianlancetaylor commented Jun 27, 2018

@bradfitz bradfitz changed the title builders: upgrade FreeBSD builders to avoid flakes x/build: upgrade FreeBSD builders to avoid flakes Aug 3, 2018

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Aug 3, 2018

Change https://golang.org/cl/127837 mentions this issue: net: skip flaky TestNotTemporaryRead on FreeBSD

gopherbot pushed a commit that referenced this issue Aug 3, 2018

net: skip flaky TestNotTemporaryRead on FreeBSD
Updates #25289

Change-Id: I662760b921be625aca988cd0b43c648ac5dfd814
Reviewed-on: https://go-review.googlesource.com/127837
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Sep 14, 2018

Change https://golang.org/cl/135475 mentions this issue: env/freebsd-amd64: add FreeBSD 10.4, 11.2 images, update build script

@paulzhol paulzhol added the Builders label Sep 22, 2018

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Sep 27, 2018

FreeBSD people, is it reasonable to take down our 10.3 and 11.1 builders and replace them with 10.4 or 11.2? I'd rather not double the number of FreeBSD builder's we're running, but we could if necessary. I don't know what versions are important/commonly used in practice.

/cc @dmitshur

gopherbot pushed a commit to golang/build that referenced this issue Sep 27, 2018

env/freebsd-amd64: add FreeBSD 10.4, 11.2 images, update build script
Use freebsd-update when preparing the downloaded -RELEASE image to get errata and security
updates made since the release.

Switch kern.timecounter.hardware from ACPI-safe to TSC on GCE
https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/000080.html

Drop vtnet hw.vtnet.csum_disable=1 workaround, should be fixed in all supported versions.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207446

Update the expect script to switch the console to serial, this makes it possible to match
prompts via a regexp (11.2 has 'root@freebsd:~ #', while the older images are 'root@:~ #').
Otherwise qemu with -display curses emits ansi escape codes which make a mess of everything.
This requires the sgabios smbios extension installed on the host preparing the images.
https://code.google.com/archive/p/sgabios/
https://packages.ubuntu.com/xenial/sgabios
SeaBIOS 1.11.0 and up should have a compatible rom builtin,
but it's not even available in Ubuntu 18.04.
qemu/seabios@d6728f3

Additionaly drop the -smp 2 directive for qemu, until MTTCG is enabled on amd64 targets.

Updates golang/go#25289

Change-Id: If33c2b35b4221f27d719938bcfc013efe3549305
Reviewed-on: https://go-review.googlesource.com/135475
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Sep 27, 2018

Change https://golang.org/cl/138258 mentions this issue: dashboard: add FreeBSD 11.2 and 10.4 builders

@paulzhol

This comment has been minimized.

Copy link
Member

paulzhol commented Sep 28, 2018

@bradfitz our builders are horribly out of date, for eg. our 11.1 builder is 11.1-RELEASE and is missing around a year worth of security patches and updates (~15 of them).
10.3-RELEASE was released in April 4, 2016 and was marked EoL since April 30, 2018 (that's over two years of missed updates).

Unfortunately 11.1 is expected to receive EoL on the 30th of September, and 10.4 a month after that.
https://www.freebsd.org/security/security.html#sup

I think we have no choice but to switch the 10.3-RELEASE to 10.4-RELEASE-p13. Maybe it's worth while having a 11.1 builder as well, but I'd rather we update it to 11.1-RELEASE-p15 if we'd need to test future Go 1.10/1.11 releases?

@ayang64

This comment has been minimized.

Copy link

ayang64 commented Sep 28, 2018

Since 10.4 and 11.2 are currently supported releases (even though they're close to EoL), I think it is reasonable to bump up the revs used by the builders.

It may also be helpful to start eyeing a 12.0 builder since it is due in Novemberish.

It looks like there are a few system call/ABI changes in 12 that might need addressing. I wonder how incompatibilities across GOOS versions are handled. It might be useful to be able to target FreeBSD 11.x and FreeBSD 12.x as separate GOOS.

gopherbot pushed a commit to golang/build that referenced this issue Oct 2, 2018

dashboard: add FreeBSD 11.2 and 10.4 builders
Not yet the defaults or trybots until we see they're working regularly
for a few days and compare timings, etc.

Updates golang/go#25289

Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0
Reviewed-on: https://go-review.googlesource.com/138258
Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Oct 2, 2018

It might be useful to be able to target FreeBSD 11.x and FreeBSD 12.x as separate GOOS.

That introduces a bunch of complexity which we don't want for ourselves or our users. Making it Just Work is much more preferred.

@paulzhol

This comment has been minimized.

Copy link
Member

paulzhol commented Nov 20, 2018

@bradfitz can we drop the 10.3 (and maybe 11.1) builders so that we can remove the workaround for regular files being added to kqueue (#19093)?

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Nov 20, 2018

@paulzhol, not until Go 1.13 (#27619).

@paulzhol

This comment has been minimized.

Copy link
Member

paulzhol commented Nov 20, 2018

@bradfitz I'm not sure if the kernel fix was added to 10.4 (I'll try to reproduce in a VM). If it's fixed, can we drop the 10.3 builder (and maybe the 11.1 one) for Go 1.12?

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Dec 29, 2018

Change https://golang.org/cl/155977 mentions this issue: env/freebsd-amd64: update FreeBSD 12.0 to a -RELEASE image

gopherbot pushed a commit to golang/build that referenced this issue Jan 8, 2019

env/freebsd-amd64: update FreeBSD 12.0 to a -RELEASE image
Install GDB from the ports collection, the in-tree was removed from the base system
in FreeBSD 12.0, and the older releases have an unsupported (6.1.1 or older version).

Updates golang/go#25289

Change-Id: I8899a2a70c0b9ee9171621206431ca24a7e97704
Reviewed-on: https://go-review.googlesource.com/c/155977
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Jan 8, 2019

I rebuilt the FreeBSD amd64 image with https://golang.org/cl/155977 & updated the coordinator from git rev 7b78c2042368d5c56ee9dbd92ab5fa988c763944 to 9730d8b657054ead7ffc67bfb2dff24b89a368a5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.