New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: upgrade FreeBSD builders to avoid flakes #25289

Open
ianlancetaylor opened this Issue May 8, 2018 · 15 comments

Comments

Projects
None yet
5 participants
@ianlancetaylor
Contributor

ianlancetaylor commented May 8, 2018

2018-05-01T14:50:33-7f6105f/freebsd-386-11_1
2018-05-01T18:22:06-a5b80d4/freebsd-amd64-11_1
2018-05-02T17:52:14-e8d417d/freebsd-amd64-11_1
2018-05-04T14:47:49-166c37a/freebsd-386-11_1
2018-05-05T00:53:45-2a7e19e/freebsd-amd64-11_1
2018-05-05T00:54:52-f7c767e/freebsd-amd64-11_1
2018-05-06T04:28:23-a8a60ac/freebsd-386-10_3
2018-05-08T13:29:22-b00f72e/freebsd-386-11_1

findflakes reports

First observed 7f6105f 01 May 14:50 2018 (92 commits ago)
Last observed  b00f72e 08 May 13:29 2018 (0 commits ago)
100% chance failure is still happening
7.6% failure probability (8 of 93 commits)
Likely culprits:
    8% 7f6105f net, syscall: make ECONNRESET/ECONNABORTED only temporary for Accept
    7% 1129ec2 database/sql: trivial documentation fixes
    6% 60c76f7 cmd/compile: optimize bvec routines
    6% d91e970 runtime: avoid unnecessary scanblock calls
    6% dda4591 runtime: add BenchmarkScanStack
    5% 9ecf899 cmd: remove some unnecessary gotos
    5% 030ac2c cmd/doc: skip directories like other go tools
    4% 8c67ca1 cmd/compile: apply TODO now that gcCompat is gone
    4% 16d1a8e runtime: move open/close/read/write from syscall to libc on Darwin
    4% ce5c387 runtime: move more syscalls to libc on Darwin
No known past failures
@ayang64

This comment has been minimized.

ayang64 commented Jun 24, 2018

I just ran this test on 12-CURRENT and 11.1-RELEASE-p4 and it passed.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jun 25, 2018

@ayang64 Thanks. To be clear, the test normally passes. As the message above suggests, it is only failing about 7.6% of the time. How many times did you run the test?

@ayang64

This comment has been minimized.

ayang64 commented Jun 27, 2018

I just tried 10000 times under 12-CURRNET (02bd6c4) and 12.0-CURRENT (58e53e526dc) and 500 under 11.1-RELEASE-p4 with no failures.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jun 27, 2018

Great, thanks!

Changing this issue to upgrade our builders.

@ianlancetaylor ianlancetaylor changed the title from net: flakes in TestNotTemporaryRead on FreeBSD to builders: upgrade FreeBSD builders to avoid flakes Jun 27, 2018

@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Unplanned Jun 27, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jun 27, 2018

@bradfitz bradfitz changed the title from builders: upgrade FreeBSD builders to avoid flakes to x/build: upgrade FreeBSD builders to avoid flakes Aug 3, 2018

@gopherbot

This comment has been minimized.

gopherbot commented Aug 3, 2018

Change https://golang.org/cl/127837 mentions this issue: net: skip flaky TestNotTemporaryRead on FreeBSD

gopherbot pushed a commit that referenced this issue Aug 3, 2018

net: skip flaky TestNotTemporaryRead on FreeBSD
Updates #25289

Change-Id: I662760b921be625aca988cd0b43c648ac5dfd814
Reviewed-on: https://go-review.googlesource.com/127837
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 14, 2018

Change https://golang.org/cl/135475 mentions this issue: env/freebsd-amd64: add FreeBSD 10.4, 11.2 images, update build script

@paulzhol paulzhol added the Builders label Sep 22, 2018

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 27, 2018

FreeBSD people, is it reasonable to take down our 10.3 and 11.1 builders and replace them with 10.4 or 11.2? I'd rather not double the number of FreeBSD builder's we're running, but we could if necessary. I don't know what versions are important/commonly used in practice.

/cc @dmitshur

gopherbot pushed a commit to golang/build that referenced this issue Sep 27, 2018

env/freebsd-amd64: add FreeBSD 10.4, 11.2 images, update build script
Use freebsd-update when preparing the downloaded -RELEASE image to get errata and security
updates made since the release.

Switch kern.timecounter.hardware from ACPI-safe to TSC on GCE
https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/000080.html

Drop vtnet hw.vtnet.csum_disable=1 workaround, should be fixed in all supported versions.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207446

Update the expect script to switch the console to serial, this makes it possible to match
prompts via a regexp (11.2 has 'root@freebsd:~ #', while the older images are 'root@:~ #').
Otherwise qemu with -display curses emits ansi escape codes which make a mess of everything.
This requires the sgabios smbios extension installed on the host preparing the images.
https://code.google.com/archive/p/sgabios/
https://packages.ubuntu.com/xenial/sgabios
SeaBIOS 1.11.0 and up should have a compatible rom builtin,
but it's not even available in Ubuntu 18.04.
qemu/seabios@d6728f3

Additionaly drop the -smp 2 directive for qemu, until MTTCG is enabled on amd64 targets.

Updates golang/go#25289

Change-Id: If33c2b35b4221f27d719938bcfc013efe3549305
Reviewed-on: https://go-review.googlesource.com/135475
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Sep 27, 2018

Change https://golang.org/cl/138258 mentions this issue: dashboard: add FreeBSD 11.2 and 10.4 builders

@paulzhol

This comment has been minimized.

Member

paulzhol commented Sep 28, 2018

@bradfitz our builders are horribly out of date, for eg. our 11.1 builder is 11.1-RELEASE and is missing around a year worth of security patches and updates (~15 of them).
10.3-RELEASE was released in April 4, 2016 and was marked EoL since April 30, 2018 (that's over two years of missed updates).

Unfortunately 11.1 is expected to receive EoL on the 30th of September, and 10.4 a month after that.
https://www.freebsd.org/security/security.html#sup

I think we have no choice but to switch the 10.3-RELEASE to 10.4-RELEASE-p13. Maybe it's worth while having a 11.1 builder as well, but I'd rather we update it to 11.1-RELEASE-p15 if we'd need to test future Go 1.10/1.11 releases?

@ayang64

This comment has been minimized.

ayang64 commented Sep 28, 2018

Since 10.4 and 11.2 are currently supported releases (even though they're close to EoL), I think it is reasonable to bump up the revs used by the builders.

It may also be helpful to start eyeing a 12.0 builder since it is due in Novemberish.

It looks like there are a few system call/ABI changes in 12 that might need addressing. I wonder how incompatibilities across GOOS versions are handled. It might be useful to be able to target FreeBSD 11.x and FreeBSD 12.x as separate GOOS.

gopherbot pushed a commit to golang/build that referenced this issue Oct 2, 2018

dashboard: add FreeBSD 11.2 and 10.4 builders
Not yet the defaults or trybots until we see they're working regularly
for a few days and compare timings, etc.

Updates golang/go#25289

Change-Id: I4baed066b5f6d9f01e5454b716cacda4986620d0
Reviewed-on: https://go-review.googlesource.com/138258
Reviewed-by: Yuval Pavel Zholkover <paulzhol@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@bradfitz

This comment has been minimized.

Member

bradfitz commented Oct 2, 2018

It might be useful to be able to target FreeBSD 11.x and FreeBSD 12.x as separate GOOS.

That introduces a bunch of complexity which we don't want for ourselves or our users. Making it Just Work is much more preferred.

@paulzhol

This comment has been minimized.

Member

paulzhol commented Nov 20, 2018

@bradfitz can we drop the 10.3 (and maybe 11.1) builders so that we can remove the workaround for regular files being added to kqueue (#19093)?

@bradfitz

This comment has been minimized.

Member

bradfitz commented Nov 20, 2018

@paulzhol, not until Go 1.13 (#27619).

@paulzhol

This comment has been minimized.

Member

paulzhol commented Nov 20, 2018

@bradfitz I'm not sure if the kernel fix was added to 10.4 (I'll try to reproduce in a VM). If it's fixed, can we drop the 10.3 builder (and maybe the 11.1 one) for Go 1.12?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment