all: the tree is highly flaky

[FiloSottile]

It's the week before the freeze, and the tree is horribly flaky, which makes it extremely painful to land anything, right at the time everyone is working to land things.

https://build.golang.org shows a smattering of failures, and yesterday over a single chain I had to restart six TryBots.

• https://go-review.googlesource.com/c/go/+/776700/2#message-3defefbed853a7999d425d252d3021815f81f203
• https://go-review.googlesource.com/c/go/+/776707/5#message-e5d7b17ea48dc26ee0adf19c948ce30aa5f423ac
• https://go-review.googlesource.com/c/go/+/776703/5#message-01d6ae5abc7f4f0b84a1bc2928c7a5b1826775d6
• https://go-review.googlesource.com/c/go/+/776705/5#message-8c14e74c573005c3dd01a293f0b564b9611fda31
• https://go-review.googlesource.com/c/go/+/777380/3#message-543374e7a41c16e93502a8b3e76aea3182518600
• https://go-review.googlesource.com/c/go/+/777384/3#message-430ecc0b5538fd2bc5b906a8106de7a4607c4c1e

I just uploaded a new set of patches, and I already got three flakes.

• https://go-review.googlesource.com/c/go/+/777320/6#message-0494b430ca7bf3ed7068dddb96139234cc57150a
• https://go-review.googlesource.com/c/go/+/776703/6#message-7b014c7e840c42ed58be09757b8714ea8c38ae07
• https://go-review.googlesource.com/c/go/+/777380/4#message-6198a426f570bbdd8e541a89ca314a9667dbb498

At this rate, it's just not worth addressing nits early in the chain, which is a terrible outcome.

Can we revert changes that introduced flakiness? Can we at least track them so this doesn't happen every six months?

/cc @golang/release

[dmitshur]

TestTransportBodyReadError and other net/http flakes are likely related to the known flakes tracked in issue #78737 (CC @neild, @nicholashusin). There have been recent changes that helped, like #79387 (comment), but there may be more left. I marked #78737 as a release blocker and I understand the intention there is to turn off the optional recently added tests and re-enable them after the tree reopens for Go 1.28.

Some of the other flakes need individual attention. Watchflakes has filed many issues recently, though I haven't yet had a chance to look over them.

[FiloSottile]

Hit a bunch more, and then it looks like the entire chain failed with

one or more test shards experienced an unknown infra failure

so I stopped tracking them.

• https://go-review.googlesource.com/c/go/+/777381/4#message-4abdf46f63aba966c4644aa45677d7f54ccaae36
• https://go-review.googlesource.com/c/go/+/776704/6#message-ff1355c6e99d6f78ffd04735c8331cea96776c81
• https://go-review.googlesource.com/c/go/+/777881/1#message-0696f6bc440afba2991d6ff40401f3bb349d6480
• https://go-review.googlesource.com/c/go/+/777384/4#message-3ec91c4a235b49f212845f0bced7f8471e358b44
• https://go-review.googlesource.com/c/go/+/776701/4#message-c0e742a8c27052c13b4a4f2b2de1c49a2a30b298
• https://go-review.googlesource.com/c/go/+/776706/6#message-1be9d042cb7e823748ba06d913daa5f29020914b
• https://go-review.googlesource.com/c/go/+/745661/8#message-980589d290a3272913df6b720058fcf7516ca1ed
• https://go-review.googlesource.com/c/go/+/776707/6#message-d6b19ab134f9ffaa99159b3ffa50642e9403ce89

Note also that TryBots have been running for over an hour, and it doesn't look like it's due to #76768. Is there an issue tracking TryBot duration? Flakiness and slowness combine, because it makes the context-switch disruption loop take a whole day.

[nicholashusin]

Sorry! Just sent out https://go.dev/cl/766740 to disable the HTTP/3 tests.
I think net/http tests are not the only flakes unfortunately, but syncing your CLs once https://go.dev/cl/766740 is submitted should still help things at least.

[nicholashusin]

Huh, I think the builders in general are having problems. TryBots in all of my recent CLs have been stuck, and eventually fails with the following:

This CL has failed the run. Reason:

Failed Tryjobs:

golang/try/x_net-go1.26-linux-amd64. Summary (view all results):

Task did not start, no resource

Some of them are x/net CLs too, where tests usually run pretty fast.

[dmitshur]

@nicholashusin Thanks, that seems like a separate builder infrastructure problem. I filed #79410 for it.

[FiloSottile]

Mailed a new chain rebased on the latest tree with the HTTP/3 skip and after the LUCI fix. 6 of 28 flaked.

• https://go-review.googlesource.com/c/go/+/776707/7#message-05a481536e89178c6ffb76e5466dbefb4658e188
• https://go-review.googlesource.com/c/go/+/776701/5#message-8910053733617e088dcb428a1fbb1a3833f6ae36
• https://go-review.googlesource.com/c/go/+/776706/7#message-7c94860e943b86269bce249564bbe306a81f138e
• https://go-review.googlesource.com/c/go/+/725040/8#message-ffb134999eb703ef880e61767f6501264a4cd9fc
• https://go-review.googlesource.com/c/go/+/777220/8#message-9e972d056a535a45113a0153490e3df8bfc99179
• https://go-review.googlesource.com/c/go/+/776705/7#message-a29a5b6920b9f64c0ef18b75ad99981264410d72

[randall77]

776707 : #76540, I'm thinking about it. #76540 (comment) Maybe the test just needs to be softened. Author is no longer on the Go team though, so I'm not sure.
776701: #78737 or related issues. races in some http tests. @nicholashusin is there more to do here?
776706: ditto
725040: #76901, probably
777220: #70602 @rhysh looked at it. #70602 (comment) Any news?
776705: looks like the trybot just ... stopped?

[nicholashusin]

All HTTP/3 tests have been disabled, so #78737 should be in the clear at this point.

776701 shows a race condition with HTTP/2. 776706 doesn't show a definitive log, but is presumably the same data race. Maybe related to the recent move of HTTP/2 from x/net to std? CC @neild

I can take a look at the HTTP/2 flake come Monday or so.

[thepudds]

FWIW, I took a peek at the goroutine leak test that Filippo hit (#76540), and sent a CL to try to deflake it (https://go.dev/cl/778620).

Maybe there is a better way to deflake and happy to hear any suggestions or feedback on the CL.

With that change, I did not see it fail locally in ~1000 tries. (However, I also wasn't able to get it to flake locally in ~1000 tries before my change either, but I did not try too hard. In contrast, it sounds like maybe Keith was able to repro in ~100 tries; maybe it is sensitive to exact CPU count or something else).

I'll also open a separate issue in a bit for tracking down a better root cause and/or possibly adjusting the leak profiler if needed. (I thought a separate issue might be a easier place for humans to collaborate compared to the two related watchflakes issues that have ~300 comments on them at this point.)

[gopherbot]

Change https://go.dev/cl/778620 mentions this issue: runtime/pprof: possibly deflake TestGoroutineLeakProfileConcurrency stress tests

[jakebailey]

I still regularly get flakes on my stacks, even though they are recent on master. e.g. https://ci.chromium.org/ui/p/golang/builders/try/gotip-linux-amd64-race/b8681519373928066737/overview

[gopherbot]

Change https://go.dev/cl/779120 mentions this issue: net/http: fix data race in TestServerNoWriteTimeout/h2

[gopherbot]

Change https://go.dev/cl/779160 mentions this issue: runtime: split gp.m.locks bits for lock vs acquirem