x/build/maintner: test failures with "416 Requested Range Not Satisfiable" from maintner.golang.org #51211

bcmills · 2022-02-15T16:26:49Z

2022/02/14 19:12:27 Downloading 16776793 bytes of https://storage.googleapis.com/golang-maintner-log/0126.ddc03bfdacbc6aa8a32f5987fd7fcad8a1ec1ee5da521071200b2ae6.mutlog ...
2022/02/14 19:12:27 wrote C:\Users\gopher\.cache\golang-maintner\0126.ddc03bfdacbc6aa8a32f5987fd7fcad8a1ec1ee5da521071200b2ae6.mutlog
2022/02/14 19:12:27 Downloading 16776890 bytes of https://storage.googleapis.com/golang-maintner-log/0127.cf0c3c7009d2d5988293283b748239bcd8b18acfb086db5c390df4eb.mutlog ...
2022/02/14 19:12:27 wrote C:\Users\gopher\.cache\golang-maintner\0127.cf0c3c7009d2d5988293283b748239bcd8b18acfb086db5c390df4eb.mutlog
2022/02/14 19:12:27 Downloading -5369 bytes of https://maintner.golang.org/logs/128 ...
2022/02/14 19:12:27 Corpus GetMutations: syncing segment 128: https://maintner.golang.org/logs/128: 416 Requested Range Not Satisfiable
--- FAIL: TestCorpusCheck (33.65s)
    godata_test.go:52: getting corpus: syncing segment 128: https://maintner.golang.org/logs/128: 416 Requested Range Not Satisfiable
2022/02/14 19:12:27 Loading data from log *maintner.netMutSource ...
2022/02/14 19:13:01 Reloaded data from log *maintner.netMutSource.
FAIL
FAIL	golang.org/x/build/maintner/godata	68.324s

This failure mode seems to occur fairly frequently on windows/amd64, and somewhat less frequently on linux/amd64:

greplogs --dashboard -md -l -e 'syncing segment \d+: .*: 416 Requested Range Not Satisfiable'

(CC @golang/release)

The text was updated successfully, but these errors were encountered:

bcmills · 2022-03-30T18:54:38Z

Curiously, a similar failure mode has also occurred with 502 Bad Gateway:

greplogs --dashboard -md -l -e 'getting corpus: .*: 502 Bad Gateway'

2022-03-17T19:24:19-8c691c3-c6244b5/windows-amd64-longtest

dmitshur · 2022-04-05T18:56:32Z

We can mark the test as known flaky sooner.

bcmills · 2022-05-09T17:17:49Z

greplogs -l -e 'syncing segment \d+: .*: 416 Requested Range Not Satisfiable' --since=2022-02-16
2022-05-06T16:18:34-b8613e0-5fcd1ba/windows-amd64-longtest
2022-03-17T18:03:05-e0ed2a2-0fca8a8/linux-amd64-longtest

dmitshur · 2022-06-27T13:55:33Z

The 502 Bad Gateway from /logs is another (non-serious) issue, filed #53569 for it.

I was going to send a small change to add a log line, to learn what range value causes the server to respond with 416, but we already have this log line and can see from the log that this is caused by issue #26922. Sent CL 414434 for it.

gopherbot · 2022-06-27T13:55:53Z

Change https://go.dev/cl/414434 mentions this issue: maintner: don't try to download negative bytes

If the maintnerd server restarts with some uncommitted transactions, clients may end up having more downloaded bytes than the server, after the server restarts. We're invariably heading towards an ErrSplit if this happens (which is fine, otherwise we'd need to add more code to avoid duplicate event processing and it's unlikely to be worthwhile). In such a situation, don't try to download new data from the server, since that's guaranteed to fail with a 416 Range Not Satisfiable error. Do try to reuse the prefix of the existing data as long as its checksum matches, otherwise download the new version of the growing file. Fixes golang/go#26922. Updates golang/go#51211. Change-Id: Ide099ee0740e854cfe764db8c3b4341836a237f4 Reviewed-on: https://go-review.googlesource.com/c/build/+/414434 TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Alex Rakoczy <jenny@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>

heschi · 2022-07-06T20:15:57Z

2022-06-28T19:45:04-950e098-a30f434/linux-amd64-longtest

dmitshur · 2022-07-06T20:27:12Z

The aforementioned failure is from golang/build@950e098, which is a few commits before CL 414434 (golang/build@4443b10).

Tentatively closing since I expect this should be completely fixed and there's nothing more to do here, but please reopen if there are further occurrences after the aforementioned commit.

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Feb 15, 2022

gopherbot added this to the Unreleased milestone Feb 15, 2022

dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 15, 2022

dmitshur added this to Planned in Go Release Team Apr 5, 2022

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 5, 2022

dmitshur closed this as completed Jul 6, 2022

Go Release Team automation moved this from Planned to Done Jul 6, 2022

x/build/maintner: test failures with "416 Requested Range Not Satisfiable" from maintner.golang.org #51211

x/build/maintner: test failures with "416 Requested Range Not Satisfiable" from maintner.golang.org #51211

bcmills commented Feb 15, 2022

bcmills commented Mar 30, 2022

dmitshur commented Apr 5, 2022

bcmills commented May 9, 2022

dmitshur commented Jun 27, 2022

gopherbot commented Jun 27, 2022

heschi commented Jul 6, 2022

dmitshur commented Jul 6, 2022

x/build/maintner: test failures with "416 Requested Range Not Satisfiable" from maintner.golang.org #51211

x/build/maintner: test failures with "416 Requested Range Not Satisfiable" from maintner.golang.org #51211

Comments

bcmills commented Feb 15, 2022

bcmills commented Mar 30, 2022

dmitshur commented Apr 5, 2022

bcmills commented May 9, 2022

dmitshur commented Jun 27, 2022

gopherbot commented Jun 27, 2022

heschi commented Jul 6, 2022

dmitshur commented Jul 6, 2022