Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: tests frequently timing out on dragonfly-amd64 since CL 412394 #53577

Closed
bcmills opened this issue Jun 27, 2022 · 6 comments
Closed

cmd/go: tests frequently timing out on dragonfly-amd64 since CL 412394 #53577

bcmills opened this issue Jun 27, 2022 · 6 comments
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Jun 27, 2022

go test proxy running at GOPROXY=http://127.0.0.1:55340/mod
--- FAIL: TestScript (0.02s)
…

FAIL
FAIL	cmd/go	1035.282s

None of the individual scripts was left running for longer than 30s or so, so it appears that the cmd/go tests in general have gotten much slower on this builder.

We had thought that CL 413634 and/or CL 413815 would fix the problem, but that appears not to have borne out.

greplogs -l -e \(\?ms\)\\Adragonfly-amd64.\*FAIL\\s+cmd/go\\s+\\d\{4}\\.\\d\*s --since=2022-04-01 --details

2022-06-26T22:12:19-351e0f4/dragonfly-amd64
2022-06-26T00:21:33-666d736/dragonfly-amd64
2022-06-25T19:08:51-3b594b9/dragonfly-amd64
2022-06-25T19:07:01-4f45ec5/dragonfly-amd64
2022-06-24T23:50:20-41e1d90/dragonfly-amd64
2022-06-24T22:08:17-bd47539/dragonfly-amd64
2022-06-24T21:22:43-6b6c64b/dragonfly-amd64
2022-06-24T18:54:27-b9c4d94/dragonfly-amd64
2022-06-24T17:48:24-73475ef/dragonfly-amd64
2022-06-24T17:45:41-3e58ef6/dragonfly-amd64
2022-06-24T00:37:46-d38f1d1/dragonfly-amd64
2022-06-23T19:05:29-de5329f/dragonfly-amd64
2022-06-22T23:27:17-2e773a3/dragonfly-amd64
2022-06-22T22:14:02-ff17b7d/dragonfly-amd64
2022-06-22T20:56:07-bdab4cf/dragonfly-amd64
2022-06-22T19:56:09-aca37d1/dragonfly-amd64
2022-06-21T15:00:37-19ed442/dragonfly-amd64
2022-06-20T18:39:08-3fcbfb0/dragonfly-amd64
2022-06-17T22:44:22-ec58e3f/dragonfly-amd64
2022-06-17T21:38:40-103cc66/dragonfly-amd64
2022-06-17T21:35:36-d42a488/dragonfly-amd64
2022-06-17T21:21:58-9e2f289/dragonfly-amd64
2022-06-17T21:21:26-dd2d00f/dragonfly-amd64
2022-06-17T15:02:55-6c25ba6/dragonfly-amd64
2022-06-16T21:58:40-9068c68/dragonfly-amd64
2022-06-16T21:25:35-7bad615/dragonfly-amd64
2022-06-16T20:59:51-f38a580/dragonfly-amd64
2022-06-16T20:01:10-635b124/dragonfly-amd64
2022-06-16T18:29:19-ef808ae/dragonfly-amd64
2022-06-16T16:44:35-74f1fa6/dragonfly-amd64
2022-06-16T16:22:37-bcce8ef/dragonfly-amd64
2022-06-16T01:03:59-ecc268a/dragonfly-amd64
2022-06-15T21:24:24-91baf5c/dragonfly-amd64
2022-06-15T21:18:07-0e3d0c9/dragonfly-amd64
2022-06-15T21:18:06-74bf90c/dragonfly-amd64
2022-06-15T21:16:14-0cd0c12/dragonfly-amd64
2022-06-15T20:07:10-937fa50/dragonfly-amd64
2022-06-15T17:37:53-c2c76c6/dragonfly-amd64
2022-06-15T17:13:55-36147dd/dragonfly-amd64

(39 matching logs)

(CC @golang/dragonfly)

@rsc is planning to investigate.

@bcmills bcmills added OS-Dragonfly NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. GoCommand cmd/go labels Jun 27, 2022
@bcmills bcmills added this to the Go1.19 milestone Jun 27, 2022
@gopherbot
Copy link

Change https://go.dev/cl/414614 mentions this issue: dashboard: add known issue for dragonfly-amd64

gopherbot pushed a commit to golang/build that referenced this issue Jun 27, 2022
The cmd/go tests have been timing out frequently on this builder.
We are still investigating the root cause.

For golang/go#53577.

Change-Id: I841117e8c2cff7296869d7beedced364714b7cf9
Reviewed-on: https://go-review.googlesource.com/c/build/+/414614
Reviewed-by: David Chase <drchase@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/416134 mentions this issue: cmd/go: use package index for std in load.loadPackageData

@gopherbot
Copy link

Change https://go.dev/cl/416178 mentions this issue: cmd/go: make module index loading O(1)

@gopherbot
Copy link

Change https://go.dev/cl/416179 mentions this issue: cmd/go: avoid spurious readdir during fsys.Walk

gopherbot pushed a commit that referenced this issue Jul 11, 2022
load.loadPackageData was only using an index for modules,
not for standard library packages. Other parts of the code were
using the index, so there was some benefit, but not as much
as you'd hope.

With the index disabled, the Script/work test takes 2.2s on my Mac.
With the index enabled before this CL, it took 2.0s.
With the index enabled after this CL, it takes 1.6s.

Before this CL, the Script/work test issued:

	 429 IsDir
	  19 IsDirWithGoFiles
	   7 Lstat
	9072 Open
	 993 ReadDir
	 256 Stat
	   7 Walk
	   3 indexModule
	  24 openIndexModule
	 525 openIndexPackage

After this CL, it issued:

	  19 IsDirWithGoFiles
	   7 Lstat
	  60 Open
	 606 ReadDir
	 256 Stat
	   7 Walk
	   3 indexModule
	  24 openIndexModule
	 525 openIndexPackage

This speedup helps the Dragonfly builder, which has very slow
file I/O and is timing out since a recent indexing change.

Times for go test -run=Script/^work$ on the Dragonfly builder:

	50s before indexing changes
	31s full module indexing of std
	46s per-package indexing of std

It cuts the time for go test -run=Script/^work$ from 44s to 20s.

For #53577.

Change-Id: I7189a77fc7fdf61de3ab3447efc4e84d1fc52c25
Reviewed-on: https://go-review.googlesource.com/c/go/+/416134
Reviewed-by: Bryan Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Jul 11, 2022
For a large module, opening the index was populating tables with
entries for every package in the module. If we are only using a small
number of those packages, this is wasted work that can dwarf the
benefit from the index.

This CL changes the index reader to avoid loading all packages
at module index open time. It also refactors the code somewhat
for clarity.

It also removes some duplication by defining that a per-package
index is a per-module index containing a single package, rather
than having two different formats and two different decoders.

It also changes the string table to use uvarint-prefixed data
instead of having to scan for a NUL byte. This makes random access
to long strings more efficient - O(1) instead of O(n) - and can significantly
speed up the strings.Compare operation in the binary search looking
for a given package.

Also add a direct test of the indexing code.

For #53577.

Change-Id: I7428d28133e4e7fe2d2993fa014896cd15af48af
Reviewed-on: https://go-review.googlesource.com/c/go/+/416178
Reviewed-by: Bryan Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Jul 12, 2022
fsys.Walk is cloned from filepath.Walk, which has always handled
a walk of a directory by reading the full directory before calling the
callback on the directory itself. So if the callback returns fs.SkipDir,
those entries are thrown away, but the expense of reading them was
still incurred. (Worse, this is the expensive directory read that also
calls Stat on every entry.) On machines with slow file system I/O,
these reads are particularly annoying. For example, if I do

	go list m...

there is a call to filepath.Walk that is told about $GOROOT/src/archive
and responds by returning filepath.SkipDir because archive does not
start with m, but it only gets the chance to do that after the archive
directory has been read. (Same for all the other top-level directories.)
Even something like go list github.com/foo/bar/... reads every top-level
$GOPATH/src directory.

When we designed filepath.WalkDir, one of the changes we made was
to allow calling the callback twice for a directory: once before reading it,
and then possibly again if the read produces an error (uncommon).
This CL changes fsys.Walk to use that same model. None of the callbacks
need changing, but now the $GOROOT/src/archive and other top-level
directories won't be read when evaluating a pattern like 'm...'.

For #53577.
Fixes #53765.

Change-Id: Idfa3b9e2cc335417bfd9d66dd584cb16f92bd12e
Reviewed-on: https://go-review.googlesource.com/c/go/+/416179
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/417094 mentions this issue: cmd/go: avoid indexing GOROOT packages when the compiler is 'gccgo'

gopherbot pushed a commit that referenced this issue Jul 12, 2022
The gccgo compiler does not load standard-library packages from
GOROOT/src, so we cannot load those packages from the GOROOT/src
index when using that compiler.

This fixes TestScript/gccgo_link_c (and perhaps other gccgo tests)
when a 'gccgo' executable is present. Unfortunately, only a few
builders caught the broken test because 'gccgo' is not installed
on most Go project builders (see #35786).

For #53577.
Fixes #53815.

Change-Id: I11a5cf6dbf4ac9893c4d02bd6ab7ef60f67b1e87
Reviewed-on: https://go-review.googlesource.com/c/go/+/417094
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/419084 mentions this issue: dashboard: use dragonfly on GCE for dragonfly-amd64 builds

jproberts pushed a commit to jproberts/go that referenced this issue Aug 10, 2022
load.loadPackageData was only using an index for modules,
not for standard library packages. Other parts of the code were
using the index, so there was some benefit, but not as much
as you'd hope.

With the index disabled, the Script/work test takes 2.2s on my Mac.
With the index enabled before this CL, it took 2.0s.
With the index enabled after this CL, it takes 1.6s.

Before this CL, the Script/work test issued:

	 429 IsDir
	  19 IsDirWithGoFiles
	   7 Lstat
	9072 Open
	 993 ReadDir
	 256 Stat
	   7 Walk
	   3 indexModule
	  24 openIndexModule
	 525 openIndexPackage

After this CL, it issued:

	  19 IsDirWithGoFiles
	   7 Lstat
	  60 Open
	 606 ReadDir
	 256 Stat
	   7 Walk
	   3 indexModule
	  24 openIndexModule
	 525 openIndexPackage

This speedup helps the Dragonfly builder, which has very slow
file I/O and is timing out since a recent indexing change.

Times for go test -run=Script/^work$ on the Dragonfly builder:

	50s before indexing changes
	31s full module indexing of std
	46s per-package indexing of std

It cuts the time for go test -run=Script/^work$ from 44s to 20s.

For golang#53577.

Change-Id: I7189a77fc7fdf61de3ab3447efc4e84d1fc52c25
Reviewed-on: https://go-review.googlesource.com/c/go/+/416134
Reviewed-by: Bryan Mills <bcmills@google.com>
jproberts pushed a commit to jproberts/go that referenced this issue Aug 10, 2022
For a large module, opening the index was populating tables with
entries for every package in the module. If we are only using a small
number of those packages, this is wasted work that can dwarf the
benefit from the index.

This CL changes the index reader to avoid loading all packages
at module index open time. It also refactors the code somewhat
for clarity.

It also removes some duplication by defining that a per-package
index is a per-module index containing a single package, rather
than having two different formats and two different decoders.

It also changes the string table to use uvarint-prefixed data
instead of having to scan for a NUL byte. This makes random access
to long strings more efficient - O(1) instead of O(n) - and can significantly
speed up the strings.Compare operation in the binary search looking
for a given package.

Also add a direct test of the indexing code.

For golang#53577.

Change-Id: I7428d28133e4e7fe2d2993fa014896cd15af48af
Reviewed-on: https://go-review.googlesource.com/c/go/+/416178
Reviewed-by: Bryan Mills <bcmills@google.com>
jproberts pushed a commit to jproberts/go that referenced this issue Aug 10, 2022
fsys.Walk is cloned from filepath.Walk, which has always handled
a walk of a directory by reading the full directory before calling the
callback on the directory itself. So if the callback returns fs.SkipDir,
those entries are thrown away, but the expense of reading them was
still incurred. (Worse, this is the expensive directory read that also
calls Stat on every entry.) On machines with slow file system I/O,
these reads are particularly annoying. For example, if I do

	go list m...

there is a call to filepath.Walk that is told about $GOROOT/src/archive
and responds by returning filepath.SkipDir because archive does not
start with m, but it only gets the chance to do that after the archive
directory has been read. (Same for all the other top-level directories.)
Even something like go list github.com/foo/bar/... reads every top-level
$GOPATH/src directory.

When we designed filepath.WalkDir, one of the changes we made was
to allow calling the callback twice for a directory: once before reading it,
and then possibly again if the read produces an error (uncommon).
This CL changes fsys.Walk to use that same model. None of the callbacks
need changing, but now the $GOROOT/src/archive and other top-level
directories won't be read when evaluating a pattern like 'm...'.

For golang#53577.
Fixes golang#53765.

Change-Id: Idfa3b9e2cc335417bfd9d66dd584cb16f92bd12e
Reviewed-on: https://go-review.googlesource.com/c/go/+/416179
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
jproberts pushed a commit to jproberts/go that referenced this issue Aug 10, 2022
The gccgo compiler does not load standard-library packages from
GOROOT/src, so we cannot load those packages from the GOROOT/src
index when using that compiler.

This fixes TestScript/gccgo_link_c (and perhaps other gccgo tests)
when a 'gccgo' executable is present. Unfortunately, only a few
builders caught the broken test because 'gccgo' is not installed
on most Go project builders (see golang#35786).

For golang#53577.
Fixes golang#53815.

Change-Id: I11a5cf6dbf4ac9893c4d02bd6ab7ef60f67b1e87
Reviewed-on: https://go-review.googlesource.com/c/go/+/417094
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
@golang golang locked and limited conversation to collaborators Aug 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly
Projects
None yet
Development

No branches or pull requests

2 participants