Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

os: TestRemoveAllRace failures with "invalid argument" on dragonfly-amd64 #50716

Closed
bcmills opened this issue Jan 20, 2022 · 12 comments
Closed

os: TestRemoveAllRace failures with "invalid argument" on dragonfly-amd64 #50716

bcmills opened this issue Jan 20, 2022 · 12 comments
Labels
Builders x/build issues (builders, bots, dashboards) early-in-cycle A change that should be done early in the 3 month dev cycle. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly
Milestone
Go1.19

Comments

@bcmills
Copy link
Contributor

bcmills commented Jan 20, 2022 

--- FAIL: TestRemoveAllRace (0.01s)
    os_test.go:2412: unexpected error: *fs.PathError, "readdirnames /tmp/workdir-host-dragonfly-amd64-master/tmp/issue3518556883/b/b/b/b: readdirent b: invalid argument"
FAIL
FAIL	os	1.031s

greplogs --dashboard -md -l -e 'FAIL: TestRemoveAllRace.*\n( .*\n)* .*: invalid argument'

2022-01-19T20:07:20-d1640d8/dragonfly-amd64
2021-12-02T18:02:46-c5c1955/dragonfly-amd64
2021-11-23T21:59:51-1ac45e0/dragonfly-amd64
2019-08-28T02:29:25-731e6fc/dragonfly-amd64

The test is here:
https://cs.opensource.google/go/go/+/master:src/os/os_test.go;l=2385-2418;drc=c8861432b84157363c84220e6ed71180531bf2b4

The readdirent part of that error message appears to come from here:
https://cs.opensource.google/go/go/+/master:src/os/dir_unix.go;l=73;drc=2580d0e08d5e9f979b943758d3c49877fb2324cb

@tuxillo, any idea what might be causing this?
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly labels Jan 20, 2022
@bcmills bcmills modified the milestones: Go1.19, Backlog Jan 20, 2022
@tuxillo
Copy link
Contributor

tuxillo commented Jan 20, 2022

Is this a new test? Has it happened before? The /tmp mount point in the Gobuilder is tmpfs btw.

@bcmills
Copy link
Contributor Author

bcmills commented Jan 20, 2022

This test is not new — see the above links for the failures in the dashboard logs. I see one failure this week, two toward the end of last year, and one from 2019 — so this looks like it used to be a somewhat rare race and is now a more common one (possibly due to scheduling effects from the GC pacing changes in Go 1.18).

dragonfly is the only builder exhibiting this failure mode, and the test is not skipped on any other Unix platform.

@tuxillo
Copy link
Contributor

tuxillo commented Jan 20, 2022

Ok, we'll see what we can find out. Thanks for reporting!

@tuxillo
Copy link
Contributor

tuxillo commented Jan 24, 2022

There has been some work that could probably fix this issue, but it's in current master:

https://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/e91c5b2636e1fdd0d9e5170f24e5e39987211a49

whereas the gobuilder is running an older kernel (October, 2021). If I understood correctly, you prefer that we don't upgrade it yet.

@bcmills
Copy link
Contributor Author

bcmills commented Jan 24, 2022

Yeah, since we can mitigate this particular failure with a test-skip it's probably better to hold off on upgrading until after the Go 1.18 release. (That way if the system upgrade introduces any other test failures we can diagnose them independent of any further changes in the Go 1.18 release itself.)

@tuxillo
Copy link
Contributor

tuxillo commented Jan 24, 2022

Fine for me. Let's keep this issue opened and let me know when I can do the upgrade.

@bcmills bcmills added the early-in-cycle A change that should be done early in the 3 month dev cycle. label Jan 26, 2022
@bcmills bcmills modified the milestones: Backlog, Go1.19 Jan 26, 2022
@bcmills bcmills added the Builders x/build issues (builders, bots, dashboards) label Feb 2, 2022
@gopherbot
Copy link
Contributor

This issue is currently labeled as early-in-cycle for Go 1.19.
That time is now, so a friendly reminder to look at it again.

@tuxillo
Copy link
Contributor

tuxillo commented Mar 17, 2022

Can I upgrade the builder already?

@bcmills
Copy link
Contributor Author

bcmills commented Mar 17, 2022

Yes, now would be a great time! 😃

@tuxillo
Copy link
Contributor

tuxillo commented Mar 21, 2022

Done! :)

@bcmills
Copy link
Contributor Author

bcmills commented Mar 30, 2022

Looks like that did the trick!

greplogs --dashboard -md -l -e 'FAIL: TestRemoveAllRace.*\n( .*\n)* .*: invalid argument' --since=2022-03-21

[no results]

@bcmills bcmills closed this as completed Mar 30, 2022
@bcmills bcmills mentioned this issue Apr 12, 2022
Open
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/402155 mentions this issue: os: skip TestRemoveAllRace on dragonfly

gopherbot pushed a commit that referenced this issue Apr 25, 2022
@gopherbot
os: skip TestRemoveAllRace on dragonfly
35f2aba 
This test occasionally fails on the dragonfly-amd64 builder with
"directory not empty". Since that is the only platform on which we
observe these failures, and since the test had a different (and also
invalid-looking) failure mode prior to this one (in #50716), we
suspect that it is due to either a bug in the platform or a
platform-specific Go bug.

For #52301.

Change-Id: Id36c499651b9c48e6b8b0107d01f73d2a7b6bab8
Reviewed-on: https://go-review.googlesource.com/c/go/+/402155
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
@golang golang locked and limited conversation to collaborators Apr 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) early-in-cycle A change that should be done early in the 3 month dev cycle. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly
Projects
None yet
Milestone
Go1.19
Development

No branches or pull requests
3 participants
@tuxillo @bcmills @gopherbot