Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: RemoveAll failures with EPERM on OpenBSD #49751

Open
mknyszek opened this issue Nov 23, 2021 · 6 comments
Open

os: RemoveAll failures with EPERM on OpenBSD #49751

mknyszek opened this issue Nov 23, 2021 · 6 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-OpenBSD
Milestone

Comments

@mknyszek
Copy link
Contributor

mknyszek commented Nov 23, 2021

#!watchflakes
post <- goos == "openbsd" && `unlinkat .*: operation not permitted`

Saw a trybot failure in TestMemProfileCheck: https://storage.googleapis.com/go-build-log/f46e4835/openbsd-amd64-68_61f03b52.log

Maybe not a real issue, and since it seems like the problem is when invoking the system ld, I'm inclined to believe this isn't our problem.

CC @thanm

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 23, 2021
@thanm
Copy link
Contributor

thanm commented Nov 24, 2021

I looked at the testpoint in question -- it's not using the system "ld" as far as I can tell. Looks more like the test built a small Go program, ran it, and the resulting binary crashed (or did an early exit) without emitting any output. Tried greplogs:

greplogs -E TestMemProfileCheck -dashboard Output
2021-10-07T18:09:53-4607ebc/plan9-arm:
--- FAIL: TestMemProfileCheck (0.01s)
    --- FAIL: TestMemProfileCheck/with_memprofile_runtime_pprof (7.36s)
        ld_test.go:336: exit status: 'go 46650: 2'
FAIL
FAIL	cmd/link/internal/ld

2021-09-08T22:41:27-47f3e1e/freebsd-amd64-race:
--- FAIL: TestMemProfileCheck (0.00s)
--- FAIL: TestMemProfileCheck/no_memprofile (8.26s)
ld_test.go:326: signal: segmentation fault (core dumped)
FAIL
FAIL cmd/link/internal/ld

2021-09-05T23:10:46-7619a45/freebsd-386-12_2:
--- FAIL: TestMemProfileCheck (0.00s)
--- FAIL: TestMemProfileCheck/with_memprofile (2.25s)
ld_test.go:326: signal: segmentation fault (core dumped)
FAIL
FAIL cmd/link/internal/ld

2021-06-14T22:42:48-7841cb1/freebsd-386-12_2:
--- FAIL: TestMemProfileCheck (0.00s)
--- FAIL: TestMemProfileCheck/with_memprofile_indirect (0.09s)
ld_test.go:324: exit status 2
FAIL
FAIL cmd/link/internal/ld

2021-04-21T20:44:00-122fca4/plan9-arm:
--- FAIL: TestMemProfileCheck (0.02s)
--- FAIL: TestMemProfileCheck/no_memprofile (4.28s)
ld_test.go:318: exit status: 'go 41673: 2'
FAIL
FAIL cmd/link/internal/ld

Looks like this may be another one of those "bad things happen on freebsd/openbsd" problems.

@bcmills
Copy link
Member

bcmills commented Feb 8, 2022

Looks like this may be another one of those "bad things happen on freebsd/openbsd" problems.

It does indeed seem to be specific to openbsd/386 these days, but now we have output from the failing command:

--- FAIL: TestMemProfileCheck (0.00s)
    --- FAIL: TestMemProfileCheck/with_memprofile (1.15s)
        ld_test.go:340: got "524288\ngo: failed to remove work dir: unlinkat /tmp/workdir/tmp/go-build3492844359/b001: operation not permitted"; want "524288"
FAIL
FAIL	cmd/link/internal/ld	15.002s

greplogs --dashboard -md -l -e 'FAIL: TestMemProfileCheck ' --since=2021-11-25

2022-02-07T18:09:07-fa04846/openbsd-386-70
2022-01-13T23:35:37-e550c30/openbsd-386-70
2021-12-22T18:43:55-0f3becf/openbsd-386-68

@bcmills bcmills added arch-386 Issues solely affecting the 32-bit x86 architecture OS-OpenBSD labels Feb 8, 2022
@bcmills
Copy link
Member

bcmills commented Feb 8, 2022

Just looking for the unlinkat … operation not permitted error string gives some additional failures.

greplogs --dashboard -md -l -E 'unlinkat .*: operation not permitted' --since=2021-01-01

2022-02-07T18:09:07-fa04846/openbsd-386-70
2022-01-13T23:35:37-e550c30/openbsd-386-70
2021-12-22T18:43:55-0f3becf/openbsd-386-68
2021-12-10T18:20:08-c473ca0/openbsd-386-70
2021-11-25T04:02:39-45bae64/openbsd-amd64-70-n1
2021-11-24T20:59:14-67dd9ee/openbsd-amd64-70-n1
2021-11-23T21:59:51-1ac45e0/openbsd-amd64-68
2021-11-10T18:24:14-6406e09/openbsd-386-68
2021-11-09T19:01:06-81f37a7/openbsd-386-68
2021-11-08T21:28:37-ccea0b2/openbsd-386-68
2021-11-04T21:52:36-8ad0a7e/openbsd-386-68

From the variety of failures — and the fact that some are on openbsd-amd64-n1, which was not believed to be affected by the memory-corruption bug — this looks like a bug in os.RemoveAll on OpenBSD.

@bcmills bcmills changed the title cmd/link: TestMemProfileCheck failure on OpenBSD os: RemoveAll failures with EPERM on OpenBSD Feb 8, 2022
@bcmills bcmills removed the arch-386 Issues solely affecting the 32-bit x86 architecture label Feb 8, 2022
@bcmills
Copy link
Member

bcmills commented Feb 8, 2022

(CC @4a6f656c)

@seankhliao seankhliao added this to the Unplanned milestone Aug 20, 2022
@bcmills
Copy link
Member

bcmills commented Sep 6, 2022

greplogs -l -E 'unlinkat .*: operation not permitted' --since=2022-02-08
2022-09-01T03:24:42-49ab44d-86e9e0e/openbsd-amd64-70
2022-08-30T19:01:26-248c34b-3c6a5cd/openbsd-386-70

(attn @golang/openbsd)

@gopherbot
Copy link

gopherbot commented Sep 20, 2022

Found new matching dashboard test flakes for:

#!watchflakes
post <- goos == "openbsd" && `unlinkat .*: operation not permitted`
2022-08-29 18:21 openbsd-386-70 tools@248c34b8 go@3c6a5cdb x/tools/go/packages.TestNewPackagesInOverlay (log)
--- FAIL: TestNewPackagesInOverlay (0.00s)
    --- FAIL: TestNewPackagesInOverlay/Modules (20.55s)
        --- FAIL: TestNewPackagesInOverlay/Modules/main_overlay (5.05s)
            invoke.go:230: 1.112629782s for GOROOT= GOPATH=/tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/modcache GO111MODULE=on GOPROXY=file:///tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/modproxy PWD=/tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/fake go list -f "{{context.GOARCH}} {{context.Compiler}}" -- unsafe
            invoke.go:230: 1.498030519s for GOROOT= GOPATH=/tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/modcache GO111MODULE=off GOPROXY=file:///tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/modproxy PWD=/tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/fake go list -e -f {{context.ReleaseTags}} -- unsafe
            invoke.go:230: 3.510514706s for GOROOT= GOPATH=/tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/modcache GO111MODULE=on GOPROXY=file:///tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/modproxy PWD=/tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/fake go list -overlay=/tmp/workdir/tmp/gopackages-3466373920/overlay.json -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports -compiled=true -test=true -export=false -deps=true -find=false -- /tmp/workdir/tmp/TestNewPackagesInOverlay_Modules2319073194/fake/e
            overlay_test.go:452: err: exit status 1: stderr: go: unlinkat /tmp/workdir/tmp/go-build3458287667: operation not permitted
2022-09-01 03:24 openbsd-amd64-70 tools@49ab44de go@86e9e0ea x/tools/gopls/internal/regtest/watch.TestCreateDependency (log)
goroutine profile: total 3
1 @ 0x430fd6 0x466325 0x5bf855 0x5bf66d 0x5bc5cb 0xce157d 0xce11e7 0x51cbeb 0x46cb41
#	0x466324	runtime/pprof.runtime_goroutineProfileWithLabels+0x24			/tmp/workdir/go/src/runtime/mprof.go:846
#	0x5bf854	runtime/pprof.writeRuntimeProfile+0xb4					/tmp/workdir/go/src/runtime/pprof/pprof.go:723
#	0x5bf66c	runtime/pprof.writeGoroutine+0x4c					/tmp/workdir/go/src/runtime/pprof/pprof.go:683
#	0x5bc5ca	runtime/pprof.(*Profile).WriteTo+0x14a					/tmp/workdir/go/src/runtime/pprof/pprof.go:330
#	0xce157c	golang.org/x/tools/internal/lsp/regtest.(*Runner).Run.func1.1+0x7c	/tmp/workdir/gopath/src/golang.org/x/tools/internal/lsp/regtest/runner.go:308
#	0xce11e6	golang.org/x/tools/internal/lsp/regtest.(*Runner).Run.func1+0xca6	/tmp/workdir/gopath/src/golang.org/x/tools/internal/lsp/regtest/runner.go:355
#	0x51cbea	testing.tRunner+0x10a							/tmp/workdir/go/src/testing/testing.go:1446

...
1 @ 0x43b336 0x40853b 0x408038 0x51d9da 0xce0445 0xce5b65 0xce5b35 0x51cbeb 0x46cb41
#	0x51d9d9	testing.(*T).Run+0x379								/tmp/workdir/go/src/testing/testing.go:1494
#	0xce0444	golang.org/x/tools/internal/lsp/regtest.(*Runner).Run+0x3e4			/tmp/workdir/gopath/src/golang.org/x/tools/internal/lsp/regtest/runner.go:268
#	0xce5b64	golang.org/x/tools/internal/lsp/regtest.Run+0x44				/tmp/workdir/gopath/src/golang.org/x/tools/internal/lsp/regtest/regtest.go:54
#	0xce5b34	golang.org/x/tools/gopls/internal/regtest/watch.TestCreateDependency+0x14	/tmp/workdir/gopath/src/golang.org/x/tools/gopls/internal/regtest/watch/watch_test.go:201
#	0x51cbea	testing.tRunner+0x10a								/tmp/workdir/go/src/testing/testing.go:1446

--- FAIL: TestCreateDependency (2.09s)
    --- FAIL: TestCreateDependency/default (2.09s)
        runner.go:309: closing the sandbox: error(s) cleaning sandbox: cleaning modcache: <nil>; removing files: unlinkat /tmp/workdir/tmp/gopls-regtest-2881688214/TestCreateDependency/default/work/b: operation not permitted

watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-OpenBSD
Projects
Status: Active
Development

No branches or pull requests

5 participants