Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: Readdirnames throws errors on CIFS #39237

Closed
rfjakob opened this issue May 24, 2020 · 12 comments
Closed

os: Readdirnames throws errors on CIFS #39237

rfjakob opened this issue May 24, 2020 · 12 comments
Milestone

Comments

@rfjakob
Copy link

@rfjakob rfjakob commented May 24, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jakob/.cache/go-build"
GOENV="/home/jakob/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jakob/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jakob/go/src/github.com/rfjakob/gocryptfs/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build655782218=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run readdirnames.go against a cifs mount:

while sleep 1 ; do ./readdirnames /mnt/synology/public/tmp/g1 ; done

Mount info:

//192.168.0.3/public on /mnt/synology/public type cifs (rw,relatime,vers=3.1.1,cache=strict,username=jakob,uid=1026,forceuid,gid=1026,forcegid,addr=192.168.0.3,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)

Linux kernel:

5.6.13-300.fc32.x86_64

What did you expect to see?

No errors, consistent results, like from the C program getdents.c

$ while sleep 1 ; do ./getdents_c /mnt/synology/public/tmp/g1 ; done
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=4176, errno=0
getdents64 fd3: n=3192, errno=0
getdents64 fd3: n=0, errno=0
total 24072 bytes

[repeats ad infinitum]

What did you see instead?

readdirnames.go :

$ while sleep 1 ; do ./readdirnames /mnt/synology/public/tmp/g1 ; done
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=694, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=520, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=694, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
2020/05/24 23:59:25 os.Open returned err=open /mnt/synology/public/tmp/g1: interrupted system call
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>

The problem is easier to trigger when the terminal window is continously resized (this triggers SIGWINCH signals).

This is problably the same underlying issue as in #24015 , and I agree that it looks like a kernel bug.

However, the C program seems to handle this fine, even with window resizing stress. So if glibc has workarounds for this, maybe Go should have them as well.

Downstream issue: rfjakob/gocryptfs#483

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 25, 2020

This seems like the same issue as #38836. Is it possible for you to check whether the problem is fixed on master? Thanks.

@rfjakob
Copy link
Author

@rfjakob rfjakob commented May 25, 2020

Yes can do. Looks the same on master:

$ go build
$ go version
go version devel +f65ad0dda7 Sun May 24 20:49:55 2020 +0000 linux/amd64
$ while sleep 1 ; do ./readdirnames /mnt/synology/public/tmp/g1 ; done
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
2020/05/25 08:20:36 os.Open returned err=open /mnt/synology/public/tmp/g1: interrupted system call
Readdirnames: len=172, err=readdirent: no such file or directory
Readdirnames: len=868, err=readdirent: no such file or directory
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=346, err=readdirent: no such file or directory
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
Readdirnames: len=1001, err=<nil>
^C
@odeke-em odeke-em changed the title os.Readdirnames throws errors on CIFS os: Readdirnames throws errors on CIFS May 26, 2020
@rfjakob
Copy link
Author

@rfjakob rfjakob commented Jun 7, 2020

I posted a summary of the issue to linux-cifs and linux-kernel a few day ago:
https://lore.kernel.org/linux-cifs/2d735cb666872099b2c4c574e35aaeac2f7356ad.camel@gmail.com/

No reply yet, but I'm pretty sure now that this is a kernel bug. Does it make sense to keep this ticket open for tracking the issue?

@networkimprov
Copy link

@networkimprov networkimprov commented Jun 7, 2020

@gopherbot remove WaitingForInfo

@toothrot toothrot added this to the Backlog milestone Jun 8, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Jun 8, 2020

Change https://golang.org/cl/236997 mentions this issue: os: always check for EINTR in calls to open

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 8, 2020

I'm going to add a check for EINTR in calls to open. I think that should fix the problem with Go.

@rfjakob
Copy link
Author

@rfjakob rfjakob commented Jun 8, 2020

Nice, that should fix the open problem. The ENOENT in the middle of readdirent is tricky, though.

Readdirnames: len=868, err=readdirent: no such file or directory
gopherbot pushed a commit that referenced this issue Jun 8, 2020
For #11180
For #20400
For #39237

Change-Id: I8de97517c8f92c08f0c8a51f651a17e31617979b
Reviewed-on: https://go-review.googlesource.com/c/go/+/236997
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jun 8, 2020

Quite right, sorry, it will only fix the problem with os.Open. I don't know what to do with the readdirent problem, but it sounds like that is a kernel bug, and not something we can avoid in Go. So I'm going to close this issue now. Please comment if you disagree.

tmm1 added a commit to fancybits/go that referenced this issue Jul 9, 2020
For golang#11180
For golang#20400
For golang#39237

Change-Id: I8de97517c8f92c08f0c8a51f651a17e31617979b
Reviewed-on: https://go-review.googlesource.com/c/go/+/236997
Reviewed-by: Bryan C. Mills <bcmills@google.com>
(cherry picked from commit e64675a)
@ncw
Copy link
Contributor

@ncw ncw commented Jul 29, 2020

Rclone users have been having the same problem (see forum)

Using go1.15rc1 appears to fix the EINTR errors but does not fix the ENOENT errors.

Using export GODEBUG=asyncpreemptoff=1 does fix the problem completely.

So I guess CIFS is incorrectly returning ENOENT instead of EINTR on receipt of a signal?

@rfjakob did you get anywhere with trying to report this to the CIFS developers or upstream? I don't see any replies to your message.

@rfjakob
Copy link
Author

@rfjakob rfjakob commented Jul 29, 2020

Hi, no, I did not get any response to my posting, unfortunately. I don't really know what going on inside CIFS, but from the debug log I got out of the kernel, this line looks like something goes wrong internally:

[168517.187072] Status code returned 0x80000006 STATUS_NO_MORE_FILES

I don't understand why this does not happen with a C program. I can hammer getdents.c with SIGWINCH signals as much as I want, but I don't get ENOENT errors.

GODEBUG=asyncpreemptoff=1 also seems to fix it here, thanks for the tip!

@networkimprov
Copy link

@networkimprov networkimprov commented Jul 29, 2020

hammer getdents.c with SIGWINCH signals

The Go runtime uses SIGURG (I believe), so perhaps that's what's tripping CIFS?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 30, 2020

I don't understand why this does not happen with a C program. I can hammer getdents.c with SIGWINCH signals as much as I want, but I don't get ENOENT errors.

Note that an EINTR error only occurs if a signal handler is installed using the sigaction system call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.