Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

goofys crash ForgetInode defer runtime error: invalid memory address or nil pointer dereference #552

Open
wph95 opened this issue Sep 6, 2020 · 2 comments · May be fixed by #553
Open

goofys crash ForgetInode defer runtime error: invalid memory address or nil pointer dereference #552

wph95 opened this issue Sep 6, 2020 · 2 comments · May be fixed by #553

Comments

@wph95
Copy link

wph95 commented Sep 6, 2020 

INFO  [2020-09-03 19:17:38,974] echo: /usr/bin/goofys[39]: main.ERROR stacktrace from panic: runtime error: invalid memory address or nil pointer dereference 
goroutine 53 [running]:
runtime/debug.Stack(0xc0000a9bd8, 0xc224a0, 0x14517c0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/kahing/goofys/api/common.LogPanic(0xc0000a9e48)
	/home/kahing/go/src/github.com/kahing/goofys/api/common/panic_logger.go:32 +0x7c
panic(0xc224a0, 0x14517c0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/kahing/goofys/internal.(*Goofys).ForgetInode(0xc0000b9ae0, 0xeb5b00, 0xc003b6b2c0, 0xc003be4500, 0x0, 0x0)
	/home/kahing/go/src/github.com/kahing/goofys/internal/goofys.go:763 +0xef
github.com/kahing/goofys/api/common.FusePanicLogger.ForgetInode(0xec7540, 0xc0000b9ae0, 0xeb5b00, 0xc003b6b2c0, 0xc003be4500, 0x0, 0x0)
	/home/kahing/go/src/github.com/kahing/goofys/api/common/panic_logger.go:61 +0x9d
github.com/kahing/goofys/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc00049c100, 0xc0001f61a0, 0xeb5b00, 0xc003b6b2c0, 0xbb6900, 0xc003be4500)
	/home/kahing/go/src/github.com/kahing/goofys/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:152 +0xa07
github.com/kahing/goofys/vendor/github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps(0xc00049c100, 0xc0001f61a0)
	/home/kahing/go/src/github.com/kahing/goofys/vendor/github.com/jacobsa/fuse/fuseutil/file_system.go:120 +0x140
github.com/kahing/goofys/vendor/github.com/jacobsa/fuse.Mount.func1(0xea5560, 0xc00049c100, 0xc0001f61a0, 0xc000276450)
	/home/kahing/go/src/github.com/kahing/goofys/vendor/github.com/jacobsa/fuse/mount.go:84 +0x3f
created by github.com/kahing/goofys/vendor/github.com/jacobsa/fuse.Mount
	/home/kahing/go/src/github.com/kahing/goofys/vendor/github.com/jacobsa/fuse/mount.go:83 +0x4fe

I'm continuing to try to reproduce the bug reliably.
@wph95
Copy link
Author

wph95 commented Sep 7, 2020 •
edited

A guess.
During the execution of inode.Parent.mu.Lock() and inode.Parent.mu.UnLock()
have other operations were running in parallel that caused modifications to inode.parent, make inode.parent to nil

@wph95 wph95 linked a pull request Sep 7, 2020 that will close this issue
Open
@wph95
Copy link
Author

wph95 commented Oct 20, 2020 •
edited

add some context to this problem

  1. we run the goofys in a docker container on the k8s.
  2. we met this problem in our customer and our test cluster. not frequently (1-3 times a week)
  3. Our Architecture
    A container:
    still create a folder to root with about 10 files every 30 seconds - 1 minute
    B container: [have this runtime crash problem ]
    just read the file A create. every 15-minute list root path to known which folder has been deleted
    S3 bucket:
    set a retention rule [7 day] on s3

guess when list root path, a lot of folders has been deleted. create a lot of ForgetInode op, make something race condition.

still working on finding a way to easily reproduce this problem and try to get the fuse_debug log when met this problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix ForgetInode func inode.parent sometimes nil, cause runtime error wph95/goofys
1 participant
@wph95