-
Notifications
You must be signed in to change notification settings - Fork 102
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
completeFilename causes invalid memory address or nil pointer dereference #49
Comments
completeFilename
causes invalid memory address or nil pointer dereference
@peterbourgon Do you have any smart ideas or suggestions where I should start looking in the code to find the root cause? |
That stack trace identifies the callstack, but it doesn't give the specific error that gets bubbled up. Do you have that? |
All I have is the panic error message:
|
And which version of diskv? |
I'm using the latest version: |
Hmm. There is not much to go on. According to your stack trace, the nil pointer dereference is in the stdlib filepath.Join, and I'm not sure how that makes sense. Does this happen when you pass an empty key, for example? Can you paste the complete verbatim crash log? |
yeah, the error is quite bizarre. We had 5 crashes of k8s pod during last 2 days. Here is the complete stack trace (there are slightly naming modifications to avoid exposing private data):
I already tried to reproduce by passing an empty key, but the crash doesn't occur. For what's worth, I also identified the exact place in Go stdlib ( func join(elem []string) string {
// If there's a bug here, fix the logic in ./path_plan9.go too.
for i, e := range elem {
if e != "" {
return Clean(strings.Join(elem[i:], string(Separator))) <- panic
}
}
return ""
} The last that occur to me is that this could be something k8s-specific, since data dir is mapped to host volume inside daemonset. |
I feel safe asserting that crashing in strings.Join is a red herring, the problem is elsewhere. If it's happening in a Kubernetes volume mount, my guess is something at the filesystem layer. My suggestion would be to come up with some patch for Erase that logged a bunch of debug information, and have you run that build for a day or two, and see what it says when it goes kaput. Would this work? |
Sounds good. Let me reach out to you again after I gather some relevant debug info. |
The conclusion we have is that a nasty race condition is occurring in the code and thus making the key invalid (runtime attempts to dereference a Thanks for being supportive. |
🙇 |
Hi
We're running a component in Kubernetes that uses
diskv
under the hood. The problem is that the process occasionally crashes when it attempts to remove the key from the store. Here is the relevant stack trace:Data directory is mounted as a regular host path (
/opt/spm/agent
) and file names are ksuid-compatible identifiers.diskv
is initialized with following configuration:Do you have any pointers or ideas why this would happen?
The text was updated successfully, but these errors were encountered: