-
-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Volume server hangs #1353
Comments
Do we have pprof or other means to dump all threads to maybe get some clarity on what is causing the stalls? Is there an overview or some way to see transfers in flight internally (replication)? |
|
you can use
seems the write request is queued up on volume servers. Adding more volume servers or reduce the concurrency should help. |
@chrislusf log attached |
log from another volume server |
It happend again on the same server. This time we had patched the source to add pprof. This attachment is from debug/pprof/goroutine?debug=2 |
Its a deadlock in Its initally (write)locked here: https://github.com/chrislusf/seaweedfs/blob/master/weed/storage/volume_read_write.go#L75 but if the errorcondition its true here: https://github.com/chrislusf/seaweedfs/blob/master/weed/storage/volume_read_write.go#L79 v.ContentSize() attempts to (read)Lock the same mutex |
@Kimbsen Thanks a lot for showing the bug! I added a fix for it. |
We've been running our testing the whole day and seemingly no problems. Closing this task :) |
Describe the bug
The volume server http interface is unresponsive. The process is not dead as it is still logging lines like this:
System Setup
Same setup as : #1346
5x masters and 5x volume servers:
Expected behavior
Normal operation
Additional context
The only thing we changed was pulseSeconds from 10 to 5, but now I'm constantly restarting volume servers that hangs. Could it be that the volume server is waiting for replication to go through? I've seen some log lines related to replication failing. This is roughly 30 mins prior to all transfers stopping to the cluster:
The text was updated successfully, but these errors were encountered: