tasks/cephfs: time out on ceph-fuses that don't die #453
Conversation
|
Refer to this link for build results (access rights to CI server needed): |
| except MaxWhileTries: | ||
| log.error("process failed to terminate after unmount. This probably" | ||
| "indicates a bug within ceph-fuse.") | ||
| raise |
There was a problem hiding this comment.
What does this actually do to the teuthology test runs that hit it? I've no idea but I think most throws in the post-yield cleanup result in hung jobs and nothing getting cleaned up...
There was a problem hiding this comment.
@gregsfortytwo in run_tasks.py, in the part following the "Unwinding manager..." log message, it looks to me like exceptions in teardown are explicitly handled.
Entirely possible for a rogue process to be left lying around, but that's why we have nuke
|
Looks like this is busting things up; several failures in ubuntu-2015-06-09_11:25:56-fs-greg-fs-testing---basic-multi.
|
For cases where we have e.g. poked the fuse abort file for a process, but it's still not dying. Because this is a special class of error (unlike e.g. when we force umount something because the network is gone) raise the error instead of trying again to kill the client. Fixes: #11835 Signed-off-by: John Spray <john.spray@redhat.com>
|
Refer to this link for build results (access rights to CI server needed): |
|
Oops, that was a typo to run.wait() (it wants a list of processes). Updated. |
Our ffsb and fsync tests contain so many small writes at random offsets that it can take >10 minutes to commit all of them to disk if we get a slower OSD cluster. 15 minutes is still a plenty-fast timeout for this stage compared to just hanging and losing the logs! Signed-off-by: Greg Farnum <gfarnum@redhat.com>
|
Refer to this link for build results (access rights to CI server needed): |
tasks/cephfs: time out on ceph-fuses that don't die Reviewed-by: Greg Farnum <gfarnum@redhat.com>
For cases where we have e.g. poked the fuse abort
file for a process, but it's still not dying. Because
this is a special class of error (unlike e.g. when
we force umount something because the network is gone)
raise the error instead of trying again to kill
the client.
Fixes: #11835
Signed-off-by: John Spray john.spray@redhat.com