xrdfs 3.3.6 can hang on exit

[smithdh]

Hi,

I've seen xrdfs hang on exit for short duration batch mode queries, i.e.:

xrdfs query stats p
< prints the query result but then doesn't return >

This is with xrootd-client-3.3.6-1.el6.x86_64 on sl6. I have one machine where is seems to consistently happen; another where it only happens every ~tens of attempts and some other machines sl6 machines where I've not seen it at all. (The machines where I have seen the problem happen to be virtual machines). The trace from a stuck process looks like:

Thread 2 (Thread 0x7fa2508d4700 (LWP 51988)):
#0  0x00007fa25239e264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fa252399508 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fa2523993d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fa25213db99 in Lock (this=0x114b8b0, socket=0x114f660, notify=false, timeout=60)
    at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysPthread.hh:149
#4  XrdSysMutexHelper (this=0x114b8b0, socket=0x114f660, notify=false, timeout=60)
    at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysPthread.hh:208
#5  XrdCl::PollerBuiltIn::EnableWriteNotification (this=0x114b8b0, socket=0x114f660, notify=false, timeout=60)
    at /usr/src/debug/xrootd-3.3.6/src/XrdCl/XrdClPollerBuiltIn.cc:420
#6  0x00007fa252141688 in DisableUplink (this=0x114f730, subStream=0)
    at /usr/src/debug/xrootd-3.3.6/src/XrdCl/XrdClAsyncSocketHandler.hh:106
#7  XrdCl::Stream::OnReadyToWrite (this=0x114f730, subStream=0)
    at /usr/src/debug/xrootd-3.3.6/src/XrdCl/XrdClStream.cc:447
#8  0x00007fa2521724d4 in XrdCl::AsyncSocketHandler::OnWrite (this=0x114f8a0)
    at /usr/src/debug/xrootd-3.3.6/src/XrdCl/XrdClAsyncSocketHandler.cc:314
#9  0x00007fa25213ccf3 in (anonymous namespace)::SocketCallBack::Event (this=0x114fde0, 
    chP=<value optimized out>, cbArg=<value optimized out>, evFlags=<value optimized out>)
    at /usr/src/debug/xrootd-3.3.6/src/XrdCl/XrdClPollerBuiltIn.cc:78
#10 0x00007fa250f26b38 in XrdSys::IOEvents::Poller::CbkXeq (this=0x114b730, cP=0x114fe00, events=5, eNum=0, 
    eTxt=0x0) at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysIOEvents.cc:683
#11 0x00007fa250f26dc2 in XrdSys::IOEvents::PollE::Dispatch (this=0x114b730, cP=0x114fe00, 
    pollEv=<value optimized out>) at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysIOEventsPollE.icc:247
#12 0x00007fa250f2701a in XrdSys::IOEvents::PollE::Begin (this=0x114b730, syncsem=<value optimized out>, 
    retcode=<value optimized out>, eTxt=<value optimized out>)
    at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysIOEventsPollE.icc:209
#13 0x00007fa250f25ad2 in XrdSys::IOEvents::BootStrap::Start (parg=0x7fff2be7e090)
    at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysIOEvents.cc:130
#14 0x00007fa250f23b8f in XrdSysThread_Xeq (myargs=0x114b7f0)
    at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysPthread.cc:86
#15 0x00007fa2523979d1 in start_thread () from /lib64/libpthread.so.0
#16 0x00007fa251259b6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fa2527b2720 (LWP 51987)):
#0  0x00007fa25239d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007fa250f26176 in Wait (this=0x114b730, cmd=...)
    at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysPthread.hh:345
#2  XrdSys::IOEvents::Poller::SendCmd (this=0x114b730, cmd=...)
    at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysIOEvents.cc:958
#3  0x00007fa250f262cb in XrdSys::IOEvents::PollE::Exclude (this=0x114b730, cP=0x114fe00, 
    isLocked=@0x7fff2be7e5ff, dover=true) at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysIOEventsPollE.icc:276
#4  0x00007fa250f25f81 in XrdSys::IOEvents::Channel::~Channel (this=<value optimized out>, 
    __in_chrg=<value optimized out>) at /usr/src/debug/xrootd-3.3.6/src/XrdSys/XrdSysIOEvents.cc:296
#5  0x00007fa25213d161 in XrdCl::PollerBuiltIn::Stop (this=0x114b8b0)
    at /usr/src/debug/xrootd-3.3.6/src/XrdCl/XrdClPollerBuiltIn.cc:206
#6  0x00007fa2521324aa in XrdCl::DefaultEnv::Finalize ()
    at /usr/src/debug/xrootd-3.3.6/src/XrdCl/XrdClDefaultEnv.cc:443
#7  0x00007fa2511a71bd in __cxa_finalize () from /lib64/libc.so.6
#8  0x00007fa25212a996 in __do_global_dtors_aux () from /usr/lib64/libXrdCl.so.1
#9  0x0000000000000000 in ?? ()

Let me know if you'd like some more information (since I seem to have a machine or environment where this happens often, I could probably get details).

Thanks,
David

[ljanyst]

Hi David,

thanks for the report. This is a known issue an fixed by: c151b86

Cheers,
Lukasz

[smithdh]

ok, thanks.