Xrootd crashed (as manager) when using f-stream monitoring? #155

tommasoboccali · 2014-11-05T09:57:25Z

Ciao, as instructed by CMS, we changed on all our nodes the monitoring endpoint from

xrootd.monitor all auth flush 30s mbuff 1472 window 5s dest files io info redir user xrootd.t2.ucsd.edu:9930

to

xrootd.monitor all fstat 60s lfn ops ssq xfr 5 ident 5m dest fstat info user CMS-AAA-EU-COLLECTOR.cern.ch:9330

after than, a few of us noticed that both in redirectors and in site-redirectors we get a xrootd crash minutes after the start (we understand that configuration is probably unneeded on a redir, but many of us still have it for config consistency).

no output from the crash, just lines like

Nov 5 09:30:09 xrootd-redic kernel: [17619227.751667] xrootd[8524]: segfault at 8 ip 0000000000417743 sp 00007f2089c06cc0 error 6 in xrootd[400000+30000]

in the /var/log/messages

It does not seem to be a collector problem, indeed if we put

xrootd.monitor all auth flush 30s mbuff 1472 window 5s dest files io info redir user CMS-AAA-EU-COLLECTOR.cern.ch:9330

all is ok

is it a t-stream vs f-stream problem?

my xrootd version is

[root@xrootd-redic xrootd]# rpm -qa|grep xrootd
xrootd-client-libs-3.3.6-1.slc6.x86_64
xrootd-3.3.6-1.slc6.x86_64
xrootd-client-3.3.6-1.slc6.x86_64
xrootd-libs-3.3.6-1.slc6.x86_64
xrootd-server-libs-3.3.6-1.slc6.x86_64

(not sure about the other cases...)

and the config says

all.role manager

thanks!

tom

abh3 · 2014-11-05T11:27:39Z

Hi Tommaso,

You are right, while the redirector doesn't need that directive neither
should it crash if it has it. I would greatly appreciate a pointer to a
core file so I can debug it. If you cannot find a core file then that
usually means the core limit was set to zero and needs to be set correctly
in the sysinit script.

Andy

On Wed, 5 Nov 2014, Tommaso Boccali wrote:

Ciao, as instructed by CMS, we changed on all our nodes the monitoring endpoint from

xrootd.monitor all auth flush 30s mbuff 1472 window 5s dest files io info redir user xrootd.t2.ucsd.edu:9930

to

xrootd.monitor all fstat 60s lfn ops ssq xfr 5 ident 5m dest fstat info user CMS-AAA-EU-COLLECTOR.cern.ch:9330

after than, a few of us noticed that both in redirectors and in site-redirectors we get a xrootd crash minutes after the start (we understand that configuration is probably unneeded on a redir, but many of us still have it for config consistency).

no output from the crash, just lines like

Nov 5 09:30:09 xrootd-redic kernel: [17619227.751667] xrootd[8524]: segfault at 8 ip 0000000000417743 sp 00007f2089c06cc0 error 6 in xrootd[400000+30000]

in the /var/log/messages

It does not seem to be a collector problem, indeed if we put

xrootd.monitor all auth flush 30s mbuff 1472 window 5s dest files io info redir user CMS-AAA-EU-COLLECTOR.cern.ch:9330

all is ok

is it a t-stream vs f-stream problem?

my xrootd version is

[root@xrootd-redic xrootd]# rpm -qa|grep xrootd
xrootd-client-libs-3.3.6-1.slc6.x86_64
xrootd-3.3.6-1.slc6.x86_64
xrootd-client-3.3.6-1.slc6.x86_64
xrootd-libs-3.3.6-1.slc6.x86_64
xrootd-server-libs-3.3.6-1.slc6.x86_64

(not sure about the other cases...)

and the config says

all.role manager

thanks!

tom

Reply to this email directly or view it on GitHub:
#155

tommasoboccali · 2014-11-05T13:23:38Z

ciao Andy!
so, here is the core file:

/afs/pi.infn.it/user/boccali/public/core.17299

you should be able to access it from lxplus.

I did some more checks:

the crash happens not soon, but after the first "disc"
strace says:

[pid 17302] write(2, "53", 2)           = 2
[pid 17302] write(2, " data bytes; status=", 20) = 20
[pid 17302] write(2, "4004", 4)         = 4
[pid 17302] write(2, "\n", 1)           = 1
[pid 17302] writev(8, [{"\2\0\17\244\0\0\0009", 8}, {"\0\0\4F", 4}, {"xrootd-cms.infn.it?tried=+1413xr"..., 53}], 3) = 65
[pid 17302] poll([{fd=8, events=POLLIN|POLLRDNORM}], 1, 3000) = 1 ([{fd=8, revents=POLLIN|POLLRDNORM}])
[pid 17302] recvfrom(8, "", 24, 0, NULL, NULL) = 0
[pid 17302] gettid()                    = 17302
[pid 17302] writev(2, [{"141105 14:20:27 17302 ", 22}, {"Xrootd", 6}, {"Xeq", 3}, {": ", 2}, {"boccali.3873:8@gridui2", 22}, {" ", 1}, {"disc", 4}, {" ", 1}, {"0:00:03", 7}, {"\n", 1}], 10) = 69
[pid 17302] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 17302 detached
[pid 17444] <... ???? resumed> )        = ? <unavailable>
[pid 17320] <... ???? resumed> )        = ? <unavailable>

and GDB:

Core was generated by `/usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-redir-cms-f'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000417743 in XrdXrootdMonFile::GetSlot(int) ()
Missing separate debuginfos, use: debuginfo-install xrootd-3.3.6-1.slc6.x86_64
(gdb) where
#0  0x0000000000417743 in XrdXrootdMonFile::GetSlot(int) ()
#1  0x0000000000417cb6 in XrdXrootdMonFile::Disc(unsigned int) ()
#2  0x000000000041c71a in XrdXrootdProtocol::Recycle(XrdLink*, int, char const*) ()
#3  0x00007f08a36b6a90 in XrdLink::Close(int) () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a36bacd5 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#5  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1

ciao ciao

tom

abh3 · 2014-11-05T13:57:14Z

Hi Tommaso,

What version of Linux ar you running?

Andy

On Wed, 5 Nov 2014, Tommaso Boccali wrote:

ciao Andy!
so, here is the core file:

/afs/pi.infn.it/user/boccali/public/core.17299

you should be able to access it from lxplus.

I did some more checks:

the crash happens not soon, but after the first "disc"

strace says:

[pid 17302] write(2, "53", 2) = 2
[pid 17302] write(2, " data bytes; status=", 20) = 20
[pid 17302] write(2, "4004", 4) = 4
[pid 17302] write(2, "\n", 1) = 1
[pid 17302] writev(8, [{"\2\0\17\244\0\0\0009", 8}, {"\0\0\4F", 4}, {"xrootd-cms.infn.it?tried=+1413xr"..., 53}], 3) = 65
[pid 17302] poll([{fd=8, events=POLLIN|POLLRDNORM}], 1, 3000) = 1 ([{fd=8, revents=POLLIN|POLLRDNORM}])
[pid 17302] recvfrom(8, "", 24, 0, NULL, NULL) = 0
[pid 17302] gettid() = 17302
[pid 17302] writev(2, [{"141105 14:20:27 17302 ", 22}, {"Xrootd", 6}, {"Xeq", 3}, {": ", 2}, {"boccali.3873:8@gridui2", 22}, {" ", 1}, {"disc", 4}, {" ", 1}, {"0:00:03", 7}, {"\n", 1}], 10) = 69
[pid 17302] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 17302 detached
[pid 17444] <... ???? resumed> ) = ?
[pid 17320] <... ???? resumed> ) = ?

ciao ciao

tom

Reply to this email directly or view it on GitHub:
#155 (comment)

tommasoboccali · 2014-11-05T13:58:28Z

[root@xrootd-redic xrootd]# uname -a
Linux xrootd-redic 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 x86_64 x86_64 GNU/Linux

[root@xrootd-redic xrootd]# cat /etc/issue
Scientific Linux release 6.3 (Carbon)
Kernel \r on an \m

xrootd-dev · 2014-11-05T14:07:00Z

Hi Tommaso,

That particular strace probably isn't relevant. I'm having difficulties
because you run Scientific Linux release 6.3 (Carbon) and I can't find
that version to run gdb in. So, while I am looking can you gdb the core
file and issue "thread apply all bt" and send me the output?

Andy

On Wed, 5 Nov 2014, Tommaso Boccali wrote:

[root@xrootd-redic xrootd]# uname -a
Linux xrootd-redic 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 x86_64 x86_64 GNU/Linux

[root@xrootd-redic xrootd]# cat /etc/issue
Scientific Linux release 6.3 (Carbon)
Kernel \r on an \m

Reply to this email directly or view it on GitHub:
#155 (comment)

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1

tommasoboccali · 2014-11-05T14:17:07Z

ok, huge output incoming ....

Thread 149 (Thread 0x7f089bd08700 (LWP 17396)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 148 (Thread 0x7f089c10c700 (LWP 17392)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 147 (Thread 0x7f089bf0a700 (LWP 17394)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 146 (Thread 0x7f089d924700 (LWP 17368)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 145 (Thread 0x7f089c20d700 (LWP 17391)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 144 (Thread 0x7f089d823700 (LWP 17369)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 143 (Thread 0x7f089c611700 (LWP 17387)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 142 (Thread 0x7f089d621700 (LWP 17371)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 141 (Thread 0x7f089c30e700 (LWP 17390)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 140 (Thread 0x7f089d41f700 (LWP 17373)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 139 (Thread 0x7f089aefa700 (LWP 17410)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 138 (Thread 0x7f089db26700 (LWP 17366)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 137 (Thread 0x7f089acf8700 (LWP 17412)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 136 (Thread 0x7f08a0550700 (LWP 17324)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 135 (Thread 0x7f089c510700 (LWP 17388)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 134 (Thread 0x7f089d31e700 (LWP 17374)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 133 (Thread 0x7f089c40f700 (LWP 17389)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 132 (Thread 0x7f089e02b700 (LWP 17361)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 131 (Thread 0x7f089b0fc700 (LWP 17408)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 130 (Thread 0x7f089e42f700 (LWP 17357)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 129 (Thread 0x7f089a9f5700 (LWP 17415)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 128 (Thread 0x7f089d21d700 (LWP 17375)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 127 (Thread 0x7f089b904700 (LWP 17400)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 126 (Thread 0x7f089e934700 (LWP 17352)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 125 (Thread 0x7f089e22d700 (LWP 17359)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 124 (Thread 0x7f089b2fe700 (LWP 17406)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 123 (Thread 0x7f089df2a700 (LWP 17362)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 122 (Thread 0x7f089c00b700 (LWP 17393)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 121 (Thread 0x7f089da25700 (LWP 17367)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 120 (Thread 0x7f089bb06700 (LWP 17398)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 119 (Thread 0x7f089fc47700 (LWP 17333)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 118 (Thread 0x7f089a8f4700 (LWP 17416)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 117 (Thread 0x7f089dd28700 (LWP 17364)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 116 (Thread 0x7f089b702700 (LWP 17402)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 115 (Thread 0x7f089e732700 (LWP 17354)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 114 (Thread 0x7f089b500700 (LWP 17404)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 113 (Thread 0x7f08a044f700 (LWP 17325)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 112 (Thread 0x7f089a2ee700 (LWP 17422)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 111 (Thread 0x7f089f843700 (LWP 17337)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 110 (Thread 0x7f089abf7700 (LWP 17413)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 109 (Thread 0x7f089de29700 (LWP 17363)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 108 (Thread 0x7f089a4f0700 (LWP 17420)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 107 (Thread 0x7f089ed38700 (LWP 17348)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 106 (Thread 0x7f089a6f2700 (LWP 17418)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 105 (Thread 0x7f08a004b700 (LWP 17329)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 104 (Thread 0x7f08995e1700 (LWP 17435)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 103 (Thread 0x7f089fa45700 (LWP 17335)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 102 (Thread 0x7f08a3f2b700 (LWP 17300)):
#0  0x00007f08a324b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f08a368956e in XrdSysCondVar::Wait(int) () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36b3736 in XrdBuffManager::Reshape() () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a36b3ae9 in XrdReshaper(void*) () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#5  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 101 (Thread 0x7f089e12c700 (LWP 17360)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 100 (Thread 0x7f08994e0700 (LWP 17436)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 99 (Thread 0x7f089e833700 (LWP 17353)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 98 (Thread 0x7f0899eea700 (LWP 17426)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 97 (Thread 0x7f0899be7700 (LWP 17429)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 96 (Thread 0x7f089c712700 (LWP 17386)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 95 (Thread 0x7f089f13c700 (LWP 17344)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 94 (Thread 0x7f08997e3700 (LWP 17433)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 93 (Thread 0x7f08a135e700 (LWP 17310)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 92 (Thread 0x7f089c914700 (LWP 17384)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 91 (Thread 0x7f0899ce8700 (LWP 17428)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 90 (Thread 0x7f089c813700 (LWP 17385)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 89 (Thread 0x7f089a0ec700 (LWP 17424)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 88 (Thread 0x7f08a024d700 (LWP 17327)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 87 (Thread 0x7f089ca15700 (LWP 17383)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 86 (Thread 0x7f08999e5700 (LWP 17431)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 85 (Thread 0x7f089e631700 (LWP 17355)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 84 (Thread 0x7f089cf1a700 (LWP 17378)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 83 (Thread 0x7f089f540700 (LWP 17340)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 82 (Thread 0x7f08989d5700 (LWP 17448)):
#0  0x00007f08a324ea5d in accept () from /lib64/libpthread.so.0
#1  0x00007f08a36a5154 in XrdNetSocket::Accept(int) () from /usr/lib64/libXrdUtils.so.1
#2  0x000000000040e2fd in XrdXrootdAdmin::Start(XrdNetSocket*) ()
#3  0x000000000040e466 in XrdXrootdInitAdmin(void*) ()
#4  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#5  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 81 (Thread 0x7f089a1ed700 (LWP 17423)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 80 (Thread 0x7f089fb46700 (LWP 17334)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 79 (Thread 0x7f089d11c700 (LWP 17376)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 78 (Thread 0x7f089f944700 (LWP 17336)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 77 (Thread 0x7f089b1fd700 (LWP 17407)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 76 (Thread 0x7f089f43f700 (LWP 17341)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 75 (Thread 0x7f08a1a65700 (LWP 17303)):
#0  0x00007f08a324ea5d in accept () from /lib64/libpthread.so.0
#1  0x00007f08a36a0f32 in XrdNet::do_Accept_TCP(XrdNetPeer&, int) () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36a17cd in XrdNet::Accept(XrdNetPeer&, int, int) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a36b3d88 in XrdInet::Accept(int, int, XrdSysSemaphore*) () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a3b11245 in XrdMain::DoIt() () from /usr/lib64/libXrdMain.so.1
#5  0x00007f08a36bacd5 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#6  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#7  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#8  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 74 (Thread 0x7f089cc17700 (LWP 17381)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 73 (Thread 0x7f089f742700 (LWP 17338)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 72 (Thread 0x7f089a5f1700 (LWP 17419)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 71 (Thread 0x7f089ea35700 (LWP 17351)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 70 (Thread 0x7f089cb16700 (LWP 17382)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 69 (Thread 0x7f089fd48700 (LWP 17332)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 68 (Thread 0x7f089aaf6700 (LWP 17414)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 67 (Thread 0x7f089f641700 (LWP 17339)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 66 (Thread 0x7f089ce19700 (LWP 17379)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 65 (Thread 0x7f089f33e700 (LWP 17342)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 64 (Thread 0x7f08a1c67700 (LWP 17301)):
#0  0x00007f08a324b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f08a368956e in XrdSysCondVar::Wait(int) () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36b9bfb in XrdScheduler::TimeSched() () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a36b9c09 in XrdStartTSched(void*) () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#5  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 63 (Thread 0x7f089adf9700 (LWP 17411)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 62 (Thread 0x7f089ec37700 (LWP 17349)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 61 (Thread 0x7f089eb36700 (LWP 17350)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 60 (Thread 0x7f089cd18700 (LWP 17380)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 59 (Thread 0x7f089be09700 (LWP 17395)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 58 (Thread 0x7f089ef3a700 (LWP 17346)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 57 (Thread 0x7f0899de9700 (LWP 17427)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x7f08992de700 (LWP 17438)):
#0  0x00007f08a25e1163 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f08a36b8335 in XrdPollE::Start(XrdSysSemaphore*, int&) () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36b7628 in XrdStartPolling(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 55 (Thread 0x7f089e530700 (LWP 17356)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 54 (Thread 0x7f089ee39700 (LWP 17347)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7f089d01b700 (LWP 17377)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7f089ba05700 (LWP 17399)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7f089ff4a700 (LWP 17330)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7f089affb700 (LWP 17409)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7f089e32e700 (LWP 17358)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7f089f23d700 (LWP 17343)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7f0898dd9700 (LWP 17443)):
#0  0x00007f08a324ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f08a22c384c in XrdOssCache::Scan(int) () from /usr/lib64/libXrdServer.so.1
#2  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7f089bc07700 (LWP 17397)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7f0899feb700 (LWP 17425)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7f089fe49700 (LWP 17331)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7f089d520700 (LWP 17372)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7f089f03b700 (LWP 17345)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7f08993df700 (LWP 17437)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7f089b3ff700 (LWP 17405)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7f0899ae6700 (LWP 17430)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7f089d722700 (LWP 17370)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7f08998e4700 (LWP 17432)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f089b601700 (LWP 17403)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f08a1964700 (LWP 17304)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f08a034e700 (LWP 17326)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f089a3ef700 (LWP 17421)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f089dc27700 (LWP 17365)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7f08996e2700 (LWP 17434)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7f089b803700 (LWP 17401)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f08a014c700 (LWP 17328)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f089a7f3700 (LWP 17417)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f08990dc700 (LWP 17440)):
#0  0x00007f08a25e1163 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f08a36b8335 in XrdPollE::Start(XrdSysSemaphore*, int&) () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36b7628 in XrdStartPolling(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f0898eda700 (LWP 17442)):
#0  0x00007f08a252bb57 in sigwaitinfo () from /lib64/libc.so.6
#1  0x00007f08a22c0186 in XrdOssAioWait(void*) () from /usr/lib64/libXrdServer.so.1
#2  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f0898fdb700 (LWP 17441)):
#0  0x00007f08a252bb57 in sigwaitinfo () from /lib64/libc.so.6
#1  0x00007f08a22c0186 in XrdOssAioWait(void*) () from /usr/lib64/libXrdServer.so.1
#2  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f08991dd700 (LWP 17439)):
#0  0x00007f08a25e1163 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f08a36b8335 in XrdPollE::Start(XrdSysSemaphore*, int&) () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36b7628 in XrdStartPolling(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f08a125d700 (LWP 17311)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f08a105b700 (LWP 17313)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f08a0853700 (LWP 17321)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f08a1661700 (LWP 17307)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f08a0c57700 (LWP 17317)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f08a0d58700 (LWP 17316)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f08a1762700 (LWP 17306)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f08a0e59700 (LWP 17315)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f08a0b56700 (LWP 17318)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f08a0954700 (LWP 17320)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f08a0752700 (LWP 17322)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f08a0a55700 (LWP 17319)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f08a115c700 (LWP 17312)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f08a0f5a700 (LWP 17314)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f08a145f700 (LWP 17309)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f0898bd7700 (LWP 17445)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a22df7b2 in XrdCmsResp::Reply() () from /usr/lib64/libXrdServer.so.1
#2  0x00007f08a22dcc49 in XrdCmsStartResp(void*) () from /usr/lib64/libXrdServer.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f08a3f2d720 (LWP 17299)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a3b10e13 in mainAccept(void*) () from /usr/lib64/libXrdMain.so.1
#2  0x00007f08a3b11195 in main () from /usr/lib64/libXrdMain.so.1
#3  0x00007f08a2516d1d in __libc_start_main () from /lib64/libc.so.6
#4  0x000000000040d289 in _start ()

Thread 6 (Thread 0x7f0898cd8700 (LWP 17444)):
#0  0x00007f08a324eb5c in recv () from /lib64/libpthread.so.0
#1  0x00007f08a36b4e26 in XrdLink::RecvAll(char*, int, int) () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a22db68d in XrdCmsClientMan::Receive() () from /usr/lib64/libXrdServer.so.1
#3  0x00007f08a22db848 in XrdCmsClientMan::Start() () from /usr/lib64/libXrdServer.so.1
#4  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#5  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f0898ad6700 (LWP 17446)):
#0  0x00007f08a324eb5c in recv () from /lib64/libpthread.so.0
#1  0x00007f08a22dec9e in XrdCmsFinderTRG::Start() () from /usr/lib64/libXrdServer.so.1
#2  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f08a0651700 (LWP 17323)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f08a1560700 (LWP 17308)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f08a1863700 (LWP 17305)):
#0  0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f08a1b66700 (LWP 17302)):
#0  0x0000000000417743 in XrdXrootdMonFile::GetSlot(int) ()
#1  0x0000000000417cb6 in XrdXrootdMonFile::Disc(unsigned int) ()
#2  0x000000000041c71a in XrdXrootdProtocol::Recycle(XrdLink*, int, char const*) ()
#3  0x00007f08a36b6a90 in XrdLink::Close(int) () from /usr/lib64/libXrdUtils.so.1
#4  0x00007f08a36bacd5 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#5  0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#6  0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#7  0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

xrootd-dev · 2014-11-05T14:55:01Z

Hi Tommaso,

OK, you will need to install the debug symbol RPM so I can get line
numbers. Additionally, we should get together tonight or tomorrow so I can
inspect the core file on your machine as I cannot get any of my systems to
make heads or tails out of it.

Andy

P.S. After installing debug symbols please just to a "where" on the thread
that got the segv to get a backtrace of that one.

On Wed, 5 Nov 2014, Tommaso Boccali wrote:

ok, huge output incoming ....

Thread 149 (Thread 0x7f089bd08700 (LWP 17396)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 148 (Thread 0x7f089c10c700 (LWP 17392)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 147 (Thread 0x7f089bf0a700 (LWP 17394)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 146 (Thread 0x7f089d924700 (LWP 17368)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 145 (Thread 0x7f089c20d700 (LWP 17391)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 144 (Thread 0x7f089d823700 (LWP 17369)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 143 (Thread 0x7f089c611700 (LWP 17387)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 142 (Thread 0x7f089d621700 (LWP 17371)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 141 (Thread 0x7f089c30e700 (LWP 17390)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 140 (Thread 0x7f089d41f700 (LWP 17373)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 139 (Thread 0x7f089aefa700 (LWP 17410)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 138 (Thread 0x7f089db26700 (LWP 17366)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 137 (Thread 0x7f089acf8700 (LWP 17412)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 136 (Thread 0x7f08a0550700 (LWP 17324)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 135 (Thread 0x7f089c510700 (LWP 17388)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 134 (Thread 0x7f089d31e700 (LWP 17374)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 133 (Thread 0x7f089c40f700 (LWP 17389)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 132 (Thread 0x7f089e02b700 (LWP 17361)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 131 (Thread 0x7f089b0fc700 (LWP 17408)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 130 (Thread 0x7f089e42f700 (LWP 17357)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 129 (Thread 0x7f089a9f5700 (LWP 17415)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 128 (Thread 0x7f089d21d700 (LWP 17375)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 127 (Thread 0x7f089b904700 (LWP 17400)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 126 (Thread 0x7f089e934700 (LWP 17352)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 125 (Thread 0x7f089e22d700 (LWP 17359)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 124 (Thread 0x7f089b2fe700 (LWP 17406)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 123 (Thread 0x7f089df2a700 (LWP 17362)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 122 (Thread 0x7f089c00b700 (LWP 17393)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 121 (Thread 0x7f089da25700 (LWP 17367)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 120 (Thread 0x7f089bb06700 (LWP 17398)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 119 (Thread 0x7f089fc47700 (LWP 17333)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 118 (Thread 0x7f089a8f4700 (LWP 17416)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 117 (Thread 0x7f089dd28700 (LWP 17364)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 116 (Thread 0x7f089b702700 (LWP 17402)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 115 (Thread 0x7f089e732700 (LWP 17354)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 114 (Thread 0x7f089b500700 (LWP 17404)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 113 (Thread 0x7f08a044f700 (LWP 17325)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 112 (Thread 0x7f089a2ee700 (LWP 17422)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 111 (Thread 0x7f089f843700 (LWP 17337)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 110 (Thread 0x7f089abf7700 (LWP 17413)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 109 (Thread 0x7f089de29700 (LWP 17363)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 108 (Thread 0x7f089a4f0700 (LWP 17420)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 107 (Thread 0x7f089ed38700 (LWP 17348)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 106 (Thread 0x7f089a6f2700 (LWP 17418)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 105 (Thread 0x7f08a004b700 (LWP 17329)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 104 (Thread 0x7f08995e1700 (LWP 17435)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 103 (Thread 0x7f089fa45700 (LWP 17335)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 102 (Thread 0x7f08a3f2b700 (LWP 17300)):
#0 0x00007f08a324b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f08a368956e in XrdSysCondVar::Wait(int) () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36b3736 in XrdBuffManager::Reshape() () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a36b3ae9 in XrdReshaper(void*) () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#5 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 101 (Thread 0x7f089e12c700 (LWP 17360)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 100 (Thread 0x7f08994e0700 (LWP 17436)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 99 (Thread 0x7f089e833700 (LWP 17353)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 98 (Thread 0x7f0899eea700 (LWP 17426)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 97 (Thread 0x7f0899be7700 (LWP 17429)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 96 (Thread 0x7f089c712700 (LWP 17386)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 95 (Thread 0x7f089f13c700 (LWP 17344)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 94 (Thread 0x7f08997e3700 (LWP 17433)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 93 (Thread 0x7f08a135e700 (LWP 17310)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 92 (Thread 0x7f089c914700 (LWP 17384)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 91 (Thread 0x7f0899ce8700 (LWP 17428)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 90 (Thread 0x7f089c813700 (LWP 17385)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 89 (Thread 0x7f089a0ec700 (LWP 17424)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 88 (Thread 0x7f08a024d700 (LWP 17327)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 87 (Thread 0x7f089ca15700 (LWP 17383)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 86 (Thread 0x7f08999e5700 (LWP 17431)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 85 (Thread 0x7f089e631700 (LWP 17355)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 84 (Thread 0x7f089cf1a700 (LWP 17378)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 83 (Thread 0x7f089f540700 (LWP 17340)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 82 (Thread 0x7f08989d5700 (LWP 17448)):
#0 0x00007f08a324ea5d in accept () from /lib64/libpthread.so.0
#1 0x00007f08a36a5154 in XrdNetSocket::Accept(int) () from /usr/lib64/libXrdUtils.so.1
#2 0x000000000040e2fd in XrdXrootdAdmin::Start(XrdNetSocket_) ()
#3 0x000000000040e466 in XrdXrootdInitAdmin(void_) ()
#4 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#5 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 81 (Thread 0x7f089a1ed700 (LWP 17423)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 80 (Thread 0x7f089fb46700 (LWP 17334)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 79 (Thread 0x7f089d11c700 (LWP 17376)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 78 (Thread 0x7f089f944700 (LWP 17336)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 77 (Thread 0x7f089b1fd700 (LWP 17407)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 76 (Thread 0x7f089f43f700 (LWP 17341)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 75 (Thread 0x7f08a1a65700 (LWP 17303)):
#0 0x00007f08a324ea5d in accept () from /lib64/libpthread.so.0
#1 0x00007f08a36a0f32 in XrdNet::do_Accept_TCP(XrdNetPeer&, int) () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36a17cd in XrdNet::Accept(XrdNetPeer&, int, int) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a36b3d88 in XrdInet::Accept(int, int, XrdSysSemaphore_) () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a3b11245 in XrdMain::DoIt() () from /usr/lib64/libXrdMain.so.1
#5 0x00007f08a36bacd5 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#6 0x00007f08a36baec9 in XrdStartWorking(void_) () from /usr/lib64/libXrdUtils.so.1
#7 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#8 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 74 (Thread 0x7f089cc17700 (LWP 17381)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 73 (Thread 0x7f089f742700 (LWP 17338)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 72 (Thread 0x7f089a5f1700 (LWP 17419)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 71 (Thread 0x7f089ea35700 (LWP 17351)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 70 (Thread 0x7f089cb16700 (LWP 17382)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 69 (Thread 0x7f089fd48700 (LWP 17332)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 68 (Thread 0x7f089aaf6700 (LWP 17414)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 67 (Thread 0x7f089f641700 (LWP 17339)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 66 (Thread 0x7f089ce19700 (LWP 17379)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 65 (Thread 0x7f089f33e700 (LWP 17342)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 64 (Thread 0x7f08a1c67700 (LWP 17301)):
#0 0x00007f08a324b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f08a368956e in XrdSysCondVar::Wait(int) () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36b9bfb in XrdScheduler::TimeSched() () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a36b9c09 in XrdStartTSched(void*) () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#5 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 63 (Thread 0x7f089adf9700 (LWP 17411)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 62 (Thread 0x7f089ec37700 (LWP 17349)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 61 (Thread 0x7f089eb36700 (LWP 17350)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 60 (Thread 0x7f089cd18700 (LWP 17380)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 59 (Thread 0x7f089be09700 (LWP 17395)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 58 (Thread 0x7f089ef3a700 (LWP 17346)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 57 (Thread 0x7f0899de9700 (LWP 17427)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x7f08992de700 (LWP 17438)):
#0 0x00007f08a25e1163 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f08a36b8335 in XrdPollE::Start(XrdSysSemaphore_, int&) () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36b7628 in XrdStartPolling(void_) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 55 (Thread 0x7f089e530700 (LWP 17356)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 54 (Thread 0x7f089ee39700 (LWP 17347)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7f089d01b700 (LWP 17377)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7f089ba05700 (LWP 17399)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7f089ff4a700 (LWP 17330)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7f089affb700 (LWP 17409)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7f089e32e700 (LWP 17358)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7f089f23d700 (LWP 17343)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7f0898dd9700 (LWP 17443)):
#0 0x00007f08a324ef3d in nanosleep () from /lib64/libpthread.so.0
#1 0x00007f08a22c384c in XrdOssCache::Scan(int) () from /usr/lib64/libXrdServer.so.1
#2 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7f089bc07700 (LWP 17397)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7f0899feb700 (LWP 17425)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7f089fe49700 (LWP 17331)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7f089d520700 (LWP 17372)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7f089f03b700 (LWP 17345)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7f08993df700 (LWP 17437)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7f089b3ff700 (LWP 17405)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7f0899ae6700 (LWP 17430)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7f089d722700 (LWP 17370)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7f08998e4700 (LWP 17432)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7f089b601700 (LWP 17403)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7f08a1964700 (LWP 17304)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7f08a034e700 (LWP 17326)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f089a3ef700 (LWP 17421)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f089dc27700 (LWP 17365)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7f08996e2700 (LWP 17434)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7f089b803700 (LWP 17401)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f08a014c700 (LWP 17328)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f089a7f3700 (LWP 17417)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f08990dc700 (LWP 17440)):
#0 0x00007f08a25e1163 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f08a36b8335 in XrdPollE::Start(XrdSysSemaphore_, int&) () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36b7628 in XrdStartPolling(void_) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f0898eda700 (LWP 17442)):
#0 0x00007f08a252bb57 in sigwaitinfo () from /lib64/libc.so.6
#1 0x00007f08a22c0186 in XrdOssAioWait(void*) () from /usr/lib64/libXrdServer.so.1
#2 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f0898fdb700 (LWP 17441)):
#0 0x00007f08a252bb57 in sigwaitinfo () from /lib64/libc.so.6
#1 0x00007f08a22c0186 in XrdOssAioWait(void*) () from /usr/lib64/libXrdServer.so.1
#2 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f08991dd700 (LWP 17439)):
#0 0x00007f08a25e1163 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f08a36b8335 in XrdPollE::Start(XrdSysSemaphore_, int&) () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36b7628 in XrdStartPolling(void_) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f08a125d700 (LWP 17311)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f08a105b700 (LWP 17313)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.1
#3 0x00007f08a368965f in XrdSysThread_Xeq () from /usr/lib64/libXrdUtils.so.1
#4 0x00007f08a32479d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f08a25e0b6d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f08a0853700 (LWP 17321)):
#0 0x00007f08a324d930 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f08a36bac56 in XrdScheduler::Run() () from /usr/lib64/libXrdUtils.so.1
#2 0x00007f08a36baec9 in XrdStartWorking(void*) () from /usr/lib64/libXrdUtils.so.

tommasoboccali · 2014-11-05T17:08:44Z

ciao, so here it is:

Core was generated by `/usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-redir-cms-f'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
    at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439
439        repTOD->tBeg = htonl(static_cast<int>(time(0)));

and the 'where' says

#0  0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
    at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439
#1  0x0000000000417cb6 in XrdXrootdMonFile::Disc (usrID=0)
    at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:200
#2  0x000000000041c71a in XrdXrootdProtocol::Recycle (this=0x7f0890000990, lp=
    0x7f0894000a88, csec=3, reason=<value optimized out>)

is this what you need? not sure how to find 'the thread that got the SEGV'.
I am leaving early tonight ... tomorrow ~ after 11 am ?

tom

abh3 · 2014-11-05T17:24:38Z

Hi Tomasso,

Too bad you are leaving tonight (weren't you supposed to talk tomorrow?).
Yes, this is what I need. You can find out which thread gt the SEGV by
doing "info threads" the one with the asterisk in the running thread that
failed. That's the one we need the traceback for (so "thread " followed
by where).

Andy

On Wed, 5 Nov 2014, Tommaso Boccali wrote:

ciao, so here it is:

Core was generated by `/usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-redir-cms-f'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439
439 repTOD->tBeg = htonl(static_cast(time(0)));

and the 'where' says

#0 0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439
#1 0x0000000000417cb6 in XrdXrootdMonFile::Disc (usrID=0)
at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:200
#2 0x000000000041c71a in XrdXrootdProtocol::Recycle (this=0x7f0890000990, lp=
0x7f0894000a88, csec=3, reason=)

is this what you need? not sure how to find 'the thread that got the SEGV'.
I am leaving early tonight ... tomorrow ~ after 11 am ?

tom

Reply to this email directly or view it on GitHub:
#155 (comment)

abh3 · 2014-11-05T17:29:55Z

Got it! Yes, tomorrow after 11 AM would be fine.

Andy

On Wed, 5 Nov 2014, Tommaso Boccali wrote:

ciao, so here it is:

Core was generated by `/usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-redir-cms-f'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439
439 repTOD->tBeg = htonl(static_cast(time(0)));

and the 'where' says

#0 0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439
#1 0x0000000000417cb6 in XrdXrootdMonFile::Disc (usrID=0)
at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:200
#2 0x000000000041c71a in XrdXrootdProtocol::Recycle (this=0x7f0890000990, lp=
0x7f0894000a88, csec=3, reason=)

is this what you need? not sure how to find 'the thread that got the SEGV'.
I am leaving early tonight ... tomorrow ~ after 11 am ?

tom

Reply to this email directly or view it on GitHub:
#155 (comment)

tommasoboccali · 2014-11-05T17:33:09Z

so, thread #1 is the crashing one, so the backtrace is exactly what i sent you before ...

* 1 Thread 0x7f08a1b66700 (LWP 17302)  0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
    at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439

so

(gdb) thread 1
[Switching to thread 1 (Thread 0x7f08a1b66700 (LWP 17302))]#0  0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
    at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439
439        repTOD->tBeg = htonl(static_cast<int>(time(0)));
(gdb) where
#0  0x0000000000417743 in XrdXrootdMonFile::GetSlot (slotSZ=8)
    at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:439
#1  0x0000000000417cb6 in XrdXrootdMonFile::Disc (usrID=0)
    at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdMonFile.cc:200
#2  0x000000000041c71a in XrdXrootdProtocol::Recycle (this=0x7f0890000990, lp=
    0x7f0894000a88, csec=3, reason=<value optimized out>)
    at /usr/src/debug/xrootd/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:548
#3  0x00007f08a36b6a90 in XrdLink::Close (this=0x7f0894000a88,
    defer=<value optimized out>)
    at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:385
#4  0x00007f08a36bacd5 in XrdScheduler::Run (this=0x7f08a3d138e0)
    at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:307
#5  0x00007f08a36baec9 in XrdStartWorking (carg=<value optimized out>)
    at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:84
#6  0x00007f08a368965f in XrdSysThread_Xeq (myargs=0x63ef30)
    at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86
#7  0x00007f08a32479d1 in start_thread (arg=0x7f08a1b66700)
    at pthread_create.c:301

tommasoboccali · 2014-11-07T08:51:36Z

exec summary. I sit with andy and looked in the problem. It seems we have hit an xrootd3 bug, already solved in xrootd4.
Basically, the problem was a missing "redir" key in the new monitoring line. We do NOT monitor redirs, but its absence was triggering the use of initialised code.

can be closed (sorry I do not know how ;)

Thanks a lot!

tom

tommasoboccali closed this as completed Nov 7, 2014

ljanyst added bug (critical) fixed labels Nov 10, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Xrootd crashed (as manager) when using f-stream monitoring? #155

Xrootd crashed (as manager) when using f-stream monitoring? #155

tommasoboccali commented Nov 5, 2014

abh3 commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

abh3 commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

xrootd-dev commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

xrootd-dev commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

abh3 commented Nov 5, 2014

abh3 commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

tommasoboccali commented Nov 7, 2014

Xrootd crashed (as manager) when using f-stream monitoring? #155

Xrootd crashed (as manager) when using f-stream monitoring? #155

Comments

tommasoboccali commented Nov 5, 2014

abh3 commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

abh3 commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

xrootd-dev commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

xrootd-dev commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

abh3 commented Nov 5, 2014

abh3 commented Nov 5, 2014

tommasoboccali commented Nov 5, 2014

tommasoboccali commented Nov 7, 2014