Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xrootd-4.2 and the same supervisor problem as it was in 4.1 #291

Closed
tsarangi opened this issue Sep 23, 2015 · 7 comments
Closed

xrootd-4.2 and the same supervisor problem as it was in 4.1 #291

tsarangi opened this issue Sep 23, 2015 · 7 comments

Comments

@tsarangi
Copy link

Dear Developers,

Please refer to this issue #227 that had a solution (patch) from @abh3. I see the same problem with the xrootd-4.2.2 version : xrootd-server-4.2.2-1.osg32.el6.x86_64.

I had also requested the patch from Andy to be included for this OSG release, but didn't get any response.
https://listserv.slac.stanford.edu/cgi-bin/wa?A2=ind1504&L=XROOTD-DEV&F=&S=&P=63754

Please suggest what can we do to solve this issue ? We are now constrained to only 64 servers.

-Tapas
@tsarangi
Copy link
Author

If it helps:
On the redirector, I see that once the nodecnt goes more than 64, cmsd process exits. Here is a snippet of log on the redirector.

150923 11:07:12 23598 Protocol: Primary T2_US_Wisconsin server.21473:86@g26n29:31094 logged in.
=====> Routing for g26n29.hep.wisc.edu: local pub4 prv4 pub6 prv6
=====> Route all4: g26n29.hep.wisc.edu Dest=[::144.92.180.131]:31094
=====> Route all6: g26n29.hep.wisc.edu Dest=[2607:f388:101c:1000::303]:31094
150923 11:07:12 23602 AddNode srv server.30381:91@g26n07:31094 cluster 1213cmsxrootd.hep.wisc.edu mask=ffffffffffffffff anum=0
150923 11:07:12 23602 Add server.30381:91@g26n07:31094 to cluster anon-s 1213cmsxrootd.hep.wisc.edu slot 63.65 (nodecnt=64 supn=1)
150923 11:07:12 23602 Update Counts Parm1=1 Parm2=0
150923 11:07:12 23602 Admit g26n07 TSpace=1GB NumFS=0 FSpace=0MB MinFR=0 MB Util=0 Share=100 TZone=-6
150923 11:07:12 23602 Admit g26n07 adding path: w /
150923 11:07:12 23602 server.30381:91@g26n07:31094 do_Space: 0MB free; 0% util
150923 11:07:12 23602 Protocol: Primary T2_US_Wisconsin server.30381:91@g26n07:31094 logged in.
=====> Routing for g26n07.hep.wisc.edu: local pub4 prv4 pub6 prv6
=====> Route all4: g26n07.hep.wisc.edu Dest=[::144.92.180.109]:31094
=====> Route all6: g26n07.hep.wisc.edu Dest=[2607:f388:101c:1000::281]:31094
150923 11:07:12 23590 Add server.1166:75@g14n10 redirected; too many subscribers.
150923 11:07:12 23599 Add server.28624:87@g25n01 redirected; too many subscribers.
150923 11:07:12 23594 Add server.5811:83@g12n15 redirected; too many subscribers.
150923 11:07:12 23600 Add server.15645:72@g20n04 redirected; too many subscribers.

@xrootd-dev
Copy link

Hi Tapas,

When you say "exists" do you mean it crashes? That would be the only way
it would exit. So, please look for a core file and then:

gdb cmsd core
where
thread apply all bt

That's the minimum I need. You will likely need to install he debug symbol
RPM so I can get line numbers, which are necessary here.

Andy

On Wed, 23 Sep 2015, Tapas Sarangi wrote:

If it helps:
On the redirector, I see that once the nodecnt goes more than 64, cmsd process exits. Here is a snippet of log on the redirector.

150923 11:07:12 23598 Protocol: Primary T2_US_Wisconsin server.21473:86@g26n29:31094 logged in.
=====> Routing for g26n29.hep.wisc.edu: local pub4 prv4 pub6 prv6
=====> Route all4: g26n29.hep.wisc.edu Dest=[::144.92.180.131]:31094
=====> Route all6: g26n29.hep.wisc.edu Dest=[2607:f388:101c:1000::303]:31094
150923 11:07:12 23602 AddNode srv server.30381:91@g26n07:31094 cluster 1213cmsxrootd.hep.wisc.edu mask=ffffffffffffffff anum=0
150923 11:07:12 23602 Add server.30381:91@g26n07:31094 to cluster anon-s 1213cmsxrootd.hep.wisc.edu slot 63.65 (nodecnt=64 supn=1)
150923 11:07:12 23602 Update Counts Parm1=1 Parm2=0
150923 11:07:12 23602 Admit g26n07 TSpace=1GB NumFS=0 FSpace=0MB MinFR=0 MB Util=0 Share=100 TZone=-6
150923 11:07:12 23602 Admit g26n07 adding path: w /
150923 11:07:12 23602 server.30381:91@g26n07:31094 do_Space: 0MB free; 0% util
150923 11:07:12 23602 Protocol: Primary T2_US_Wisconsin server.30381:91@g26n07:31094 logged in.
=====> Routing for g26n07.hep.wisc.edu: local pub4 prv4 pub6 prv6
=====> Route all4: g26n07.hep.wisc.edu Dest=[::144.92.180.109]:31094
=====> Route all6: g26n07.hep.wisc.edu Dest=[2607:f388:101c:1000::281]:31094
150923 11:07:12 23590 Add server.1166:75@g14n10 redirected; too many subscribers.
150923 11:07:12 23599 Add server.28624:87@g25n01 redirected; too many subscribers.
150923 11:07:12 23594 Add server.5811:83@g12n15 redirected; too many subscribers.
150923 11:07:12 23600 Add server.15645:72@g20n04 redirected; too many subscribers.

Reply to this email directly or view it on GitHub:
#291 (comment)

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1

@tsarangi
Copy link
Author

Hi Andy,

Yes, I meant it crashes when I said it exits. It has created core every time it crashes. I am using one of those cores.

Here is the gdb output that you have asked by running gdb cmsd core.xxxx

Missing separate debuginfo for 
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/ff/4cbaae51a93124ed31c2b1386ce92ff24aebc3
Reading symbols from /usr/lib64/libXrdServer.so.2.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libXrdServer.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXrdServer.so.2.0.0
Reading symbols from /usr/lib64/libXrdUtils.so.2.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libXrdUtils.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXrdUtils.so.2.0.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/bin/cmsd -l /var/log/xrootd/cmsd.log -c /etc/xrootd/xrootd.cfg -k 7 -b -s'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000033e8c093a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.7.x86_64 libgcc-4.4.7-11.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64
(gdb) where
#0  0x00000033e8c093a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x0000000000438349 in Lock (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:149
#2  Lock (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsNode.hh:143
#3  XrdCmsProtocol::Process (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:480
#4  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fcce8002628) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#5  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#6  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#7  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf8001220) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033e88e88fd in clone () from /lib64/libc.so.6
(gdb) 
(gdb) thread apply all bt

Thread 88 (Thread 0x7fccd73f3700 (LWP 26971)):
#0  0x00000033e8c0eb5c in recv () from /lib64/libpthread.so.0
#1  0x00007fcd06067ee6 in recv (this=0x7fccb4001de8, Buff=0x7fcca8007c00 "", Blen=150, timeout=<value optimized out>) at /usr/include/bits/socket2.h:45
#2  XrdLink::RecvAll (this=0x7fccb4001de8, Buff=0x7fcca8007c00 "", Blen=150, timeout=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:653
#3  0x00000000004365c9 in XrdCmsProtocol::Dispatch (this=0x7fcca8001310, cDir=XrdCmsProtocol::isLateral, maxWait=-1, maxTries=2) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:925
#4  0x0000000000438308 in XrdCmsProtocol::Process (this=0x7fcca8001310, lp=0x7fccb4001de8) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:469
#5  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fccb4001de8) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#6  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#7  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#8  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x217beb0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#9  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 87 (Thread 0x7fcd05c86700 (LWP 26905)):
#0  0x00000033e8c0b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fcd060307b6 in XrdSysCondVar::WaitMS (this=0x6484f0, msec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:152
#2  0x00007fcd060665e6 in XrdBuffManager::Reshape (this=0x6483e8) at /usr/src/debug/xrootd/src/Xrd/XrdBuffer.cc:224
#3  0x00007fcd06066999 in XrdReshaper (pp=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdBuffer.cc:54
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2180850) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 86 (Thread 0x7fcd05b85700 (LWP 26906)):
#0  0x00000033e8c0b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fcd060307b6 in XrdSysCondVar::WaitMS (this=0x648328, msec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:152
#2  0x00007fcd0606c51b in XrdScheduler::TimeSched (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:571
#3  0x00007fcd0606c529 in XrdStartTSched (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:79
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x21809b0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 85 (Thread 0x7fcd05539700 (LWP 26911)):
#0  0x00000033e88e8ef3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fcd0606b655 in XrdPollE::Start (this=0x2181d80, syncsem=<value optimized out>, retcode=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPollE.icc:232
#2  0x00007fcd0606a908 in XrdStartPolling (parg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPoll.cc:106
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2181e20) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 84 (Thread 0x7fcd05a84700 (LWP 26907)):
#0  0x00000033e8c0ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fcd06030987 in XrdSysTimer::Wait (mills=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:239
#2  0x00000000004280ff in XrdCmsConfig::DoIt (this=0x64d2a0) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsConfig.cc:592
#3  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#4  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#5  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2181160) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#6  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 83 (Thread 0x7fcd057bf700 (LWP 26909)):
#0  0x00000033e88e8ef3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fcd0606b655 in XrdPollE::Start (this=0x2181420, syncsem=<value optimized out>, retcode=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPollE.icc:232
#2  0x00007fcd0606a908 in XrdStartPolling (parg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPoll.cc:106
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x21814c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 82 (Thread 0x7fcd052ee700 (LWP 26913)):
#0  0x00000033e8833857 in sigwaitinfo () from /lib64/libc.so.6
#1  0x00007fcd06305eb1 in XrdOssAioWait (mySigarg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdOss/XrdOssAio.cc:388
#2  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182640) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
....
....
....
... Similar messages with different objects....
....
Thread 15 (Thread 0x7fcd048e4700 (LWP 26923)):
#0  0x00000033e88df0d3 in poll () from /lib64/libc.so.6
#1  0x00007fcd06067e96 in XrdLink::RecvAll (this=0x7fccf4001328, Buff=0x7fccf40026b0 "", Blen=8, timeout=60000) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:637
#2  0x0000000000436445 in XrdCmsProtocol::Dispatch (this=0x7fccf40020e0, cDir=XrdCmsProtocol::isDown, maxWait=60000, maxTries=2) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:883
#3  0x0000000000438308 in XrdCmsProtocol::Process (this=0x7fccf40020e0, lp=0x7fccf4001328) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:469
#4  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fccf4001328) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#5  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#6  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#7  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf00008c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fcd04ae6700 (LWP 26921)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000417d64 in Wait (setSock=<value optimized out>, newSock=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  XrdCmsAdmin::Relay (setSock=<value optimized out>, newSock=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsAdmin.cc:322
#3  0x000000000041817d in XrdCmsAdminSend (carg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsAdmin.cc:147
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccfc0008c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 13 (Thread 0x7fcd04ce8700 (LWP 26919)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000435192 in Wait () at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  XrdCmsPrepArgs::getRequest () at /usr/src/debug/xrootd/src/XrdCms/XrdCmsPrepArgs.cc:104
#3  0x00000000004351e5 in XrdCmsPrepArgs::Process () at /usr/src/debug/xrootd/src/XrdCms/XrdCmsPrepArgs.cc:131
#4  0x00000000004260d9 in XrdCmsStartPreparing (carg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsConfig.cc:149
#5  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf80008c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#6  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fcd050ec700 (LWP 26915)):
#0  0x00000033e8c0ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fcd060309e7 in XrdSysTimer::Snooze (sec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:180
#2  0x000000000041dd5e in XrdCmsCluster::MonPerf (this=0x648d40) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsCluster.cc:685
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182a90) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fcd051ed700 (LWP 26914)):
#0  0x00000033e8833857 in sigwaitinfo () from /lib64/libc.so.6
#1  0x00007fcd06305eb1 in XrdOssAioWait (mySigarg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdOss/XrdOssAio.cc:388
#2  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x21827a0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#3  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fcd049e5700 (LWP 26922)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000439a48 in Wait (this=0x66a740) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  XrdCmsState::Monitor (this=0x66a740) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsState.cc:118
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf8000b80) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fcd05438700 (LWP 26912)):
#0  0x00000033e8c0ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fcd060309e7 in XrdSysTimer::Snooze (sec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:180
#2  0x000000000041a7ab in XrdCmsCache::TickTock (this=0x648780) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsCache.cc:415
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182220) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fcd04be7700 (LWP 26920)):
#0  0x00000033e88eac2f in accept4 () from /lib64/libc.so.6
#1  0x00007fcd06055c59 in XrdSysFD_Accept (this=0x21821b0, timeout=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysFD.hh:50
#2  XrdNetSocket::Accept (this=0x21821b0, timeout=<value optimized out>) at /usr/src/debug/xrootd/src/XrdNet/XrdNetSocket.cc:117
#3  0x000000000041785d in XrdCmsAdmin::Start (this=<value optimized out>, AdminSock=0x21821b0) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsAdmin.cc:381
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf8000a20) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fcd0567c700 (LWP 26910)):
#0  0x00000033e88e8ef3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fcd0606b655 in XrdPollE::Start (this=0x2181b80, syncsem=<value optimized out>, retcode=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPollE.icc:232
#2  0x00007fcd0606a908 in XrdStartPolling (parg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPoll.cc:106
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2181c20) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fcd05983700 (LWP 26908)):
#0  0x00000033e88df0d3 in poll () from /lib64/libc.so.6
#1  0x00007fcd06067e96 in XrdLink::RecvAll (this=0x7fccf0001de8, Buff=0x7fccf0002750 "", Blen=8, timeout=60000) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:637
---Type <return> to continue, or q <return> to quit---
#2  0x0000000000436445 in XrdCmsProtocol::Dispatch (this=0x7fccf00020e0, cDir=XrdCmsProtocol::isDown, maxWait=60000, maxTries=2) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:883
#3  0x0000000000438308 in XrdCmsProtocol::Process (this=0x7fccf00020e0, lp=0x7fccf0001de8) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:469
#4  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fccf0001de8) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#5  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#6  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#7  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x21812c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fcd05f95700 (LWP 26904)):
#0  0x00000033e9803f94 in clock_nanosleep () from /lib64/librt.so.1
#1  0x00007fcd06030e49 in XrdSysTimer::Wait4Midnight () at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:259
#2  0x00007fcd06037f6c in XrdSysLogger::zHandler (this=0x648178) at /usr/src/debug/xrootd/src/XrdSys/XrdSysLogger.cc:652
#3  0x00007fcd06038269 in XrdSysLoggerRT (carg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysLogger.cc:85
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2188500) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fcd04de9700 (LWP 26918)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000439506 in Wait (this=0x64e320) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  XrdCmsRRQ::TimeOut (this=0x64e320) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsRRQ.cc:409
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182eb0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fcd06004720 (LWP 26903)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000416cb3 in Wait (parg=0x7fffd77262f0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  mainAccept (parg=0x7fffd77262f0) at /usr/src/debug/xrootd/src/Xrd/XrdMain.cc:130
#3  0x0000000000416f69 in main (argc=<value optimized out>, argv=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdMain.cc:211

Thread 2 (Thread 0x7fcd04feb700 (LWP 26916)):
#0  0x00000033e8c0ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fcd060309e7 in XrdSysTimer::Snooze (sec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:180
#2  0x000000000041ce17 in XrdCmsCluster::MonRefs (this=0x648d40) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsCluster.cc:712
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182bf0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fccd45c5700 (LWP 27063)):
#0  0x00000033e8c093a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x0000000000438349 in Lock (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:149
#2  Lock (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsNode.hh:143
#3  XrdCmsProtocol::Process (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:480
#4  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fcce8002628) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#5  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#6  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#7  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf8001220) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033e88e88fd in clone () from /lib64/libc.so.6

@abh3
Copy link
Member

abh3 commented Sep 23, 2015

Hi Tapas,

Yes, this particular problem was fixed in 4.2.3,

0228d04

Andy

On Wed, 23 Sep 2015, Tapas Sarangi wrote:

Hi Andy,

Yes, I meant it crashes when I said it exits. It has created core every time it crashes. I am using one of those cores.

Here is the gdb output that you have asked by running gdb cmsd core.xxxx

Missing separate debuginfo for
Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/ff/4cbaae51a93124ed31c2b1386ce92ff24aebc3
Reading symbols from /usr/lib64/libXrdServer.so.2.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libXrdServer.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXrdServer.so.2.0.0
Reading symbols from /usr/lib64/libXrdUtils.so.2.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libXrdUtils.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libXrdUtils.so.2.0.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/bin/cmsd -l /var/log/xrootd/cmsd.log -c /etc/xrootd/xrootd.cfg -k 7 -b -s'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000033e8c093a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.7.x86_64 libgcc-4.4.7-11.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64
(gdb) where
#0  0x00000033e8c093a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x0000000000438349 in Lock (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:149
#2  Lock (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsNode.hh:143
#3  XrdCmsProtocol::Process (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:480
#4  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fcce8002628) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#5  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#6  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#7  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf8001220) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033e88e88fd in clone () from /lib64/libc.so.6
(gdb)
(gdb) thread apply all bt

Thread 88 (Thread 0x7fccd73f3700 (LWP 26971)):
#0  0x00000033e8c0eb5c in recv () from /lib64/libpthread.so.0
#1  0x00007fcd06067ee6 in recv (this=0x7fccb4001de8, Buff=0x7fcca8007c00 "", Blen=150, timeout=<value optimized out>) at /usr/include/bits/socket2.h:45
#2  XrdLink::RecvAll (this=0x7fccb4001de8, Buff=0x7fcca8007c00 "", Blen=150, timeout=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:653
#3  0x00000000004365c9 in XrdCmsProtocol::Dispatch (this=0x7fcca8001310, cDir=XrdCmsProtocol::isLateral, maxWait=-1, maxTries=2) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:925
#4  0x0000000000438308 in XrdCmsProtocol::Process (this=0x7fcca8001310, lp=0x7fccb4001de8) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:469
#5  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fccb4001de8) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#6  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#7  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#8  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x217beb0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#9  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 87 (Thread 0x7fcd05c86700 (LWP 26905)):
#0  0x00000033e8c0b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fcd060307b6 in XrdSysCondVar::WaitMS (this=0x6484f0, msec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:152
#2  0x00007fcd060665e6 in XrdBuffManager::Reshape (this=0x6483e8) at /usr/src/debug/xrootd/src/Xrd/XrdBuffer.cc:224
#3  0x00007fcd06066999 in XrdReshaper (pp=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdBuffer.cc:54
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2180850) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 86 (Thread 0x7fcd05b85700 (LWP 26906)):
#0  0x00000033e8c0b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fcd060307b6 in XrdSysCondVar::WaitMS (this=0x648328, msec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:152
#2  0x00007fcd0606c51b in XrdScheduler::TimeSched (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:571
#3  0x00007fcd0606c529 in XrdStartTSched (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:79
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x21809b0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 85 (Thread 0x7fcd05539700 (LWP 26911)):
#0  0x00000033e88e8ef3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fcd0606b655 in XrdPollE::Start (this=0x2181d80, syncsem=<value optimized out>, retcode=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPollE.icc:232
#2  0x00007fcd0606a908 in XrdStartPolling (parg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPoll.cc:106
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2181e20) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 84 (Thread 0x7fcd05a84700 (LWP 26907)):
#0  0x00000033e8c0ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fcd06030987 in XrdSysTimer::Wait (mills=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:239
#2  0x00000000004280ff in XrdCmsConfig::DoIt (this=0x64d2a0) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsConfig.cc:592
#3  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#4  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#5  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2181160) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#6  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 83 (Thread 0x7fcd057bf700 (LWP 26909)):
#0  0x00000033e88e8ef3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fcd0606b655 in XrdPollE::Start (this=0x2181420, syncsem=<value optimized out>, retcode=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPollE.icc:232
#2  0x00007fcd0606a908 in XrdStartPolling (parg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPoll.cc:106
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x21814c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 82 (Thread 0x7fcd052ee700 (LWP 26913)):
#0  0x00000033e8833857 in sigwaitinfo () from /lib64/libc.so.6
#1  0x00007fcd06305eb1 in XrdOssAioWait (mySigarg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdOss/XrdOssAio.cc:388
#2  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182640) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
....
....
....
... Similar messages with different objects....
....
Thread 15 (Thread 0x7fcd048e4700 (LWP 26923)):
#0  0x00000033e88df0d3 in poll () from /lib64/libc.so.6
#1  0x00007fcd06067e96 in XrdLink::RecvAll (this=0x7fccf4001328, Buff=0x7fccf40026b0 "", Blen=8, timeout=60000) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:637
#2  0x0000000000436445 in XrdCmsProtocol::Dispatch (this=0x7fccf40020e0, cDir=XrdCmsProtocol::isDown, maxWait=60000, maxTries=2) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:883
#3  0x0000000000438308 in XrdCmsProtocol::Process (this=0x7fccf40020e0, lp=0x7fccf4001328) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:469
#4  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fccf4001328) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#5  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#6  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#7  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf00008c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fcd04ae6700 (LWP 26921)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000417d64 in Wait (setSock=<value optimized out>, newSock=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  XrdCmsAdmin::Relay (setSock=<value optimized out>, newSock=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsAdmin.cc:322
#3  0x000000000041817d in XrdCmsAdminSend (carg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsAdmin.cc:147
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccfc0008c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 13 (Thread 0x7fcd04ce8700 (LWP 26919)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000435192 in Wait () at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  XrdCmsPrepArgs::getRequest () at /usr/src/debug/xrootd/src/XrdCms/XrdCmsPrepArgs.cc:104
#3  0x00000000004351e5 in XrdCmsPrepArgs::Process () at /usr/src/debug/xrootd/src/XrdCms/XrdCmsPrepArgs.cc:131
#4  0x00000000004260d9 in XrdCmsStartPreparing (carg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsConfig.cc:149
#5  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf80008c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#6  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#7  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fcd050ec700 (LWP 26915)):
#0  0x00000033e8c0ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fcd060309e7 in XrdSysTimer::Snooze (sec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:180
#2  0x000000000041dd5e in XrdCmsCluster::MonPerf (this=0x648d40) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsCluster.cc:685
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182a90) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fcd051ed700 (LWP 26914)):
#0  0x00000033e8833857 in sigwaitinfo () from /lib64/libc.so.6
#1  0x00007fcd06305eb1 in XrdOssAioWait (mySigarg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdOss/XrdOssAio.cc:388
#2  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x21827a0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#3  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fcd049e5700 (LWP 26922)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000439a48 in Wait (this=0x66a740) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  XrdCmsState::Monitor (this=0x66a740) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsState.cc:118
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf8000b80) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fcd05438700 (LWP 26912)):
#0  0x00000033e8c0ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fcd060309e7 in XrdSysTimer::Snooze (sec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:180
#2  0x000000000041a7ab in XrdCmsCache::TickTock (this=0x648780) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsCache.cc:415
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182220) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fcd04be7700 (LWP 26920)):
#0  0x00000033e88eac2f in accept4 () from /lib64/libc.so.6
#1  0x00007fcd06055c59 in XrdSysFD_Accept (this=0x21821b0, timeout=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysFD.hh:50
#2  XrdNetSocket::Accept (this=0x21821b0, timeout=<value optimized out>) at /usr/src/debug/xrootd/src/XrdNet/XrdNetSocket.cc:117
#3  0x000000000041785d in XrdCmsAdmin::Start (this=<value optimized out>, AdminSock=0x21821b0) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsAdmin.cc:381
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf8000a20) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fcd0567c700 (LWP 26910)):
#0  0x00000033e88e8ef3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fcd0606b655 in XrdPollE::Start (this=0x2181b80, syncsem=<value optimized out>, retcode=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPollE.icc:232
#2  0x00007fcd0606a908 in XrdStartPolling (parg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdPoll.cc:106
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2181c20) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fcd05983700 (LWP 26908)):
#0  0x00000033e88df0d3 in poll () from /lib64/libc.so.6
#1  0x00007fcd06067e96 in XrdLink::RecvAll (this=0x7fccf0001de8, Buff=0x7fccf0002750 "", Blen=8, timeout=60000) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:637
---Type <return> to continue, or q <return> to quit---
#2  0x0000000000436445 in XrdCmsProtocol::Dispatch (this=0x7fccf00020e0, cDir=XrdCmsProtocol::isDown, maxWait=60000, maxTries=2) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:883
#3  0x0000000000438308 in XrdCmsProtocol::Process (this=0x7fccf00020e0, lp=0x7fccf0001de8) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:469
#4  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fccf0001de8) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#5  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#6  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#7  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x21812c0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fcd05f95700 (LWP 26904)):
#0  0x00000033e9803f94 in clock_nanosleep () from /lib64/librt.so.1
#1  0x00007fcd06030e49 in XrdSysTimer::Wait4Midnight () at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:259
#2  0x00007fcd06037f6c in XrdSysLogger::zHandler (this=0x648178) at /usr/src/debug/xrootd/src/XrdSys/XrdSysLogger.cc:652
#3  0x00007fcd06038269 in XrdSysLoggerRT (carg=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysLogger.cc:85
#4  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2188500) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#5  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fcd04de9700 (LWP 26918)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000439506 in Wait (this=0x64e320) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  XrdCmsRRQ::TimeOut (this=0x64e320) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsRRQ.cc:409
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182eb0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fcd06004720 (LWP 26903)):
#0  0x00000033e8c0d930 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000416cb3 in Wait (parg=0x7fffd77262f0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:345
#2  mainAccept (parg=0x7fffd77262f0) at /usr/src/debug/xrootd/src/Xrd/XrdMain.cc:130
#3  0x0000000000416f69 in main (argc=<value optimized out>, argv=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdMain.cc:211

Thread 2 (Thread 0x7fcd04feb700 (LWP 26916)):
#0  0x00000033e8c0ef3d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fcd060309e7 in XrdSysTimer::Snooze (sec=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysTimer.cc:180
#2  0x000000000041ce17 in XrdCmsCluster::MonRefs (this=0x648d40) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsCluster.cc:712
#3  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x2182bf0) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#4  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fccd45c5700 (LWP 27063)):
#0  0x00000033e8c093a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x0000000000438349 in Lock (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.hh:149
#2  Lock (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsNode.hh:143
#3  XrdCmsProtocol::Process (this=0x7fcd000011a0, lp=<value optimized out>) at /usr/src/debug/xrootd/src/XrdCms/XrdCmsProtocol.cc:480
#4  0x00007fcd0606a0c9 in XrdLink::DoIt (this=0x7fcce8002628) at /usr/src/debug/xrootd/src/Xrd/XrdLink.cc:397
#5  0x00007fcd0606d635 in XrdScheduler::Run (this=0x648238) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:333
#6  0x00007fcd0606d829 in XrdStartWorking (carg=<value optimized out>) at /usr/src/debug/xrootd/src/Xrd/XrdScheduler.cc:85
#7  0x00007fcd060308ef in XrdSysThread_Xeq (myargs=0x7fccf8001220) at /usr/src/debug/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8  0x00000033e8c079d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000033e88e88fd in clone () from /lib64/libc.so.6

Reply to this email directly or view it on GitHub:
#291 (comment)

@zvada
Copy link

zvada commented Sep 25, 2015

@tsarangi , you can fetch 4.2.3 from osg-testing repo to upgrade directly your 4.2.2. Don't be afraid upgrade from there, it more-less freeze before it gets to osg production repo. Or if you prefer take rpms directly from official resource: http://xrootd.cern.ch/sw/repos/stable/slc/6/x86_64/

@tsarangi
Copy link
Author

Hi all,

I have tested 4.2.3 on our cluster and this problem is solved. Ticket can be closed.

-Tapas

On Sep 25, 2015, at 11:19 AM, Marian Zvada notifications@github.com wrote:

@tsarangi https://github.com/tsarangi , you can fetch 4.2.3 from osg-testing repo to upgrade directly your 4.2.2. Don't be afraid upgrade from there, it more-less freeze before it gets to osg production repo. Or if you prefer take rpms directly from official resource: http://xrootd.cern.ch/sw/repos/stable/slc/6/x86_64/ http://xrootd.cern.ch/sw/repos/stable/slc/6/x86_64/
—
Reply to this email directly or view it on GitHub #291 (comment).

@abh3
Copy link
Member

abh3 commented Oct 11, 2015

Problem fixed.

@abh3 abh3 closed this as completed Oct 11, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tsarangi @abh3 @xrootd-dev @zvada