xrdcp/3.2 complains for a certificate error, while xrdcp/4.0 crashes #136

tommasoboccali · 2014-09-04T13:26:30Z

Ciao, I was trying to debug a problem in a server (gridtutorial8.ba.infn.it), which results in xrdcp/3.2. (as shipped by CMS) in

xrdcp -d 3 root://gridtutorial8.ba.infn.it:8080///store/test/xrootd/CMSSAM//store/mc/SAM/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0013/CE4D66EB-5AAE-E111-96D6-003048D37524.root .

--->

140904 15:16:03 724 Xrd: GoToAnotherServer: Error handshaking to [gridtutorial8.ba.infn.it:8080]
140904 15:16:03 724 Xrd: Open: Error in handling a redirection.
140904 15:16:03 724 Xrd: CheckErrorStatus: Error while being redirected for request Open
Last server error 3010 ('Secgsi: ErrParseBuffer: certificate chain verification failed: Proxy: certificate expired: kXGC_cert: unable to get protocol object.')

ok, probably there is a problem with certificate chain, we will look into it.

But then I tried with a newer xrdcp (as shipped by xrootd4-client-4.0.3-1.slc6.x86_64)

there I get as below. Error is the same (certificate problems, but then a segfault ...)

Is it known?

thanks

tom

[2014-09-04 15:23:51.247747 +0200][Debug  ][XRootDTransport   ] [gridtutorial8.ba.infn.it:8080 #0.0] Sending authentication data
[2014-09-04 15:23:51.252843 +0200][Debug  ][XRootDTransport   ] [gridtutorial8.ba.infn.it:8080 #0.0] Trying to authenticate using gsi
[2014-09-04 15:23:51.270223 +0200][Dump   ][AsyncSock         ] [gridtutorial8.ba.infn.it:8080 #0.0] Wrote a message:  (0x4c0e8a80), 136 bytes
[2014-09-04 15:23:51.295807 +0200][Dump   ][XRootDTransport   ] [msg: 0x4c000950] Expecting 1853 bytes of message body
[2014-09-04 15:23:51.295832 +0200][Dump   ][AsyncSock         ] [gridtutorial8.ba.infn.it:8080 #0.0] Received message header, size: 8
[2014-09-04 15:23:51.295847 +0200][Dump   ][AsyncSock         ] [gridtutorial8.ba.infn.it:8080 #0.0] Received a message of 1861 bytes
[2014-09-04 15:23:51.295858 +0200][Debug  ][XRootDTransport   ] [gridtutorial8.ba.infn.it:8080 #0.0] Sending more authentication data for gsi
[2014-09-04 15:23:51.297965 +0200][Dump   ][AsyncSock         ] [gridtutorial8.ba.infn.it:8080 #0.0] Wrote a message:  (0x4c0f49b0), 7853 bytes
[2014-09-04 15:23:51.341891 +0200][Dump   ][XRootDTransport   ] [msg: 0x4c0f49b0] Expecting 105 bytes of message body
[2014-09-04 15:23:51.341923 +0200][Dump   ][AsyncSock         ] [gridtutorial8.ba.infn.it:8080 #0.0] Received message header, size: 8
[2014-09-04 15:23:51.341976 +0200][Dump   ][AsyncSock         ] [gridtutorial8.ba.infn.it:8080 #0.0] Received a message of 113 bytes
[2014-09-04 15:23:51.341991 +0200][Error  ][XRootDTransport   ] [gridtutorial8.ba.infn.it:8080 #0.0] Authentication with gsi failed: Secgsi: ErrParseBuffer: certificate chain verification failed: Proxy: certificate expired: kXGC_cert
*** glibc detected *** xrdcp: free(): invalid next size (fast): 0x00007fc54c0f49f0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76166)[0x7fc554c65166]
/lib64/libc.so.6(+0x78ca3)[0x7fc554c67ca3]
/usr/lib64/libXrdCl.so.2(_ZN5XrdCl15XRootDTransport16DoAuthenticationEPNS_13HandShakeDataEPNS_17XRootDChannelInfoE+0x590)[0x7fc5556cf2e0]
/usr/lib64/libXrdCl.so.2(_ZN5XrdCl15XRootDTransport13HandShakeMainEPNS_13HandShakeDataERNS_9AnyObjectE+0x176)[0x7fc5556cf706]
/usr/lib64/libXrdCl.so.2(_ZN5XrdCl15XRootDTransport9HandShakeEPNS_13HandShakeDataERNS_9AnyObjectE+0x113)[0x7fc5556cf8f3]
/usr/lib64/libXrdCl.so.2(_ZN5XrdCl18AsyncSocketHandler22OnReadWhileHandshakingEv+0x55)[0x7fc5557076c5]
/usr/lib64/libXrdCl.so.2(_ZN5XrdCl18AsyncSocketHandler5EventEhPNS_6SocketE+0x138)[0x7fc555707918]
/usr/lib64/libXrdCl.so.2(+0x3b5b3)[0x7fc5556c35b3]
/usr/lib64/libXrdUtils.so.2(_ZN6XrdSys8IOEvents6Poller6CbkXeqEPNS0_7ChannelEiiPKc+0x5f8)[0x7fc55522e308]
/usr/lib64/libXrdUtils.so.2(_ZN6XrdSys8IOEvents5PollE8DispatchEPNS0_7ChannelEj+0x52)[0x7fc55522e552]
/usr/lib64/libXrdUtils.so.2(_ZN6XrdSys8IOEvents5PollE5BeginEP15XrdSysSemaphoreRiPPKc+0x132)[0x7fc55522e7c2]
/usr/lib64/libXrdUtils.so.2(_ZN6XrdSys8IOEvents9BootStrap5StartEPv+0x3d)[0x7fc55522cf9d]
/usr/lib64/libXrdUtils.so.2(XrdSysThread_Xeq+0x3f)[0x7fc55522addf]
/lib64/libpthread.so.0(+0x79d1)[0x7fc5547d49d1]
/lib64/libc.so.6(clone+0x6d)[0x7fc554cd7b5d]
======= Memory map: ========
00400000-0040c000 r-xp 00000000 fd:01 290117                             /usr/bin/xrdcp
0060c000-0060d000 rw-p 0000c000 fd:01 290117                             /usr/bin/xrdcp
01d92000-01db3000 rw-p 00000000 00:00 0                                  [heap]
31bc800000-31bc803000 r-xp 00000000 fd:01 135019                         /lib64/libcom_err.so.2.1
31bc803000-31bca02000 ---p 00003000 fd:01 135019                         /lib64/libcom_err.so.2.1
31bca02000-31bca03000 r--p 00002000 fd:01 135019                         /lib64/libcom_err.so.2.1
31bca03000-31bca04000 rw-p 00003000 fd:01 135019                         /lib64/libcom_err.so.2.1
31bcc00000-31bccdb000 r-xp 00000000 fd:01 139537                         /lib64/libkrb5.so.3.3
31bccdb000-31bceda000 ---p 000db000 fd:01 139537                         /lib64/libkrb5.so.3.3
31bceda000-31bcee4000 r--p 000da000 fd:01 139537                         /lib64/libkrb5.so.3.3
31bcee4000-31bcee6000 rw-p 000e4000 fd:01 139537                         /lib64/libkrb5.so.3.3
31bec00000-31bec41000 r-xp 00000000 fd:01 139541                         /lib64/libgssapi_krb5.so.2.2
31bec41000-31bee41000 ---p 00041000 fd:01 139541                         /lib64/libgssapi_krb5.so.2.2
31bee41000-31bee42000 r--p 00041000 fd:01 139541                         /lib64/libgssapi_krb5.so.2.2
31bee42000-31bee44000 rw-p 00042000 fd:01 139541                         /lib64/libgssapi_krb5.so.2.2
3289800000-3289820000 r-xp 00000000 fd:01 132591                         /lib64/ld-2.12.so
3289a1f000-3289a20000 r--p 0001f000 fd:01 132591                         /lib64/ld-2.12.so
3289a20000-3289a21000 rw-p 00020000 fd:01 132591                         /lib64/ld-2.12.so
3289a21000-3289a22000 rw-p 00000000 00:00 0 
3825800000-3825815000 r-xp 00000000 fd:01 140013                         /lib64/libz.so.1.2.3
3825815000-3825a14000 ---p 00015000 fd:01 140013                         /lib64/libz.so.1.2.3
3825a14000-3825a15000 r--p 00014000 fd:01 140013                         /lib64/libz.so.1.2.3
3825a15000-3825a16000 rw-p 00015000 fd:01 140013                         /lib64/libz.so.1.2.3
3826000000-382601d000 r-xp 00000000 fd:01 140014                         /lib64/libselinux.so.1
382601d000-382621c000 ---p 0001d000 fd:01 140014                         /lib64/libselinux.so.1
382621c000-382621d000 r--p 0001c000 fd:01 140014                         /lib64/libselinux.so.1
382621d000-382621e000 rw-p 0001d000 fd:01 140014                         /lib64/libselinux.so.1
382621e000-382621f000 rw-p 00000000 00:00 0 
3827000000-3827016000 r-xp 00000000 fd:01 135473                         /lib64/libgcc_s-4.4.7-20120601.so.1
3827016000-3827215000 ---p 00016000 fd:01 135473                         /lib64/libgcc_s-4.4.7-20120601.so.1
3827215000-3827216000 rw-p 00015000 fd:01 135473                         /lib64/libgcc_s-4.4.7-20120601.so.1
3828000000-38280e8000 r-xp 00000000 fd:01 285160                         /usr/lib64/libstdc++.so.6.0.13
38280e8000-38282e8000 ---p 000e8000 fd:01 285160                         /usr/lib64/libstdc++.so.6.0.13
38282e8000-38282ef000 r--p 000e8000 fd:01 285160                         /usr/lib64/libstdc++.so.6.0.13
38282ef000-38282f1000 rw-p 000ef000 fd:01 285160                         /usr/lib64/libstdc++.so.6.0.13
38282f1000-3828306000 rw-p 00000000 00:00 0 
3828800000-3828829000 r-xp 00000000 fd:01 140119                         /lib64/libk5crypto.so.3.1
3828829000-3828a29000 ---p 00029000 fd:01 140119                         /lib64/libk5crypto.so.3.1
3828a29000-3828a2a000 r--p 00029000 fd:01 140119                         /lib64/libk5crypto.so.3.1
3828a2a000-3828a2b000 rw-p 0002a000 fd:01 140119                         /lib64/libk5crypto.so.3.1
3828a2b000-3828a2c000 rw-p 00000000 00:00 0 
3828c00000-3828c02000 r-xp 00000000 fd:01 131185                         /lib64/libkeyutils.so.1.3
3828c02000-3828e01000 ---p 00002000 fd:01 131185                         /lib64/libkeyutils.so.1.3
3828e01000-3828e02000 r--p 00001000 fd:01 131185                         /lib64/libkeyutils.so.1.3
3828e02000-3828e03000 rw-p 00002000 fd:01 131185                         /lib64/libkeyutils.so.1.3
3829400000-382940a000 r-xp 00000000 fd:01 140116                         /lib64/libkrb5support.so.0.1
382940a000-3829609000 ---p 0000a000 fd:01 140116                         /lib64/libkrb5support.so.0.1Aborted (core dumped)

The text was updated successfully, but these errors were encountered:

ljanyst · 2014-09-04T13:31:31Z

Nope, it's not known. I will have a look.

ljanyst · 2014-09-04T13:32:00Z

Any chance you can reproduce with debug symbols installed?

ljanyst · 2014-09-04T13:42:14Z

Or a valgrind trace (also with debug symbols)?

tommasoboccali · 2014-09-04T13:53:19Z

problematic, since this is production machine where I cannot install stuff (and valgrind is not there...)

let me see if I can find a more "friendly" machine with the same version installed ...

tom

tommasoboccali · 2014-09-04T13:57:40Z

in the end, since I do not have a similar machine I can tweak, I asked brian if he can reproduce it + provide debug. stay tuned, and sorry if I am not of much help here ...

tom

ljanyst · 2014-09-04T13:59:26Z

No worries, thanks for spotting and reporting!

bbockelm · 2014-09-04T14:16:44Z

Hi,

Here's what I got from valgrind:

==8771== Thread 2:
==8771== Invalid write of size 1
==8771==    at 0x4C6F29A: XrdCl::XRootDTransport::DoAuthentication(XrdCl::HandShakeData*, XrdCl::XRootDChannelInfo*) (XrdClXRootDTransport.cc:1466)
==8771==    by 0x4C6F705: XrdCl::XRootDTransport::HandShakeMain(XrdCl::HandShakeData*, XrdCl::AnyObject&) (XrdClXRootDTransport.cc:363)
==8771==    by 0x4C6F8F2: XrdCl::XRootDTransport::HandShake(XrdCl::HandShakeData*, XrdCl::AnyObject&) (XrdClXRootDTransport.cc:272)
==8771==    by 0x4CA76C4: XrdCl::AsyncSocketHandler::OnReadWhileHandshaking() (XrdClAsyncSocketHandler.cc:576)
==8771==    by 0x4CA7917: XrdCl::AsyncSocketHandler::Event(unsigned char, XrdCl::Socket*) (XrdClAsyncSocketHandler.cc:183)
==8771==    by 0x4C635B2: (anonymous namespace)::SocketCallBack::Event(XrdSys::IOEvents::Channel*, void*, int) (XrdClPollerBuiltIn.cc:76)
==8771==    by 0x5100307: XrdSys::IOEvents::Poller::CbkXeq(XrdSys::IOEvents::Channel*, int, int, char const*) (XrdSysIOEvents.cc:686)
==8771==    by 0x5100551: XrdSys::IOEvents::PollE::Dispatch(XrdSys::IOEvents::Channel*, unsigned int) (XrdSysIOEventsPollE.icc:262)
==8771==    by 0x51007C1: XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) (XrdSysIOEventsPollE.icc:219)
==8771==    by 0x50FEF9C: XrdSys::IOEvents::BootStrap::Start(void*) (XrdSysIOEvents.cc:131)
==8771==    by 0x50FCDDE: XrdSysThread_Xeq (XrdSysPthread.cc:86)
==8771==    by 0x30D60079D0: start_thread (in /lib64/libpthread-2.12.so)
==8771==  Address 0x56aeee5 is not stack'd, malloc'd or (recently) free'd
==8771== 
==8771== Invalid write of size 1
==8771==    at 0x4A08DAF: memcpy (mc_replace_strmem.c:882)
==8771==    by 0x4C6F2AD: XrdCl::XRootDTransport::DoAuthentication(XrdCl::HandShakeData*, XrdCl::XRootDChannelInfo*) (XrdClXRootDTransport.cc:1467)
==8771==    by 0x4C6F705: XrdCl::XRootDTransport::HandShakeMain(XrdCl::HandShakeData*, XrdCl::AnyObject&) (XrdClXRootDTransport.cc:363)
==8771==    by 0x4C6F8F2: XrdCl::XRootDTransport::HandShake(XrdCl::HandShakeData*, XrdCl::AnyObject&) (XrdClXRootDTransport.cc:272)
==8771==    by 0x4CA76C4: XrdCl::AsyncSocketHandler::OnReadWhileHandshaking() (XrdClAsyncSocketHandler.cc:576)
==8771==    by 0x4CA7917: XrdCl::AsyncSocketHandler::Event(unsigned char, XrdCl::Socket*) (XrdClAsyncSocketHandler.cc:183)
==8771==    by 0x4C635B2: (anonymous namespace)::SocketCallBack::Event(XrdSys::IOEvents::Channel*, void*, int) (XrdClPollerBuiltIn.cc:76)
==8771==    by 0x5100307: XrdSys::IOEvents::Poller::CbkXeq(XrdSys::IOEvents::Channel*, int, int, char const*) (XrdSysIOEvents.cc:686)
==8771==    by 0x5100551: XrdSys::IOEvents::PollE::Dispatch(XrdSys::IOEvents::Channel*, unsigned int) (XrdSysIOEventsPollE.icc:262)
==8771==    by 0x51007C1: XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) (XrdSysIOEventsPollE.icc:219)
==8771==    by 0x50FEF9C: XrdSys::IOEvents::BootStrap::Start(void*) (XrdSysIOEvents.cc:131)
==8771==    by 0x50FCDDE: XrdSysThread_Xeq (XrdSysPthread.cc:86)
==8771==  Address 0x56aeee4 is not stack'd, malloc'd or (recently) free'd
==8771==

This happened with 4.0.3 (fresh from xrootd.org) and did not happen with the 4.0.0 pre-release that I used to have on my development host. I think (but am not sure) it happens before authorization is finished (meaning Lukasz could probably reproduce this too).

ljanyst · 2014-09-04T14:19:37Z

Merci!

ljanyst · 2014-09-04T14:25:27Z

It's gets a malformed kXR_error response and my retarded code does not seem to compensate. Will fix. Thanks for reporting.

abh3 · 2014-09-04T18:12:34Z

Hi Lukasz,

Please document the malformed response so we can fix that in the server.

Andy

On Thu, 4 Sep 2014, Lukasz Janyst wrote:

It's gets a malformed kXR_error response and my retarded code does not seem to compensate. Will fix. Thanks for reporting.

Reply to this email directly or view it on GitHub:
#136 (comment)

ljanyst · 2014-09-22T07:18:34Z

The response is OK, I just made a silly mistake when allocating a buffer for the error message.

…#136.

ljanyst added the bug label Sep 4, 2014

ljanyst added the fixed label Sep 22, 2014

ljanyst closed this as completed in 2e7369d Sep 22, 2014

ljanyst added a commit that referenced this issue Oct 20, 2014

XrdCl: Properly allocate a buffer for the error message. Fixes #136.

ddf8fe3

sponce pushed a commit to sponce/xrootd that referenced this issue Jan 19, 2015

XrdCl: Properly allocate a buffer for the error message. Fixes xrootd…

b95242c

…#136.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

xrdcp/3.2 complains for a certificate error, while xrdcp/4.0 crashes #136

xrdcp/3.2 complains for a certificate error, while xrdcp/4.0 crashes #136

tommasoboccali commented Sep 4, 2014

ljanyst commented Sep 4, 2014

ljanyst commented Sep 4, 2014

ljanyst commented Sep 4, 2014

tommasoboccali commented Sep 4, 2014

tommasoboccali commented Sep 4, 2014

ljanyst commented Sep 4, 2014

bbockelm commented Sep 4, 2014

ljanyst commented Sep 4, 2014

ljanyst commented Sep 4, 2014

abh3 commented Sep 4, 2014

ljanyst commented Sep 22, 2014

xrdcp/3.2 complains for a certificate error, while xrdcp/4.0 crashes #136

xrdcp/3.2 complains for a certificate error, while xrdcp/4.0 crashes #136

Comments

tommasoboccali commented Sep 4, 2014

ljanyst commented Sep 4, 2014

ljanyst commented Sep 4, 2014

ljanyst commented Sep 4, 2014

tommasoboccali commented Sep 4, 2014

tommasoboccali commented Sep 4, 2014

ljanyst commented Sep 4, 2014

bbockelm commented Sep 4, 2014

ljanyst commented Sep 4, 2014

ljanyst commented Sep 4, 2014

abh3 commented Sep 4, 2014

ljanyst commented Sep 22, 2014