Segv in GSIStack<XrdCryptoX509Crl>::Del(XrdCryptoX509Crl*)

[simonmichal]

If xrootd client runs against a server that enforces both encryption and request signing, it crashes on exit due to static deinitialization fiasco:

$ ./xrdcp -f Makefile roots://slc7-test.cern.ch//tmp
[57.49kB/57.49kB][100%][==================================================][57.49kB/s]  
=================================================================
==31590==ERROR: AddressSanitizer: heap-use-after-free on address 0x6190000185f0 at pc 0x7fe60b292cc5 bp 0x7fff18451570 sp 0x7fff18451560
READ of size 8 at 0x6190000185f0 thread T0
    #0 0x7fe60b292cc4 in XrdOucHash<XrdCryptoX509Crl>::Find(char const*, long*) /home/simonm/xrootd/src/./XrdOuc/XrdOucHash.icc:173
    #1 0x7fe60b26d3d0 in GSIStack<XrdCryptoX509Crl>::Del(XrdCryptoX509Crl*) /home/simonm/xrootd/src/./XrdSecgsi/XrdSecProtocolgsi.hh:265
    #2 0x7fe60b26d3d0 in gsiHSVars::~gsiHSVars() /home/simonm/xrootd/src/./XrdSecgsi/XrdSecProtocolgsi.hh:534
    #3 0x7fe60b26d3d0 in XrdSecProtocolgsi::Delete() /home/simonm/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:1069
    #4 0x7fe616706e83 in XrdCl::XRootDTransport::CleanUpAuthentication(XrdCl::XRootDChannelInfo*) /home/simonm/xrootd/src/XrdCl/XrdClXRootDTransport.cc:2526
    #5 0x7fe6167076a2 in XrdCl::XRootDTransport::CleanUpProtection(XrdCl::XRootDChannelInfo*) /home/simonm/xrootd/src/XrdCl/XrdClXRootDTransport.cc:2549
    #6 0x7fe61670f388 in XrdCl::XRootDTransport::Disconnect(XrdCl::AnyObject&, unsigned short) /home/simonm/xrootd/src/XrdCl/XrdClXRootDTransport.cc:1409
    #7 0x7fe61690dbec in XrdCl::AsyncSocketHandler::Close() /home/simonm/xrootd/src/XrdCl/XrdClAsyncSocketHandler.cc:188
    #8 0x7fe6166d37a0 in XrdCl::Stream::Disconnect(bool) /home/simonm/xrootd/src/XrdCl/XrdClStream.cc:350
    #9 0x7fe6166da782 in XrdCl::Stream::~Stream() /home/simonm/xrootd/src/XrdCl/XrdClStream.cc:139
    #10 0x7fe6166cee2b in XrdCl::Channel::~Channel() /home/simonm/xrootd/src/XrdCl/XrdClChannel.cc:136
    #11 0x7fe6166c098f in XrdCl::PostMaster::Finalize() /home/simonm/xrootd/src/XrdCl/XrdClPostMaster.cc:151
    #12 0x7fe616660a78 in XrdCl::DefaultEnv::Finalize() /home/simonm/xrootd/src/XrdCl/XrdClDefaultEnv.cc:738
    #13 0x7fe61507c059 in __cxa_finalize (/lib64/libc.so.6+0x3a059)
    #14 0x7fe61661e432  (/home/simonm/xrootd/build/src/XrdCl/libXrdCl.so.3+0x2a7432)

0x6190000185f0 is located 1136 bytes inside of 1152-byte region [0x619000018180,0x619000018600)
freed by thread T0 here:
    #0 0x7fe616e39508 in __interceptor_free (/lib64/libasan.so.4+0xde508)
    #1 0x7fe60b290a3f in XrdOucHash<XrdCryptoX509Crl>::~XrdOucHash() /home/simonm/xrootd/src/./XrdOuc/XrdOucHash.hh:186
    #2 0x7fe60b290a3f in GSIStack<XrdCryptoX509Crl>::~GSIStack() /home/simonm/xrootd/src/./XrdSecgsi/XrdSecProtocolgsi.hh:253

previously allocated by thread T1 here:
    #0 0x7fe616e39a88 in __interceptor_calloc (/lib64/libasan.so.4+0xdea88)
    #1 0x7fe60b24b4c0 in XrdOucHash<XrdCryptoX509Crl>::XrdOucHash(int, int, int) /home/simonm/xrootd/src/./XrdOuc/XrdOucHash.icc:52
    #2 0x7fe60b24b4c0 in GSIStack<XrdCryptoX509Crl>::GSIStack() /home/simonm/xrootd/src/./XrdSecgsi/XrdSecProtocolgsi.hh:253
    #3 0x7fe60b24b4c0 in __static_initialization_and_destruction_0 /home/simonm/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:199
    #4 0x7fe60b24b4c0 in _GLOBAL__sub_I_XrdSecProtocolgsi.cc /home/simonm/xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc:5807

Thread T1 created by T0 here:
    #0 0x7fe616d92a7f in pthread_create (/lib64/libasan.so.4+0x37a7f)
    #1 0x7fe615ce252d in XrdSysThread::Run(unsigned long*, void* (*)(void*), void*, int, char const*) /home/simonm/xrootd/src/XrdSys/XrdSysPthread.cc:323

SUMMARY: AddressSanitizer: heap-use-after-free /home/simonm/xrootd/src/./XrdOuc/XrdOucHash.icc:173 in XrdOucHash<XrdCryptoX509Crl>::Find(char const*, long*)
Shadow bytes around the buggy address:
  0x0c327fffb060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c327fffb070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c327fffb080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c327fffb090: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c327fffb0a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c327fffb0b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]fd
  0x0c327fffb0c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c327fffb0d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c327fffb0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c327fffb0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c327fffb100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==31590==ABORTING

[simonmichal]

In more details, XrdCl will finalize all open connections on exit on destruction of PostMaster which is a global static object, in particular in the XRootDTransport::Disconnect it will call XRootDTransport::CleanUpProtection that will trigger deletion of XrdSecProtocol. The XrdSecProtocol, in turn, will destroy gsiHSVars, which will try to

xrootd/src/XrdSecgsi/XrdSecProtocolgsi.hh

Line 534 in a384ecb

XrdSecProtocolgsi::stackCRL.Del(Crl);

The XrdSecProtocolgsi::stackCRL is another static object, and if the compiler decided this one should be finalized first xrootd client will segv on exit.