[XrdHttp] 4.10.0 crash in XrdHttpProtocol::Process #1106
Comments
osschar
changed the title
|
Hi @osschar - Anything special about this client? Are they hitting the host with a large level of concurrency (meaning we should look at threading issues) or malformed requests (meaning we should look at the protocol compliance)? It's not clear from your original message - is a crash triggered every time that the scan is run (i.e., does the scan run daily and xrootd crash weekly?)? Brian |
|
Hi Brian, I'm not sure how frequently the scan runs, I will check. The crashes occur about weekly though. This is a security scan, so they check for write-accessible standard directories and all possible buffer overflow etc hacks in the wild ... I suspect we are caught by one of those. We need to check for protocol compliance, the buffer they've sent is clearly garbage. Matevz |
|
I checked on an apache server ... the scan is done weekly. So this is consistent with out crash frequency. I can dump the queries to a file, if it's of any help. There seems to be 23k requests per scan :) |
|
Are we so lucky as to be able to determine precisely which request is causing a failure? Of course, that's going to be the easiest way to narrow this down. |
|
Well, I'd hope so ... that's why I attached my gdb acrobatics to the first post and asked what to look for and if myBuff->buff is indeed the thing that killed it :) |
|
Hi, a nice one apparently. |
|
You already committed the fix, right? I'll try just sneaking in libXrdHttp :) Otherwise I can always apply the patch on top of 4.11-stable and make a rpm. |
|
it's running on ucsd servers ... i'll report if there are new crashes. |
|
Hi Matevz,
I suspect that the hammering is that QUALYS sends over a line of text that
is not being handled correctly. A core file would eb handy to see what got
put in the buffer and why the return code doesn't correspond. This is
definitely a bug and, yes, likely to be there today.
Andy
…On Mon, 6 Jan 2020, Matev? Tadel wrote:
Hi,
Our campus security is using Qualys to hammer all http servers/ports and this leads to crashes of our http-enabled xrootd servers about weekly. This is with 4.10, but the code around this does not seem to have changed since that release.
There is some digging I did with gdb below, not sure if myBuff remains valid during processing.
Let me know if you need me to do some more digging.
Cheers,
Matevz
```
(gdb) bt
#0 0x00007f137e64f337 in __GI_raise ***@***.***=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f137e650a28 in __GI_abort () at abort.c:90
#2 0x00007f137f87a1b9 in XrdOucString::operator[] ***@***.***=0x7f1280750d98, ***@***.***=1) at /usr/src/debug/xrootd/xrootd/src/XrdOuc/XrdOucString.cc:1031
#3 0x00007f137b618824 in XrdHttpProtocol::Process (this=0x7f1280750cd0, lp=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdHttp/XrdHttpProtocol.cc:677
#4 0x00007f137f8989f9 in XrdLink::DoIt (this=0x7f12d9948808) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:441
#5 0x00007f137f89bdaf in XrdScheduler::Run (this=0x610e58 <XrdMain::Config+440>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:357
#6 0x00007f137f89bef9 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdScheduler.cc:87
#7 0x00007f137f861a27 in XrdSysThread_Xeq (myargs=0x7f12bc63da60) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:86
#8 0x00007f137f415e65 in start_thread (arg=0x7f0e5fefe700) at pthread_create.c:307
#9 0x00007f137e71788d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) up
#1 0x00007f137e650a28 in __GI_abort () at abort.c:90
90 raise (SIGABRT);
(gdb)
#2 0x00007f137f87a1b9 in XrdOucString::operator[] ***@***.***=0x7f1280750d98, ***@***.***=1) at /usr/src/debug/xrootd/xrootd/src/XrdOuc/XrdOucString.cc:1031
1031 abort();
(gdb) p *this
$1 = {_vptr.XrdOucString = 0x7f137fac0c10 <vtable for XrdOucString+16>, str = 0x7f1255854c50 "", len = 0, siz = 37, static blksize = -1}
(gdb) up
#3 0x00007f137b618824 in XrdHttpProtocol::Process (this=0x7f1280750cd0, lp=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdHttp/XrdHttpProtocol.cc:677
677 if ((rc == 2) && (tmpline[rc - 1] == '\n')) {
(gdb) p rc
$2 = 2
(gdb) p tmpline
$3 = {_vptr.XrdOucString = 0x7f137fac0c10 <vtable for XrdOucString+16>, str = 0x7f1255854c50 "", len = 0, siz = 37, static blksize = -1}
(gdb) info local
rc = 2
(gdb) up
#4 0x00007f137f8989f9 in XrdLink::DoIt (this=0x7f12d9948808) at /usr/src/debug/xrootd/xrootd/src/Xrd/XrdLink.cc:441
441 do {rc = Protocol->Process(this);} while (!rc && XrdSched->canStick());
(gdb) down
#3 0x00007f137b618824 in XrdHttpProtocol::Process (this=0x7f1280750cd0, lp=<optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdHttp/XrdHttpProtocol.cc:677
677 if ((rc == 2) && (tmpline[rc - 1] == '\n')) {
(gdb) p myBuffStart
$4 = 0x7f12541b9063 ""
(gdb) p myBuffEnd
$5 = 0x7f12541b907e ""
(gdb) p myBuff->buff
$10 = 0x7f12541b9000 "\005"
(gdb) p myBuff
$11 = (XrdBuffer *) 0x7f125405c8b0
(gdb) p *myBuff
$12 = {buff = 0x7f12541b9000 "\005", bsize = 1048576, bindex = 10, next = 0x7f12a4126d60, static pagesz = <optimized out>}
(gdb) x/64s myBuff->buff
0x7f12541b9000: "\005"
0x7f12541b9002: "\v"
0x7f12541b9004: "\020"
0x7f12541b9006: ""
0x7f12541b9007: ""
0x7f12541b9008: "~"
0x7f12541b900a: "."
0x7f12541b900c: "Y\233\211\031\320\026\320\026"
0x7f12541b9015: ""
0x7f12541b9016: ""
0x7f12541b9017: ""
0x7f12541b9018: "\001"
0x7f12541b901a: ""
0x7f12541b901b: ""
0x7f12541b901c: ""
0x7f12541b901d: ""
0x7f12541b901e: "\001"
0x7f12541b9020: "\270J\237M\034}\317\021\206\036"
0x7f12541b902b: " \257n|W"
0x7f12541b9031: ""
0x7f12541b9032: ""
0x7f12541b9033: ""
0x7f12541b9034: "\004]\210\212\353\034\311\021\237\350\b"
0x7f12541b9040: "+\020H`\002"
0x7f12541b9046: ""
0x7f12541b9047: ""
0x7f12541b9048: "\n\002"
0x7f12541b904b: ""
0x7f12541b904c: "\370\355\024"
0x7f12541b9050: "NTLMSSP"
0x7f12541b9058: "\001"
0x7f12541b905a: ""
0x7f12541b905b: ""
0x7f12541b905c: "\a\262"
0x7f12541b905f: ""
0x7f12541b9060: "\n"
0x7f12541b9062: "\n"
0x7f12541b9064: "$"
0x7f12541b9066: ""
0x7f12541b9067: ""
0x7f12541b9068: "\004"
0x7f12541b906a: "\004"
0x7f12541b906c: " "
0x7f12541b906e: ""
0x7f12541b906f: ""
0x7f12541b9070: "QUALYSGUARD123"
0x7f12541b907f: ""
0x7f12541b9080: ""
0x7f12541b9081: ""
0x7f12541b9082: ""
0x7f12541b9083: ""
0x7f12541b9084: ""
0x7f12541b9085: "\b"
0x7f12541b9087: ""
0x7f12541b9088: ""
0x7f12541b9089: ""
0x7f12541b908a: ""
0x7f12541b908b: ""
0x7f12541b908c: "$\305\177\205"
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#1106
|
|
Hi Matevz, Andy,
yes, I have already committed a (likely) fix
Thanks
Fabrizio
…On 2020-01-09 17:50, Matevž Tadel wrote:
You already committed the fix, right? I'll try just sneaking in libXrdHttp :)
Otherwise I can always apply the patch on top of 4.11-stable and make a rpm.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#1106?email_source=notifications&email_token=ABJBUT7YRVYCFLX4NGYQNCTQ45IVTA5CNFSM4KDMLAWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIQ7ODY#issuecomment-572651279>,
or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJBUTYPHCUBF6JCACVK7FTQ45IVTANCNFSM4KDMLAWA>.
|
|
Yess, and I'm running with the fix since about noon yesterday :) |
|
This appears to be fixed. |
Hi,
Our campus security is using Qualys to hammer all http servers/ports and this leads to crashes of our http-enabled xrootd servers about weekly. This is with 4.10, but the code around this does not seem to have changed since that release.
There is some digging I did with gdb below, not sure if myBuff remains valid during processing.
Let me know if you need me to do some more digging.
Cheers,
Matevz
The text was updated successfully, but these errors were encountered: