Opensips 1.8.8 crash in get_ha1 while doing authorization on REGISTER

[neha31oct]

Below are the core fille debug log.. Please help me out.......
We havenot changed anything

(gdb) bt
#0  0x0000003b8a1336ff in __strlen_sse42 () from /lib64/libc.so.6
#1  0x00007f9e73739d42 in get_ha1 (_username=0x7f9e7590d3b8, _domain=0x7fff60c42220, _table=0x7fff60c42210, _ha1=0x7fff60c42240 "\340\064\002\212;",
    res=0x7fff60c42208) at authorize.c:126
#2  0x00007f9e7373a74b in authorize (_m=0x7f9e75900828, _realm=0x7f9e758fd540, _table=0x7f9e758e7f70 "ALB_subscriber", _hftype=HDR_AUTHORIZATION_T) at authorize.c:248
#3  0x00007f9e7373a960 in www_authorize (_m=0x7f9e75900828, _realm=0x7f9e758fd540 "\001", _table=0x7f9e758e7f70 "ALB_subscriber") at authorize.c:291
#4  0x0000000000414fde in do_action (a=0x7f9e758e8008, msg=0x7f9e75900828) at action.c:1581
#5  0x000000000040f0f7 in run_action_list (a=0x7f9e758e8008, msg=0x7f9e75900828) at action.c:158
#6  0x000000000046053f in eval_elem (e=0x7f9e758e80e0, msg=0x7f9e75900828, val=0x0) at route.c:1453
#7  0x00000000004619c0 in eval_expr (e=0x7f9e758e80e0, msg=0x7f9e75900828, val=0x0) at route.c:1798
#8  0x0000000000461aba in eval_expr (e=0x7f9e758e8130, msg=0x7f9e75900828, val=0x0) at route.c:1814
#9  0x0000000000461af4 in eval_expr (e=0x7f9e758e8180, msg=0x7f9e75900828, val=0x0) at route.c:1819
#10 0x00000000004126cb in do_action (a=0x7f9e758e8750, msg=0x7f9e75900828) at action.c:997
#11 0x000000000040f0f7 in run_action_list (a=0x7f9e758e7d68, msg=0x7f9e75900828) at action.c:158
#12 0x0000000000412802 in do_action (a=0x7f9e758e94f0, msg=0x7f9e75900828) at action.c:1014
#13 0x000000000040f0f7 in run_action_list (a=0x7f9e758d9010, msg=0x7f9e75900828) at action.c:158
#14 0x000000000040efc4 in run_actions (a=0x7f9e758d9010, msg=0x7f9e75900828) at action.c:123
#15 0x000000000040f1f5 in run_top_route (a=0x7f9e758d9010, msg=0x7f9e75900828) at action.c:198
#16 0x0000000000451d31 in receive_msg (
    buf=0x7736e0 "REGISTER sip:192.168.20.220 SIP/2.0\r\nVia: SIP/2.0/UDP 192.20.162.46:16072;branch=z9hG4bK-d87543-f5273d72fb5eff23-1--d87543-;rport\r\nMax-Forwards: 69\r\nContact: <sip:4226@192.20.162.46:16072;rinstance="..., len=766, rcv_info=0x7fff60c43b80) at receive.c:169
#17 0x00000000004892fb in udp_rcv_loop () at udp_server.c:422
#18 0x0000000000427627 in main_loop () at main.c:884
#19 0x00000000004299ed in main (argc=7, argv=0x7fff60c43df8) at main.c:1557
(gdb) up
#1  0x00007f9e73739d42 in get_ha1 (_username=0x7f9e7590d3b8, _domain=0x7fff60c42220, _table=0x7fff60c42210, _ha1=0x7fff60c42240 "\340\064\002\212;",
    res=0x7fff60c42208) at authorize.c:126
126     authorize.c: No such file or directory.
        in authorize.c
(gdb) p res->r
res_rows  rows
(gdb) p res->r
res_rows  rows
(gdb) p res->rows
Display all 31878 possibilities? (y or n)
(gdb) p res->rows->values
Display all 31878 possibilities? (y or n)
(gdb) p res->rows->values
$1 = (db_val_t *) 0x7f9e75901210
(gdb) p *res->rows->values
$2 = {type = DB_INT, nul = 0, free = 0, val = {int_val = 959460407, bigint_val = 959460407, double_val = 4.7403642564356027e-315, time_val = 959460407,
    string_val = 0x39303437 <Address 0x39303437 out of bounds>, str_val = {s = 0x39303437 <Address 0x39303437 out of bounds>, len = 0}, blob_val = {
      s = 0x39303437 <Address 0x39303437 out of bounds>, len = 0}, bitmap_val = 959460407}}
(gdb

[neha31oct]

Can someone help me out...........

[ionutrazvanionita]

@neha31oct Can you please tell me what database engine are you using? And also, might be a stupid question, but are you sure your ha1/ha1b column is not INT?

[neha31oct]

I am using mysql database and i am also sure ha1/ha1b column is not INT . i have re-checked it also.

[ionutrazvanionita]

ok i'll try to replicate the bug for myself. please can you tell me what column you are using? ha1 or ha1b?

[neha31oct]

i am using ha1 column
after executing the query for authorization for Subscriber table , then tries to access the result of the query (see below)

1. result.s = (char_)ROW_VALUES(RES_ROWS(_res))[0].val.string_val; (authorize.c:126)
2. result.len = strlen(result.s);

opensips crashes while using strlen function in above point 2 because it gets address out of bound in above string (point1) .

I dont know why it is getting invalid address .i am also not closing database here .

[neha31oct]

opensips is also getting crash in alias_db_module
this one is also crashing while accessing result of query in the same way as above problem

[ionutrazvanionita]

what's strange is the type of that value (type = DB_INT in your log). That's why it crashes. I'll try to replicate the bug and come back with an answer(hope i'll have one)

[ionutrazvanionita]

@neha31oct do you use the load_credentials parameter?

[ionutrazvanionita]

@neha31oct can you try compiling your opensips with memory debugging[0] ?

[0] http://www.opensips.org/Documentation/TroubleShooting-OutOfMem

[neha31oct]

yes i am using load_credentials parameters but with default values

modparam("auth_db", "load_credentials", "")

[neha31oct]

I have also compiled my opensips with memory debbugging as you said above but it is not giving any memory logs in to opensips logs

[neha31oct]

i have again tried for memory debugging and found these logs(see below) . these logs are not of server on which opensips crash .These are of testing server .i cant enable memory debugging on that production server . please tell me how to read these logs.? i am also unable to recreate this crash .

/usr/local/sbin/opensips[17884]: Memory status (pkg):
/usr/local/sbin/opensips[17884]: fm_status (0x7fc8b768f010):
/usr/local/sbin/opensips[17884]: heap size= 8388608
/usr/local/sbin/opensips[17884]: used= 159328, used+overhead=225592, free=8229280
/usr/local/sbin/opensips[17884]: max used (+overhead)= 231968
/usr/local/sbin/opensips[17884]: dumping free list:
/usr/local/sbin/opensips[17884]: hash = 1 fragments no.: 5, unused: 0#012#011#011 bucket size: 8 - 8 (first 8)
/usr/local/sbin/opensips[17884]: hash = 2 fragments no.: 3, unused: 0#012#011#011 bucket size: 16 - 16 (first 16)
/usr/local/sbin/opensips[17884]: hash = 3 fragments no.: 2, unused: 0#012#011#011 bucket size: 24 - 24 (first 24)
/usr/local/sbin/opensips[17884]: hash = 4 fragments no.: 1, unused: 0#012#011#011 bucket size: 32 - 32 (first 32)
/usr/local/sbin/opensips[17884]: hash = 6 fragments no.: 2, unused: 0#012#011#011 bucket size: 48 - 48 (first 48)
/usr/local/sbin/opensips[17884]: hash = 7 fragments no.: 17, unused: 0#012#011#011 bucket size: 56 - 56 (first 56)
/usr/local/sbin/opensips[17884]: hash = 8 fragments no.: 4, unused: 0#012#011#011 bucket size: 64 - 64 (first 64)
/usr/local/sbin/opensips[17884]: hash = 9 fragments no.: 11, unused: 0#012#011#011 bucket size: 72 - 72 (first 72)
/usr/local/sbin/opensips[17884]: hash = 14 fragments no.: 1, unused: 0#012#011#011 bucket size: 112 - 112 (first 112)
/usr/local/sbin/opensips[17884]: hash = 28 fragments no.: 1, unused: 0#012#011#011 bucket size: 224 - 224 (first 224)
/usr/local/sbin/opensips[17884]: hash = 30 fragments no.: 1, unused: 0#012#011#011 bucket size: 240 - 240 (first 240)
/usr/local/sbin/opensips[17884]: hash = 55 fragments no.: 1, unused: 0#012#011#011 bucket size: 440 - 440 (first 440)
/usr/local/sbin/opensips[17884]: hash = 77 fragments no.: 2, unused: 0#012#011#011 bucket size: 616 - 616 (first 616)
/usr/local/sbin/opensips[17884]: hash = 233 fragments no.: 1, unused: 0#012#011#011 bucket size: 1864 - 1864 (first 1864)
/usr/local/sbin/opensips[17884]: hash = 2057 fragments no.: 1, unused: 0#012#011#011 bucket size: 4194304 - 8388608 (first 8156280)
/usr/local/sbin/opensips[17884]: TOTAL: 53 free fragments = 8162656 free bytes
/usr/local/sbin/opensips[17884]: TOTAL: 8156280 large bytes
/usr/local/sbin/opensips[17884]: TOTAL: 24 overhead
/usr/local/sbin/opensips[17884]: -----------------------------

[neha31oct]

I recreated this crash after changing type of PASSWORD column database from CHAR to INT . so its clear that there is no handling in opensips if anyone changed type of this column . but actually the scenario in which opensips crashes there the type of PASSWORD column is CHAR only but i think that while accessing the database through query its type changes from CHAR TO INT as we can see in the above logs. How can we identify that?

[liviuchircu]

Just FYI, there is an ongoing discussion regarding a lot more strict SQL column type checking

[ionutrazvanionita]

I am trying to recreate the bug but i just can't get it. Can you please provide the script? Or at least the modules you are using? Also, are you using some custom tables(modified the ones created by opensips), or functions/modules?

[ionutrazvanionita]

I'm asking because I suspect some kind of memory corruption.

[ionutrazvanionita]

one more thing, thanks to @razvancrainea . In gdb, you should print *((*res)->rows->values) since "res" is a double pointer.

[neha31oct]

we are using load_balancer module of opensips

[neha31oct]

We have not changed anything in authorization module where crash happens . Please specify what other information you need ? please be specific

[ionutrazvanionita]

Not only the auth module. Anything else that is not as in the git version!

[neha31oct]

As you can see in the above dump logs that no any other module (except auth module ) is used in this stack of crash .
We have done some query or database related changes (like added more column in load_balancer table ).
We have not modified any existing code in load_balancer.

[ionutrazvanionita]

Ok. Think I've found the issue. Just to confirm it to me, can you please trace only the mysql queries connections with wireshark and send me a pcap on my email?

[ionutrazvanionita]

@neha31oct any success in getting that pcap? i didn't receive any email

[ionutrazvanionita]

Hi @neha31oct, since there is no progress with this we decided to close this issue. Please feel free to reopen it or open another one if you have any updates.