b2b_logic can't handle more than 300-400 CPS and has memory corruption

[nikbyte]

Hi team,

I found that if start load tests with common calls, starting from 300-400 CPS b2b_logic starts to have memory corruptions.
It's easy reproduceable in two cases:

1. Enable write-through db writing (db_mode=1) and you'll get corrupted dlg structures on b2bl_db_update. If enable db_mode=2 even with 1 second interval this doesn't reproduce.

2. From time to time call b2b_cback() in b2b_entities module crashes opensips because b2b_cback pointer points to some released memory.

Usually it occurs on BYE or final ACK when there is a large amount of requests and especially when there are retransmits.

[rvlad-patrascu]

Hi @nikbyte ,

Did you test this with the "top_hiding" scenario ? If not, can you also post the script logic that triggers this?

[nikbyte]

It seems, I found a reason. I'll post pool request soon.

[nikbyte]

I'm testing with non-top hiding, but scenario is pretty simple: create server, create client, init b2b.

[nikbyte]

@rvlad-patrascu this helps a lot: #2426
Also I have patch which helps, but I'm not sure it's proper:

diff --git a/modules/b2b_entities/dlg.c b/modules/b2b_entities/dlg.c
index f4f1370e8..d2281a6a5 100644
--- a/modules/b2b_entities/dlg.c
+++ b/modules/b2b_entities/dlg.c
@@ -1070,7 +1070,7 @@ logic_notify:

 				if(dlg->uas_tran && dlg->uas_tran!=T_UNDEFINED)
 				{
-					if(dlg->uas_tran->uas.request)
+					if(method_value != METHOD_BYE && dlg->uas_tran->uas.request)
 					/* there is another transaction for which no reply
 					 * was sent out */
 					{

Without this patch there is dlg->uas_tran exists, but dlg->uas_tran.uas points to released memory.
Maybe you know how to fix this properly.

With these two patches it's not crashing for me.

[bogdan-iancu]

@nick, could you explain a bit your fix here, please ?

[nikbyte]

@bogdan-iancu I'm not sure that this fix is proper, it may be more workaround. But.

Initial condition is if(dlg->uas_tran->uas.request) .
Under load it's crashing because of dlg->uas_tran.uas points to released memory, however dlg->uas_tran structure is still exists. It seems, that BYE transaction is already released when we come to this part of code.

Adding condition method_value != METHOD_BYE resolves this issue, but I'm not sure if this is a proper solution or I'm trying just to hide another issue.

[rvlad-patrascu]

@nikbyte Do you by any chance still have any backtraces for these crashes ?

[nikbyte]

Unfortunately not possible for the moment. :(

[nikbyte]

From another side, you can try sipp with high cps, I think you'll see these crashes.

[rvlad-patrascu]

@nikbyte I could not reproduce these crashes. And since you are saying that the issue is now solved with your own patches and you cannot provide any other info, I think we can close this ticket for now.

Feel free to re-open this when you will be using the official 3.2 code. If the crashes persist at that point, it will confirm that the fix in aba8ae5 is not enough and you should also be able to provide backtraces.