TCP packets are Malformed

[varghesepaul]

Hi Team,

Recently we upgraded our kamailo server to 4.2.1 and it was running successfuly with out any issue. We are seeing some issue with the TCP calls which are relayed from the kamilio.

We have set of physical servers and the calls are distributed by checking the dispacther module. Sometimes we are seeing the TCP packets are malformed from kamilio to one server. We can see the UDP calls are working fine with the same server at the same time TCP packets are malformed. That means there is no issue with the network connectivity between the kamailio server and our media server. Once we restart the kamilio we are not seeing this issue and the TCP calls will start working with kamilio and media server.

Can you help us to answer these questions.

1. Any idea about this TCP packet malformed error ?. We were not able to reproduce this TCP packet issue and we are seeing this error in our production environment.
2. Sometimes we have some errors for TCP max conn (ERROR) : 2048 (the default). We are plannig to increase the TCP connection to 4096 (tcp_max_connections=4096). Whether it will create any issue if we are increasing to 4096 or it will affect kamailio performance ?.

Thanks for looking in to this.

[oej]

Question number two is better handled on the sr-users mailing list than in the bug tracker.

Can you give us more details on #1 - a TCP packet can be "malformed" in many ways. How did you notice this - kamailio logs, packet dumps?

[varghesepaul]

Hi oej,

Thanks for looking into. As per our observation we are not seeing any PACKET in our media server from kamailio server and we thought that packets might be malformed.

But after checking the PCAP in kamailio we are not seeing the packet is malformed but packets are not reaching in the server side.

Kamailo Log:

Example one:

Kamailio Log:

 INFO: <script>: CALL 1183440664-5061-3@BJC.BGI.B.HD new branch at sip:3519709109@ms3-ch-rk.plivo.com:5060 (DIspatching the call to server)
 INFO: <script>: <<< 1183440664-5061-3@BJC.BGI.B.HD Current Gateway failed, trying next gateway (didn't get any response for 10 seconds)

PCAP: https://dl.dropboxusercontent.com/u/263974527/SIP-1183440664-5061-3%40BJC.BGI.B.HD-0.pcap

We are not sure what is the problem here ? Thanks again for looking into this.

Once we restarted kamilo we can see the successful TCP calls b/w kamilio server and our media server.

[miconda]

It looks like kamailio was not able to connect to your media server. It could be the limit of active connections, try to increase it if you have a lot of tcp connections.

Look in the code at doc/tcp_tunning.txt for hints on heavy use of tcp.

[varghesepaul]

Thanks Daniel. We will increase the tcp connections to 4096 and will update you the result.

[oej]

Any updates? Is this still an issue or can we close the bug report?

[varghesepaul]

Hi Oej,

Yes the issue is still occurring. Today we saw the TCP calls to some set of Media servers is failing.

From the PCAP we can see the INVITE packet generated from kamailio is not correct and we are not seeing this error after restarting kamailio.

PCAP: https://www.dropbox.com/s/m3nwcdfk8774al8/KAMILIO-TCP-FRAGEMENTED.pcap?dl=0

Could you please advise what might be the issue here ?

[miconda]

How did you get the pcap file? For example ngrep is usually truncating big packets.

[varghesepaul]

We captured using ngrep only. Yeah that can be one reason for the packet truncation.

But when we restarted the kamailio the issue is resolved. Is there anything else we can monitor in our end for debugging this issue in detail ?

[miconda]

Do you get any parsing errors in at the destination? That would be the an indication there is a transmission error.

If you want to see what kamailio is sending out, add onsend_route and print $snd(buf) -- be sure you have the pv buffer big enough (see core cookbook for the global parameter which controls this value).

[varghesepaul]

In the destination i can see the PACKETS reached in the machine and Freeswitch (destination sip ) not able to detect those SIP packets.

[miconda]

You can try to use 'tcpdump -s 0 ...' to see what comes there. Beware that tcp is a stream protocol and network sniffing tools can see parts of the same messages coming as separate packet on the wire, which can be a matter of how the kernel flushes the outbuffer.

Also, don't forget about the option with onsend_route to see what kamailio is writting to the socket.

[varghesepaul]

Thanks Daniel for the inputs.

[miconda]

No followup in a while, closing it as not looking as potential problem in the code. Re-open and add new details if still an issue.