mTCP as client interoperability with server in syncookie mode

[vincentmli]

Hi,

I have to say mTCP is so far the best user space TCP stack I used on top of DPDK that can generate millions of stateful TCP connections on cheap hardware, I am wondering what is hold mTCP back to spread the word in DPDK community. Thank you for the great work!

Now here is an issue I noticed, and it is only my guess, so I am seeking idea here:

Here is what i am doing two test:

test 1:

a: set server syncookie threshhold to 16 to trigger server start syncookie mode earlier than normal
b: start tcpdump on server to capture packet from epwget
c: use epwget to generate tcp connection

./apps/example/epwget 10.9.3.6 16000 -N 16 -c 3200

the epwget will trigger server in syncookie mode and then I see tons of packet retransmission from both mTCP and server and the whole tcpdump packet size is around ~91MB

test 2:
a: set server syncookie threshhold to 1638400000, basically disables server syncookie
b: start tcpdump on server to capture packet from epwget
c: use epwget to generate tcp connection

./apps/example/epwget 10.9.3.6 16000 -N 16 -c 3200

the server is not in syncookie mode, there is no any tcp packet retransmission between mTCP and server. the whole tcpdump packet size is only around ~1.28MB.

so my question is in test1, why there is so many packet retransmission between mTCP and server when server in syncookie mode? does mTCP has interoperability issue with server in syncookie mode, is this related to mTCP batch mode TCP packet processing?

Let me know if you need to see the sample tcpdump capture and I appreciate any input from you.

Regards,

Vincent

[ajamshed]

Hi Vincent,

Thanks! We will actually be interested in seeing the packet capture. Retransmissions can happen if the cost of performing crypto functions overwhelms the server. The client may then end up retransmitting the SYN requests.

[vincentmli]

here is the capture
test1: http://www.vcn.bc.ca/~vli/syncookie.pcap
test2: http://www.vcn.bc.ca/~vli/nosyncookie.pcap

[vincentmli]

FYI, My server is actually a virtual server with simple irule running on BIGIP 8900 which I think should have good performance

[ajamshed]

Please give us some days before we can provide you feedback. Thanks.

[vincentmli]

no problem, please let me know if you need anything else. appreciated it

[ajamshed]

Vincent... Can you also try the same experiment with varying number of concurrent connections (-c)? I think the retransmissions may be related to this issue. How many CPU cores are you using to run the virtual server?

[vincentmli]

I am running mTCP on 16 cores on 16 nic queues, virtual server on BIGIP runs on 8 cores, I can try -c with different number of cores

[vincentmli]

Oh sorry I mean -c with different number of concurrent connections. My goal is to have million of concurrent connections, is there going to be any limitation with epwget or mTCP stack with over million connection other than memory constraints?

[vincentmli]

Hi Asim

I think I have better trouble shooting data this time for you, it seems related to sequence number validation in mTCP. this time I turned on mTCP debugging and here is the capture and debugging log:

http://www.vcn.bc.ca/~vli/mTCP/syncookie-test.pcap
http://www.vcn.bc.ca/~vli/mTCP/log_3

http://www.vcn.bc.ca/~vli/mTCP has all the other mTCP thread logs too

for example, looking at capture syncookie-test.pcap with filter 'tcp.port == 1033', mTCP keeps retransmiting "GET / HTTP/1.0" after virtual server responded HTTP/1.0 200 OK. from the wireshark, the sequence number of HTTP/1.0 200 OK look fine, but it appears mTCP fail to validate the sequence number, thus keeps retransmitting HTTP get request

search tcp port 1033 in http://www.vcn.bc.ca/~vli/mTCP/log_3, you will find the sequence number validation error, I am trying to line up the log_3 with the capture and look into why mTCP complains about wrong sequence number.

[vincentmli]

so here is the relevant log:

[STREAM: CreateTCPStream: 322] CREATED NEW TCP STREAM 2: 10.9.3.9(1033) -> 10.9.3.249(80) (ISS: 741292118)

[ValidateSequence: 114] No timestamp found. <===this reference capture frame 1240 HTTP/1.0 200 OK, it has no timestamp, I recall BIGIP encode syncookie with timestamp, is this related?

[ProcessTCPPacket:1186] Stream 2: Unexpected sequence: 3753907991, expected: 3753907991
10.9.3.249(80) -> 10.9.3.9(1033) IP_ID=10742 TTL=255 TCP A seq 3753907991 ack 741292222 WDW=4380 len=119 <===== HTTP/1.0 200 OK, the unexpected sequence same as expected, why
[ValidateSequence: 114] No timestamp found.
[ProcessTCPPacket:1186] Stream 2: Unexpected sequence: 3753908070, expected: 3753907991
10.9.3.249(80) -> 10.9.3.9(1033) IP_ID=10744 TTL=255 TCP F A seq 3753908070 ack 741292222 WDW=4483 len=40
[ValidateSequence: 114] No timestamp found.
[ProcessTCPPacket:1186] Stream 3: Unexpected sequence: 3213386535, expected: 3213386535
10.9.3.249(80) -> 10.9.3.9(1037) IP_ID=62769 TTL=255 TCP A seq 3213386535 ack 981741806 WDW=4380 len=119
[ValidateSequence: 114] No timestamp found.

I am going to do a capture without syncookie and compare the debug log next

[vincentmli]

http://www.vcn.bc.ca/~vli/mTCP-nosyncookie/ link include mTCP debug log and capture without syncookie, I can clearly see there is no sequence validation log, for example

'tcp.port == 1199' on http://www.vcn.bc.ca/~vli/mTCP-nosyncookie/nosyncookie-test.pcap
and search 1199 in http://www.vcn.bc.ca/~vli/mTCP-nosyncookie/log_3

this time the HTTP/1.0 200 OK has timestamp option

no no timestamp found log and sequence validation log in log_3.

it seems there is interoperability issue between mTCP and BIGIP TCP stack when syncookie activated.

[ajamshed]

Hi Vincent,

I looked at the logs and the ValidateSequence(...) function definition. mTCP currently does not support those TCP connections that stop sending timestamps mid-stream. That is what's happening in this scenario. The server starts with timestamps (with syn cookies).. but then does not include the timestamps when it sends the HTTP/1.0 200 OK message. I guess we need to implement that patch for this.

[vincentmli]

it would be nice if mTCP can be flexible on the timestamp check. wondering how Linux deal with this middle stream missing timestamp, anyway, thanks for looking into it and hoping you can come up with some patch 👍

[vincentmli]

I looked at Linux and it appears it ACK packet missing timestamp and I used Linux client to test to confirm that Linux ACK the HTTP/1.0 200 OK, no retransmission. in Linux

static int tcp_fast_parse_options(struct sk_buff *skb, struct tcphdr *th,
                                  struct tcp_sock *tp)
{
        if (th->doff == sizeof(struct tcphdr) >> 2) {
                tp->rx_opt.saw_tstamp = 0;
                return 0;
        } else if (tp->rx_opt.tstamp_ok &&
                   th->doff == (sizeof(struct tcphdr)>>2)+(TCPOLEN_TSTAMP_ALIGNED>>2)) {
                if (tcp_parse_aligned_timestamp(tp, th))
                        return 1;
        }
        tcp_parse_options(skb, &tp->rx_opt, 1);
        return 1;
}

so would a patch below for mTCP ok? I know if timestamp missing, it defeat the PAWS purpose

diff --git a/mtcp/src/tcp_in.c b/mtcp/src/tcp_in.c
index 25eb40e..154135c 100644
--- a/mtcp/src/tcp_in.c
+++ b/mtcp/src/tcp_in.c
@@ -1179,6 +1179,8 @@ ProcessTCPPacket(mtcp_manager_t mtcp,
        /* Validate sequence. if not valid, ignore the packet */
        if (cur_stream->state > TCP_ST_SYN_RCVD) {
+               if(tcph->doff == sizeof(struct tcphdr) >> 2)
+                       cur_stream->saw_timestamp = FALSE;
                ret = ValidateSequence(mtcp, cur_stream,
                                cur_ts, tcph, seq, ack_seq, payloadlen);
                if (!ret) {

[eunyoung14]

I'm afraid it's not enough.
When estimating RTT, mTCP uses the returned timestamp in current implementation. We did not implement measuring RTT when there is no timestamp option. (see line 483 in mtcp/src/tcp_in.c)
To update RTT correctly, this part also needs to be implemented.

[vincentmli]

does structure tcp_stream or tcp_send_vars or tcp_recv_vars need a new field say ts_sent to track the time/ticks when a tcp segment is sent and use cur_ts - ts_sent as RTT measure parameter to function EstimateRTT? I checked early Linux version seems used the idea to caculate RTT and RTO. I noticed there is last_active_ts in tcp_stream, can last_active_ts be used for this purpose?

[ajamshed]

Perhaps..... I would have to look at the RFC and/or look at how other networking stacks behave in this situation. Let me get back to you on this issue. Please give us some time.