NETOBSERV-1112: Support adding of connection setup timestamp

[praveingk]

Currently the flow records from ebpf agent provide two timestamps : 1) connection start, and 2) connection end.
Additionally, we could have connection setup timestamp, which is either (SYN+ACK), or ACK+Seq1 timestamp depending on the direction of the flow.

The logic could :

Ingress:

1. Store timestamp (t2) when ACK+SEQ1 packet is received. Incoming SYN packet timestamp (t1) is already stored.
2. Store timestamp (t2) when SYN+ACK packet is received. Outgoing SYN packet timestamp (t1) would already be stored. Mostly this case is already captured in eBPF data path, since we store the timestamp when an entry is created in the map. In this case SYN+ACK packet will create a new entry in the map.

Using existing marker stored (for SYN+ACK, SYN) in TCP flags, we can calculate t2-t1 in user space to get connection setup time.

This connection setup RTT can be captured per flow, and exported. The consumer can then estimate the Path RTT based on the moving average or some other calculation.

CC : @eranra @jotak @msherif1234

[jotak]

sounds good to me!
@praveingk is that something you want to work on? (I see you took the assignment).

[praveingk]

@jotak : Yes, I was planning to work on it. If someone else wants to work on it, I am ok wth that. 👍🏽

[jotak]

@praveingk no problem you can go ahead, thanks! :-)

[dushyantbehl]

@praveingk @jotak @msherif1234 @eranra

While working on this issue I noticed a problem with the above design. The hypothesis above assumes that the packets start with sequence number 0 for SYN and hence we can record the timestamp of SYN+ACK and ACK+SEQ1 in the agent and perform RTT calculation in the userspace.
The actual TCP behaviour is slightly different because TCP standard starts packets with a random sequence number from both side like below,

The behaviour which was noticed by @praveingk above where packet sequence numbers start with 0 was on packet captures on Wireshark which explicitly converts the sequence numbers to relative sequence numbers because of a default setting in wireshark and can show absolute numbers as well if the setting is disabled. Linux does not convert sequence numbers to relative sequences so the packet that we get in our agent contains actual sequence number.

I have tested a small piece of code in ebpf which stores the first SYN packet sequence number from each side and matches the ACK packet to see if it is the next packet in sequence to the first syn packet and calculates RTT based on the difference of occurance of both.
To implement this in our agent I will need to save packet start sequence numbers to track ACK from reverse flow, if SYN is sent from egress, SYN+ACK will be received on ingress; if SYN+ACK is sent on egress, ACK will be received on ingress.
And this is what I plan to implement.

Let me know if anyone has comments on the design or any questions on the finding above.

[dushyantbehl]

Also, can someone please add me to the repo and assign this issue to me formally.

[praveingk]

Hi Dushyant, Thanks for investigating this.
I think your plan sounds reasonable. I assume you will be using a common (for ingress/egress) Hashmap with key as <Seqno., ifindex> and value as u64 (timestamp)

1. When SYN is observed, the seq no. is updated in the map with timestamp value
2. When ACK pkt is received, lookup is performed on the Hashmap:
   i) If match is found, rtt is computed and stored in the regular flow hash map.
   ii) No match, proceed as usual.

@dushyantbehl @jotak @msherif1234 : Please let me know

[dushyantbehl]

@praveingk yes that's exactly what I had tested in ebpf code and it works! but I think for the second check where we check ACK packets against the hashmap to calculate RTT or not will put an extra hashmap check in the ebpf data path for all ACK packets which might slow down the data path.

So, I'm going to record the timestamp and try to perform this calculation in the userspace.

[dushyantbehl]

@praveingk The agent merges a lot of state in userspace while aggregation which would've meant that doing this calculation in userspace would have been very cumbersome.
In the end I added the RTT calculation in ebpf space which passes it to the userspace as aggregated metrics or via ringbuffer like it generally does, keeping the design simple.

It should not be huge overhead as I have seen other papers follow similar approach.

The final approach is this,
See a SEQ packet at Egress -> Record it in a hashmap with (4 tuple + seq id) as key and (ts) as value

See an ACK packet at Ingress -> Check the hashmap for reverse (4 tuple + seq id) -> if not found move forward, if found calculate the RTT and pass it up.

The PR #117 is implementing this logic.

Let me know any feedback. guys.