-
Notifications
You must be signed in to change notification settings - Fork 1
Headers
Ilya Baldin edited this page Jul 29, 2024
·
10 revisions
EJ-FAT uses gRPC messages and UDP packets for its control plane and UDP only for its data plane. E2SAR has 3 pre-defined headers that go immediately after the UDP header: Sync header for Sync messages sent from the sender to the control plane, LoadBalancer (LB) and Reassembly (RE) headers. The latter can appear together (LB+RE) when sent by the data sender and when the load balancer strips off the LB header needed for it, only the RE header and data proceed. As the name suggests, LB header is needed by the Load Balancer and the RE header is needed by the receiver to reassemble the event data.
IP
IP Source = IP address of the Data Source
IP Destination = Sync IP from ReserveLoadBalancerReply
IP Protocol/NextHeader = UDP
UDP
UDP Destination Port = 19523 (0x4C43) ("LC")
UDP Source Port = Any
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| L | C | Version | Rsvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| EventSrcId |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ EventNumber +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| AvgEventRateHz |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ UnixTimeNano +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
32b Control Word
16b Magic = 0x4C43 ("LC")
8b Version = 1
8b Rsvd, MBZ
32b EventSrcId
Set this to a unique ID for the Event Source that produced this message
64b EventNumber
Set this to the EventNumber that would be used if an Event Data message were to be sent right now
32b AvgEventRateHz
Set this to the best available estimate of how much the EventNumber increases per second
If no estimate is available, set this field to zero
64b UnixTimeNano
Set this to the current wall clock time of this Event Source. Time is in nanoseconds since 1970-01-01T00:00:00Z (ie. midnight UTC)
If no wall clock timestamp is available, set this field to zero
The C/C++ definition looks as follows:
using EventNum_t = u_int64_t;
using UnixTimeNano_t = u_int64_t;
using EventRate_t = u_int32_t;
struct SyncHdr
{
char preamble[2] {'L', 'C'};
u_int8_t version{synchdrVersion};
u_int8_t rsvd{0};
u_int32_t eventSrcId{0};
EventNum_t eventNumber{0LL};
EventRate_t avgEventRateHz{0};
UnixTimeNano_t unixTimeNano{0LL};
} __attribute__((__packed__));
IP
IP Source = Any
IP Destination = LB Hardware Data Plane IPv4 or IPv6
IP Protocol/NextHeader = UDP
UDP
UDP Destination Port = 19522 (0x4C42) (“LB”)
UDP Source Port = Any (typically set to 16 least significant bits of Event Number). It is important to have entropy in this so that packets are spread across LAGs.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| L | B | Version | NextProtocol |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Rsvd | Entropy |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ EventNumber +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
64b LB Control Word
16b Magic = 0x4C42 (“LB”)
8b Version = 2
8b NextProtocol ID
16b Reserved, MBZ
16b Entropy
64b LB Event Number - this can just be a timestamp
Used to partition the event space into Calendar Epochs
Repeats across multiple packets related to the same event
MUST NOT wrap
The C/C++ definition looks as follows:
struct LBHdr
{
char preamble[2] {'L', 'B'};
u_int8_t version{lbhdrVersion};
u_int8_t nextProto{rehdrVersion};
u_int16_t rsvd{0};
u_int16_t entropy{0};
EventNum_t eventNum{0L};
} __attribute__((__packed__));
IP
IP Source = Any
IP Destination = LB Hardware Data Plane IPv4 or IPv6
IP Protocol/NextHeader = UDP
UDP
UDP Destination Port = 19522 (0x4C42) (“LB”)
UDP Source Port = Any (typically set to 16 least significant bits of Event Number). It is important to have entropy in this so that packets are spread across LAGs.
LB Header
Stripped off by load balancer and not seen by the receiver
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| Rsvd | Data-ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Buffer Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Buffer Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ EventNumber +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4b Version = 1
12b Reserved
16b Data-ID = source identifier
32b Buffer Offset = The byte index of where the data in this packet should be put in the buffer
32b Buffer Length = The size of buffer needed to store this event
64b Event Number = Same as in the event data header but for your application
The C/C++ definition looks like this:
struct REHdr
{
u_int8_t preamble[2] {rehdrVersionNibble, 0}; // 4 bit version + reserved
u_int16_t dataId{0}; // source identifier
u_int32_t bufferOffset{0};
u_int32_t bufferLength{0};
EventNum_t eventNum{0};
} __attribute__((__packed__));
There is a scapy script that can generate and listen for dataplane and sync control plane messages. It can listen/generate sync packages, data plane with LB+RE headers and dataplane with only RE headers:
$ ./snifgen.py -h
usage: snifgen.py [-h] (-l | -g) [-p PORT] [-c COUNT] [--ip IP] [--show] [--entropy ENTROPY] [--event EVENT] [--rate RATE] [--dataid DATAID] [--srcid SRCID] [--mtu MTU] [--pld PLD] [--iface IFACE] (--sync | --lbre | --re)
options:
-h, --help show this help message and exit
-l, --listen listen for incoming packets and try to parse and validate them
-g, --generate generate new packets of specific types
-p PORT, --port PORT UDP port (for -l and -g)
-c COUNT, --count COUNT
number of packet streams (if pld larger than mtu, otherwise packets) to generate or expect
--ip IP IP address to which to send the packet(s) or listen from
--show only show the packet without sending it (with -g)
--entropy ENTROPY entropy value for LB+RE packet
--event EVENT event number for sync, LB+RE and RE packet
--rate RATE event rate in Hz for Sync packet
--dataid DATAID data id for RE packet
--srcid SRCID source id for Sync packet
--mtu MTU set the MTU length, so LB+RE and RE packets can be fragmented.
--pld PLD payload for LB+RE or RE packets. May be broken up if MTU size insufficient
--iface IFACE which interface should we listen on (defaults to all)
--sync listen for or generate sync packets
--lbre listen for or generate packets with LB+RE header
--re listen for or generate packets with just the RE header