BufferMonitoringSystem
In order to record the development of buffer sizes in switches, a system is needed to record arrival and departure of packets into the queue. The buffer monitoring subsystem is designed to meet this need. The subsystem works by monitoring events such as arrival/departure/drop of a packet on the queues, then recording a Timestamp as well as the length of the packet. These events are then output in a packet with a certain format called the event packet.
The system has two software components: monitor_ctrl and rcv_evts. These two components control the buffer monitoring system and record the events at the receiver respectively.
The monitor_ctrl program is found in netfpga/projects/router_buffer_sizing/sw. It implements all the functions to control and view the status of the buffer monitoring system. The program works by reading and changing the control register of the buffer monitoring system. In addition, several read-only registers were added to observe the behavior of the subsystem. Executing monitor_ctrl -h will show available options. monitor_ctrl -p will print the status.
For more information on buffer monitoring registers please see the Register Map.
The rcv_evts program is supposed to record incoming evts, parse them, and display the output to stdout. It can be used as a basis on which to do more interesting things with the event packet information (such as build the queue occupancy in time). To use the program, simply run:
rcv_evts -i nf2c3 -vwhere nf2c3 is the interface receiving the event packets.
The system is composed of two main modules: evt_rcrdr and evt_pkt_wrtr. The evt_rcrdr module records individual events as they happen and serializes them to be sent out to the evt_pkt_wrtr module. The evt_pkt_wrtr then reads each event and stores it, and then when the send_egress_pkt is ready, it sends out a complete event packet.
On every clock cycle, there are 4 types of possible events: A packet stored, a packet removed, a packet dropped, and a Timestamp event. The first three event types we call short events because they only need 32 bits. Whereas the Timestamp is a long event since it uses 64 bits. The short events carry the 19 least significant bits of the clock. Periodically, a Timestamp event is signaled and recorded. The evt_rcrdr rearranges the events as they come in and stores them in a single clock cycle into the event fifo. The event fifo is a shallow fifo (depth=8) with a variable input size. The input is composed of five 32-bit words of which we can store a variable number of words (the first x words are stored). This is the number of events at the current clock cycle (the Timestamp event takes 2 words). Note that the evt_rcrdr assumes that the events are all independent. The maximum sustained event recording capability is 62.5 million events per second, whereas the peak event recording capability is 8 events in any 32 ns interval (after which events should not go over the average or events would be lost). It is possible to adapt the evt_rcrdr to record any signal that meets these requirements with an additional field of 9-bits (usually the length field) to record any additional data. In addition, the evt_rcrdr make no assumption on the sizes of the fields. These could be modified by changing the sizes in unet_defines.v. However, note that the evt_pkt_wrtr does make the assumption that the word sizes are 32 bits since that would simplify writing to the send_egress_pkt. However, this could be easily adapted as well.
This module reads events from the evt_rcrdr module and delineates them into packets. The module also monitors the datapath for activity and only injects event packets when the datapath is idle.
The system follows the reference user data path.
Following the Ethernet, IP, and UDP headers, there are 4 reserved bits, then 4 bits indicating the version of the buffer monitoring system, then 8 bits indicating the number of event types excluding the Timestamp event type. This is followed by a 32 bit sequence number (starting at 0) and a list of the queue sizes before the first short event. A series of short events then follows. Periodic Time Stamps make it possible to keep track of the exact time of each event.
There are four event types:
1- Arrival/Departure/Drop (short) Events. Note that the packet length here is in words of 64-bits. The default time resolution is 8ns.
2- Timestamp Event: Periodically recorded to keep the time updated.
- You can go through the Reference NIC Walkthrough if you are interested in using the NetFPGA as a network interface card
- You can go through the SCONE Software Walkthrough if you are interested in writing software for the NetFPGA to run in the host
- You can go through the Linux Router Kit Walkthrough if you are interested in using the NetFPGA as a high performance router
- You can go through the Reference Router Walkthrough if you are interested in modifying the gateware on the FPGA
- Return to the Main Guide