Adam edited this page Jan 14, 2013 · 2 revisions

Table of Contents

Buffer Monitoring System


In order to record the development of buffer sizes in switches, a system is needed to record arrival and departure of packets into the queue. The buffer monitoring subsystem is designed to meet this need. The subsystem works by monitoring events such as arrival/departure/drop of a packet on the queues, then recording a Timestamp as well as the length of the packet. These events are then output in a packet with a certain format called the event packet.

Using the system

The system has two software components: monitor_ctrl and rcv_evts. These two components control the buffer monitoring system and record the events at the receiver respectively.


The monitor_ctrl program is found in netfpga/projects/router_buffer_sizing/sw. It implements all the functions to control and view the status of the buffer monitoring system. The program works by reading and changing the control register of the buffer monitoring system. In addition, several read-only registers were added to observe the behavior of the subsystem. Executing monitor_ctrl -h will show available options. monitor_ctrl -p will print the status.

For more information on buffer monitoring registers please see the Register Map.


The rcv_evts program is supposed to record incoming evts, parse them, and display the output to stdout. It can be used as a basis on which to do more interesting things with the event packet information (such as build the queue occupancy in time). To use the program, simply run:

 rcv_evts -i nf2c3 -v 
where nf2c3 is the interface receiving the event packets.

Design Details

The system is composed of two main modules: evt_rcrdr and evt_pkt_wrtr. The evt_rcrdr module records individual events as they happen and serializes them to be sent out to the evt_pkt_wrtr module. The evt_pkt_wrtr then reads each event and stores it, and then when the send_egress_pkt is ready, it sends out a complete event packet.


On every clock cycle, there are 4 types of possible events: A packet stored, a packet removed, a packet dropped, and a Timestamp event. The first three event types we call short events because they only need 32 bits. Whereas the Timestamp is a long event since it uses 64 bits. The short events carry the 19 least significant bits of the clock. Periodically, a Timestamp event is signaled and recorded. The evt_rcrdr rearranges the events as they come in and stores them in a single clock cycle into the event fifo. The event fifo is a shallow fifo (depth=8) with a variable input size. The input is composed of five 32-bit words of which we can store a variable number of words (the first x words are stored). This is the number of events at the current clock cycle (the Timestamp event takes 2 words). Note that the evt_rcrdr assumes that the events are all independent. The maximum sustained event recording capability is 62.5 million events per second, whereas the peak event recording capability is 8 events in any 32 ns interval (after which events should not go over the average or events would be lost). It is possible to adapt the evt_rcrdr to record any signal that meets these requirements with an additional field of 9-bits (usually the length field) to record any additional data. In addition, the evt_rcrdr make no assumption on the sizes of the fields. These could be modified by changing the sizes in unet_defines.v. However, note that the evt_pkt_wrtr does make the assumption that the word sizes are 32 bits since that would simplify writing to the send_egress_pkt. However, this could be easily adapted as well.


This module reads events from the evt_rcrdr module and delineates them into packets. The module also monitors the datapath for activity and only injects event packets when the datapath is idle.


The system follows the reference user data path.

Packet Format

Following the Ethernet, IP, and UDP headers, there are 4 reserved bits, then 4 bits indicating the version of the buffer monitoring system, then 8 bits indicating the number of event types excluding the Timestamp event type. This is followed by a 32 bit sequence number (starting at 0) and a list of the queue sizes before the first short event. A series of short events then follows. Periodic Time Stamps make it possible to keep track of the exact time of each event.

There are four event types:

1- Arrival/Departure/Drop (short) Events. Note that the packet length here is in words of 64-bits. The default time resolution is 8ns.

2- Timestamp Event: Periodically recorded to keep the time updated.

What to do From Here

Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.