Skip to content

Latest commit

 

History

History
210 lines (163 loc) · 7.21 KB

README.md

File metadata and controls

210 lines (163 loc) · 7.21 KB
Raw

Pigasus Multi-FPGA Example

Pigasus Multi-FPGA example splits Pigasus pipeline into two sub-pipeline and map them onto two FPGAs. The splitting point can be configured using Fluid Framework(please see pigasus_multi.py). Most of Pigasus source code is under pigasus directory. This directory only contains files related to multi-FPGA RTL simulation and hardware test. The default example split Pigasus after fast-pattern matcher and before the port-group.

Build Pigasus Multi-FPGA Example

Hardware Development Requirements

Please refer to Pigasus Hardware Development Requirements.

RTL Simulation

The testbench instantiates two sub-pipeline (top_0.sv and top_1.sv, which are generated by the Fluid Framework). The core logic of Pigasus is copied from pigasus\hardware\rtl_sim\src. To run simulation, you only need to do:

./run_vsim.sh ./input_gen/output.pkt

Hardware Synthesis

Create Quartus project.

cd $pigasus_rep_dir/pigasus_multi/scripts/
./run_quartus_create_multi.sh

Run synthesis.

cd $pigasus_rep_dir/pigasus_multi/scripts
./run_quartus_synth_0.sh
./run_quartus_synth_1.sh

Software Configuration

This is the same as pigasus\software. You only need to install the software on the host machine of the second FPGA (if the two FPGAs are not using the same host).

Running Pigasus Multi-FPGA Example

Below is based on an example setup with two machines. One hosts Pigasus (Intel Stratix 10 MX FPGA + Full Matcher running on CPU), the other generates packets. We use DPDK pkt-gen to push empirical traces to the attached 100G Mellanox NIC. The Mellanox NIC is connected back-to-back with the Pigasus FPGAs through a 100Gb cable. Pigasus is partitioned to two FPGAs. We refer them as pigasus_first and pigasus_second. You can host these two FPGAs on the same machine or separate machines. The NIC connect with pigasus_first port 0 (lower port). Then pigasus_first port 1 (upper port) is connected to pigasus_second port 0(lower port).

Hardware-only Test

We provide an easy way to just exercise the FPGA datapath without interacting with the software. In this case, you don't need to install and run the software.

Load the bitstream to the Intel MX development kit.

cd $pigasus_rep_dir/pigasus_multi/hw_test_0
./load_bitstream.sh

Open another terminal and do:

cd $pigasus_rep_dir/pigasus_multi/hw_test_1
./load_bitstream.sh

Note that the JTAG system console should be closed when loading the bitstream and the USB port number should be adjusted according to the machine.

Run the JTAG system console with:

cd $pigasus_rep_dir/pigasus_multi/hw_test_0
./run_console

Same for the pigasu_second FPGA

cd $pigasus_rep_dir/pigasus_multi/hw_test_0
./run_console

After the console starts, run on both terminals:

source path.tcl

It may return some error the first time. Exit it using Ctrl-C. Then relaunch the console (./run_console) and rerun source path.tcl.

In the tcl console, you will need to type some tcl commands. On the pigaus_second FPGA, do:

disable_pcie
get_results

This would return all 0 as no packets have been sent at this point. You may consider leaving the tcl console open while you run Pigasus to let you check the counters. Once you are ready to exit it, use Ctrl-C.

Start the packet generator in the other machine (here we use m10_100.pcap in $pigasus_rep_dir/pigasus/hardware/rtl_sim/input_gen as the example pcap):

cp example.pcap /dev/shm/test.pcap
cd dpdk/pktgen-dpdk/
./run_pktgen.sh

Refer to the DPDK Pktgen docs for details on how to run it.

Remember to set the number of packets to match the pcap, otherwise the packet generator sends the pcap repeatedly, and set the rate. Y = 1 means 1 Gbps.

set 0 count X
set 0 rate Y
str

After the packet generator finishes sending the number of packets that you specified, recheck the counters on the FPGA using the tcl console on pigasus_first FPGA:

## recheck the top counters, you can see the recv pkts, processed pkts and dma pkts.
get_top_stats

If you are using our sample rule and pcap (m10_100.pcap). You should expect to see the results below.

IN_PKT: 100
SM_PKT: 100
SM_META: 10
SM_RULE: 10
SM_CHECK_PKT: 10
OUT_1_PKT: 30

This suggests 10 packets and 10 rules are matched after fast-pattern matcher. And packet data, rule data and metadata are muxed and sent to next FPGA. Therefore, you should see OUT_1_PKT to be 30, which is the total packet number that go out through Ethernet Port 1.

Now, check the pigasus_second FPGA.

## recheck the top counters, you can see the recv pkts, processed pkts and dma pkts.
get_top_stats

You should expect to see the results below.

IN_PKT: 30
PG_PKT: 10
NF_PKT: 10
DMA_PKT: 10

This suggests that those 10 packets and rules pass the port-group and non-fast-pattern matcher. Since PCIe is disabled, those DMA_PKT will be discarded.

Full system test

Once both the software and hardware components are configured, you can run full system test.

For pigasus_first FPGA, please follow the instructions in above section. We now describe the instructions for pigasus_second FPGA.

After the bitstream finishes loading, reboot the machine to allow the PCIe core on the pigasus_second FPGA be recognized by the system:

sudo reboot

Once the system finishes booting, run the JTAG system console with:

cd $pigasus_rep_dir/pigasus/hw_test_1
./run_console

After the console starts, run:

source path.tcl

In the tcl console, set the buffer size:

set_buf_size 262143

And set the number of cores:

set_core_num 1

You can also verify the FPGA counter stats with:

get_results

To run the software, first insert the kernel module:

cd $pigasus_rep_dir/pigasus/software/src/pigasus/pcie/kernel/linux
sudo ./install

Then execute the software application, specifying the snort.lua configuration file and the rule_list file (specifying the sid of the rules are used). We only provide a sample rule as the Snort Registered Rules requires registration on Snort. This sample rule will be triggered 10 times by our sample trace $pigasus_rep_dir/hardware/rtl_sim/input_gen/m10_100.pcap.

cd $pigasus_rep_dir/pigasus/software/lua
sudo pigasus -c snort.lua --patterns ./rule_list

Run following commands on the packet generator in the other machine:

cp example.pcap /dev/shm/test.pcap
cd dpdk/pktgen-dpdk/
./run_pktgen.sh

Remember to set the number of packets to match the pcap, otherwise the packet generator sends the pcap repeatedly, and set the rate. Y = 1 means 1 Gbps.

set 0 count X
set 0 rate Y
str

After the packet generator finishes sending the number of packets that you specified, recheck the counters on the FPGA using the tcl console:

## recheck the top counters, you can see the recv pkts, processed pkts and dma pkts.
get_results

If you are using our sample rule and pcap (m10_100.pcap). You should expect to see the results below.

IN_PKT: 100
PROCESSED_PKT: 100
DMA_PKT: 10

Now exit the software application with Ctrl-C. You should expect to see that the number of rx_pkt should match up the DMA_PKT.