Verification
| Attention: This guide applies to the Python-based verification infrastructure available in 3.0.0 and above releases. |
This document provides a reference for the Python-based verification infrastructure. The infrastructure allows for testing in simulation and on hardware using the same source test files.
There are three new major features in the new verification infrastructure: unified simulation and hardware tests, barrier synchronization, and support for multiple testing configurations.
In the new python testing infrastructure, simulation and hardware (previously regression) tests have been unified, so a test can be written once and run as either a simulation or hardware test, unless hardware specific functions are needed. Tests should be placed in a project's test directory. Test directories should be named both_<major>_<minor> if they can be run in both simulation and hardware, hw_<major>_<minor> if the test can only be run as a hardware test, and sim_<major>_<minor>; if the test can only be run as a simulation test. Neither major or minor can have underscores in the name, nor can they be blank.
Instead of specifying times for packet and register operations, the new infrastructure uses a barrier statement for synchronization. The barrier blocks until all expected packets arrive, or it times out, causing the test to fail. This ensures that register operations occur at the correct time relative to the packet operations. See both_oq_sramSzMAC in the reference_router as an example of many register operations interspersed with packet operations.
Notes: In order to ensure register reads and writes occur at the correct time relative to packet send and expects, a barrier must be placed between all register operations and packet operations. Both nftest_start and nftest_finish call a barrier, so putting a barrier directly after nftest_start or directly before nftest_finish is redundant.
The new testing infrastructure supports tests running in simulation and multiple hardware configurations by passing arguments to nftest_init.
nftest_init(sim_loop = [], hw_config = None)
For a simulation test, a list of interfaces to put into loopback is passed to the keyword argument sim_loop. For example, to put nf2c2 and nf2c3 (ports 3 and 4) into loopback, the argument is =nftest_init(sim_loop = ['nf2c2',].
In order for a test to be run in hardware, the physical connections must be known. These connections are specified in a connections file in the “connections” directory. The testing infrastructure supports running tests with different hardware connections, so an initial loopback setting is also required for each connections file. The (connections, loopback) pair is a hardware configuration. A list of possible hardware configurations is passed as a keyword argument to nftest_init. If a connections file is passed to nf_test.py, nftest_init will go through the list of hardware configurations and attempt to find a matching configuration to run, otherwise the first configuration in the list will be run.
The directory structure is as follows (excerpted from reference router):
test both_router_cpusend config.txt run.py both_router_table config.txt run.py connections 2phy 4phy global setup hw_send_receive run.py
connections is the connections folder, where connections files for the project can be placed. A connections file specifies how the network interfaces are physically connected. The connections file is formatted with one connection per line, where the connection is specified by nf2cX:ethY, denoting that the interface nf2cX is physically connected to ethY. For example,
nf2c0:eth1 nf2c1:eth2
shows that nf2c0 is connected to eth1 and nf2c1 is connected to eth2.
run.py is the executable script which runs the test. If run.py calls another script which calls nftest_init, it is important to know that arguments passed to run.py must be passed along so the test will know whether to run in hardware mode instead of simulation mode. For example, run.py in the router_buffer_sizing test hw_store_events test calls send_pkts.py, which runs nftest_init, so the arguments are passed to send_pkts.py, as shown below.
send_pkts = "./send_pkts.py" for arg in sys.argv: send_pkts += " %s"%arg subprocess.call(send_pkts.split())
config.txt is only necessary for tests which can run in simulation, and specifies a test description. It is no longer necessary to specify a finish time, as this is done by the last barrier.
- --major <string>
- --minor <string>
For a complete listing of arguments, call nf_test.py --help
Due to the way simulations work, there are a few additional considerations for simulation tests.
In addition, there are cases where a register read or write may occur while a packet is still being processed, resulting in unexpected behavior. The simulation infrastructure will attempt to detect if a register read was performed immediately after a barrier, and print a warning if it finds that the register read is off by 1.
WARNING: Register read expected and seen differed by 1 after a barrier.
Register reads are not always delayed appropriately by a barrier, try adding a delay. Although this warning only appears if the register read is checking a counter which increments by 1, this problem is not so limited in scope. The fix is to add a register delay in simulation to prevent the register read from occurring too early. The time to delay will vary, and it is advised to verify roughly how much delay is needed in the gui. Examples of this can be found in two reference router tests: both_badipchksum_packet and both_wrong_destMAC.
Below are the most commonly used methods for writing tests. Arguments and further detail can be found in the pydoc.NFTest - package which provides project specific register defines and scapy, and the main test library, NFTestLib
Initialization and finalization
- nftest_init - loads and parses the connections file, map file
- nftest_start - starts packet sniffing threads, performs initial reset
- nftest_finish - writes resultant pcap files
- nftest_send_phy
- nftest_send_dma
- nftest_expect_phy
- nftest_expect_dma
- nftest_regwrite
- nftest_regread_expect
- nftest_barrier - synchronization tool. waits for expected packets to arrive or times out
- nftest_fpga_reset - resets the fpga
- isHW - returns true if test is being run as a hardware test, used to enable a test to have hardware only checks which cannot be done in sim while maintaining support for both hardware and sim
- make_IP_pkt ( src_MAC, dst_MAC, EtherType, src_IP, dst_IP, TTL )
- make_ICMP_request_pkt ( src_MAC, dst_MAC, EtherType, src_IP, dst_IP, TTL )
- make_ICMP_reply_pkt ( src_MAC, dst_MAC, EtherType, src_IP, dst_IP, TTL )
- make_ICMP_ttl_exceed_pkt ( src_MAC, dst_MAC, EtherType, src_IP, dst_IP, TTL )
- make_ARP_request_pkt ( src_MAC, dst_MAC, EtherType, src_IP, dst_IP)
- make_ARP_request_pkt ( src_MAC, dst_MAC, EtherType)
- generate_load (length)
- reset_phy - resets all the PHYs
- regread - register read and returned, no comparison done
- phy_loopback - puts the specified nf2 interface into loopback
Note: Loopbacks should be specified in the nftest_init configuration, unless they must be changed during the test, which can only be done in hardware tests
- restart - resets received and expected packet lists
- regDelay - needed for sim specific synchronization issues
#!/bin/env python
from NFTest import *
import random
import sys
phy0loop4 = ('../connections/conn', ['nf2c0', 'nf2c1', 'nf2c2', 'nf2c3'])
nftest_init(sim_loop = ['nf2c0', 'nf2c1', 'nf2c2', 'nf2c3'], hw_config = [phy0loop4])
nftest_start()
# set parameters
SA = "aa:bb:cc:dd:ee:ff"
TTL = 64
DST_IP = "192.168.1.1"
SRC_IP = "192.168.0.1"
nextHopMAC = "dd:55:dd:66:dd:77"
if isHW():
NUM_PKTS = 50
else:
NUM_PKTS = 5
print "Sending now: "
totalPktLengths = [0,0,0,0]
# send NUM_PKTS from ports nf2c0...nf2c3
for i in range(NUM_PKTS):
sys.stdout.write('\r'+str(i))
sys.stdout.flush()
for port in range(4):
DA = "00:ca:fe:00:00:%02x"%port
pkt = make_IP_pkt(dst_MAC=DA, src_MAC=SA, dst_IP=DST_IP,
src_IP=SRC_IP, TTL=TTL,
pkt_len=random.randint(60,1514))
totalPktLengths[port] += len(pkt)
nftest_send_dma('nf2c' + str(port), pkt)
nftest_expect_dma('nf2c' + str(port), pkt)
print ""
nftest_barrier()
print "Checking pkt errors"
# check counter values
for i in range(4):
nftest_regread_expect(reg_defines.MAC_GRP_0_RX_QUEUE_NUM_PKTS_STORED_REG() + i*reg_defines.MAC_GRP_OFFSET(), NUM_PKTS)
nftest_regread_expect(reg_defines.MAC_GRP_0_TX_QUEUE_NUM_PKTS_SENT_REG() + i*reg_defines.MAC_GRP_OFFSET(), NUM_PKTS)
nftest_regread_expect(reg_defines.MAC_GRP_0_RX_QUEUE_NUM_BYTES_PUSHED_REG() + i*reg_defines.MAC_GRP_OFFSET(), totalPktLengths[i])
nftest_regread_expect(reg_defines.MAC_GRP_0_TX_QUEUE_NUM_BYTES_PUSHED_REG() + i*reg_defines.MAC_GRP_OFFSET(), totalPktLengths[i])
nftest_finish()
Line 1 specifies that this is a python script
Lines 3-5 import the libraries needed for the test
Line 7 specifies a hardware configuration which will be passed to nftest_init
Line 9 calls nftest_init, specifying that all 4 ports should be in loopback for sim, and there is one valid hardware configuration, the one from line 7
Lines 13-21 specify parameters to create packets
Line 23 prints output so if the test fails in hardware, we know what it was doing. Print statements are not helpful in simulation tests, since the test is run later.
Line 24 creates an array of counters to keep track of bytes sent
Line 25 loops for NUM_PKTS
Lines 27-28 print verbose output for hardware tests
Line 29 loops over 4 ports
Line 30 specifies a destination MAC, based on the port
Lines 31-33 creates a random sized packet using the previously specified parameters
Line 34 increments the bytes sent counter
Line 36 sends the packet over DMA from the specified port
Line 37 expects the packet over DMA on the specified port, since all ports are in loopback
Line 41 waits until all expected packets are received, or times out and produces an error
Line 43 prints output for hardware tests
Line 45-49 perform register reads to check the hardware counters against our expected values
Line 51 finishes and ends the test with the appropriate exit value