Skip to content

Latest commit

 

History

History
165 lines (83 loc) · 7.75 KB

README.md

File metadata and controls

165 lines (83 loc) · 7.75 KB

Innova2 XDMA OpenCAPI

This is a Vivado 2023.2 starter project for the XCKU15P FPGA on the Innova-2 SmartNIC MNV303212A-ADLT that uses the OpenCAPI connector for PCIe and I2C.

The GTY Transceivers connected to the OpenCAPI Connector are in a column that does not contain the Configuration Block so it is impossible for the FPGA to be programmed within the 100ms PCIe power-up time limit.

XCKU15P FFVE1517 Bank Diagram.png

Motherboard boot must be delayed to allow the FPGA to configure itself before PCIe devices are enumerated. This can be accomplished by toggling the POWER button, then pressing and holding the RESET button for a second before releasing it. Or, connect a capacitor across the reset pins of an ATX motherboard's Front Panel Header.

Delay Boot Using Front Panel Header Capacitor

Currently testing using a Second Revision OpenCAPI-to-PCIe adapter. PCIe x8 using the OpenCAPI connector works but requires a high quality cable and uses a PCIe Lane to Transceiver Channel ordering that Vivado complains about.

Vivado Critical Warning

Refer to this tutorial for detailed instructions on generating a similar project from scratch.

Related Projects: OpenCAPI-to-PCIe, OpenCAPI_Breakout, PCIe_x8_Breakout.

Block Design

PCIe XDMA using OpenCAPI Block Diagram

An offset is used for AXI_Lite to simplify memory-mapping the IIC (I2C) Block.

PCIe-to-AXI Translation Offset

AXI Addresses

Block Address (Hex) Size
M_AXI BRAM_CTRL_1 0x00000000 128K
M_AXI_LITE BRAM_CTRL_0 0x40040000 64K
M_AXI_LITE GPIO_3 0x40050000 64K
M_AXI_LITE IIC_0 0x40060000 64K

AXI Addresses

Program the Design into the XCKU15P Configuration Memory

Refer to the innova2_flex_xcku15p_notes project's instructions on Loading a User Image. Binary Memory Configuration Bitstream Files are included in this project's Releases.

wget https://github.com/mwrnd/innova2_xdma_opencapi/releases/download/v0.1/innova2_xdma_opencapi_bitstream.zip
unzip innova2_xdma_opencapi_bitstream.zip
sha256sum *bin
echo 4b95eea589159197ab7c973038cfea28e6a4fe04385a92a70a736d428d8ab882 should be Checksum of innova2_xdma_opencapi_primary.bin
echo f22c0fefabcf8eb9ced863bf11f3b252b336faa0f2778f6cac1588dd74e08daf should be Checksum of innova2_xdma_opencapi_secondary.bin

Testing the Design

lspci

After programming the bitstream and rebooting, the design should show up as Memory controller: Xilinx Corporation Device 9038 under lspci. It shows up at PCIe Bus Address 01:00 for me but this depends on the PCIe connector you plug your board into.

sudo lspci -tv | grep -i "Mellanox\|0000\|Xilinx\|1d"

lspci Tree and Xilinx Devices

Using a 3M 8ES8-1DF21-0.75 cable and a Second Revision OpenCAPI-to-PCIe adapter the PCIe Link Status is usually excellent:

sudo lspci -v -d 10ee: ; sudo lspci -vvv -d 10ee: | grep LnkSta

lspci Link Status

dmesg | grep -i xdma provides details on how Xilinx's PCIe XDMA driver has loaded.

dmesg xdma

PCIe Link Status versus Cable

Using a 3M 8ES8-1DF21-0.75 cable:

System with 3M 8ES8-1DF21-0.75 Cable

PCIe Link Status is usually excellent:

lspci Link Status

Using an SFPCables.com SFF-8654 to SFF-8654 8i cable:

System with SFPCables SFF-8654 8i 85Ohm Cable

PCIe Link Status is downgraded:

lspci Link Status

I am working on a third revision of the OpenCAPI-to-PCIe adapter to improve signal integrity.

OpenCAPI I2C over XDMA

The OpenCAPI-to-PCIe adapter routes the OpenCAPI I2C signals to an external connector which has the same pin ordering as a TC74 Temperature Sensor. Note 3.3V is from the PCIe connector.

TC74A0-3.3VAT in OpenCAPI-to-PCIe Adapter

Complete system:

TC74A0-3.3VAT Being Tested in a System

innova2_xdma_opencapi_iic_tc74_test.c is a simple program to read the temperature and configuration registers of the sensor. Refer to these notes on AXI IIC (I2C) over XDMA for more information.

Compile and run the TC74 test program:

make
sudo ./innova2_xdma_opencapi_iic_tc74_test

Read TC74 Registers

After spraying the sensor with some Freeze Spray the temperature drops to below 0C. The data is in 2's-complement format.

After using Freeze Spray

There are only two I2C registers in the TC74:

TC74 I2C Registers

Generating the Design in Vivado

Run the source command from the Vivado 2023.2 Tcl Console:

cd innova2_xdma_opencapi
dir
source innova2_xdma_opencapi.tcl

Source Project Files

Wait for Synthesis and Implementation runs to finish. The programming files innova2_xdma_opencapi_primary.bin and innova2_xdma_opencapi_secondary.bin will be generated.

Wait On Synthesis and Implementation Runs to Finish

Resources used for the design:

Design Run Results