Skip to content

ML605 and VC707

Kermin Elliott Fleming edited this page Jun 9, 2015 · 6 revisions

ML605 and VC707

The Xilinx ML605 is a PCIe board with a Virtex-6 LX240T FPGA and 512MB of on-board SDRAM. The VC707 has a Virtex-7 XC7VX485T FPGA and 1GB of on-board SDRAM. LEAP supports both boards using BlueNoC Linux drivers.

Installation for both boards is similar, with the exception of programming cables for the VC707 (see below). Install the LEAP Base Platform Tools and BlueNoC Linux drivers. Configuration of the control script for managing the FPGA boards during programming is, unfortunately, specific to OS versions and host hardware. On machines with support for hot-plug it is necessary to disable the board’s PCIe bus before programming and restore the bus following programming. The example control scripts (installed by default in /usr/share/leap/scripts) are known to work on SUSE SLES11 on hardware supporting hot-plug.

Note in particular the update of PCIe configuration space following FPGA programming in the example scripts. Many machines lose PCIe configuration during hot-plug and the setup must be restored for proper operation. The example scripts copy saved configuration from /usr/share/leap/state/_config to the file-mapped configuration space in the /sys tree. The original configuration space was saved after a clean configuration and boot:

  1. Program the FPGA
  2. Reboot the computer without attempting to access the FPGA
  3. After reboot, copy the device’s configuration from the /sys tree to /usr/share/leap/state

See either the default ML605 or VC707 control scripts for hints at where to find the device configuration files.

Digilent JTAG Chipset

The VC707 uses a Digilent chipset for JTAG programming. Xilinx does not install Digital drivers by default, but does provide them with ISE. To install the drivers:

  1. cd /bin/lin64/digilent
  2. ./install_digilent.sh
  3. cd /ISE/lib/lin64/plugins
  4. mkdir Digilent
  5. cd Digilent
  6. ln -s ../../../../bin/lin64/digilent/libCseDigilent_2.2.10-x86_64/lin64/14.1/libCseDigilent

Immediately after installation, the drivers may be accessible only as root. Rebooting appears to solve this.

Power-on Reset

On machines that support PCIe hot-plug it is often necessary to have the FPGAs PCIe driver working as the operating system boots. One way to achieve this after powering on the machine is to boot the machine, program the FPGA with a design and then reboot. Subsequent reprogramming can use hot-plug and the FPGA will be recognized without rebooting. For VC707, loading the on-board PROM is a better option.

To enable power-on programming, make sure VC707 SW1 has only position 4 (M1) set. All others (1-3, 5) should be zero (toward the numbers on the switch). Build any design that includes a PCIe driver and then perform the following steps (sample commands assume you are using the rrrtest_hybrid_vc707 workload):

  • In the “pm” directory of a workload, generate a bit file suitable for power-on loading:

    bitgen -w -g StartUpClk:CClk -g Compress -g ConfigRate:3 -g BPI_sync_mode:Type1 -g BPI_page_size:1 -g BPI_1st_read_cycle:1 .xilinx/rrrtest_hybrid_vc707_par.ncd .xilinx/flash.bit
  • Generate a PROM MCS file from the bit file:

    promgen -w -p mcs -c FF -bpi_dc parallel -data_width 16 -o .xilinx/flash.mcs -s 131072 -u 00000000 .xilinx/flash.bit
  • Program the MCS file into a PROM by running “impact -batch”. The should be replaced by the ID of your board, e.g. -target “device=SN:210201345455”.

    setMode -bs
    setCable -target “digilent_plugin frequency=30000000 ”
    identify -inferir
    identifyMPM
    attachflash -position 1 -bpi “28F00AG18F”
    assignfiletoattachedflash -position 1 -file “flash.mcs”
    program -p 1 -dataWidth 16 -rs1 25 -rs0 24 -bpionly -e -loadfpga
    verify -p 1 -dataWidth 16 -rs1 25 -rs0 24 -bpionly
    closeCable
    quit

The board should now be programmable using hot-plug immediately after booting.

Setting Up Automated Programming

LEAP attempts to provide coarse-grained management of system-level FPGA resources, through leap-fpga-ctrl. In order for LEAP to achieve this, the programmer must describe the board level resources in the system.
LEAP provides a sample script for the VC707: source:tools/platforms/leap-fpga-ctrl/scripts/VC707.example

Some edits are necessary, as shown in the attached document below. Discovering the PCIe address for a specific card requires examination of individual system hardware. Tools like lspci and dmesg can be helpful in finding these values. Examining the file representation of the PCIe drivers can also help, as shown:

[keflemin-vs-fpga-4 mem_test_hybrid_vc707_vivado_synplify(22)] dmesg
œôó¦
[9741920.272210] pci 0000:04:00.0: reg 10 32bit mmio: [0x000000-0x007fff]
[9741920.272301] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot
[9741920.272313] pci 0000:04:00.0: PME# disabled
[9741920.273399] bluenoc: PCI probe for 0x1be7 0xb100
[9741920.273406] bluenoc: board_number = 1
[9741920.273419] bluenoc 0000:04:00.0: enabling device (0000 -> 0002)
[9741920.273435] bluenoc 0000:04:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
[9741920.273652] bluenoc: revision = 1.0
[9741920.273659] bluenoc: build_version = 34078
[9741920.273665] bluenoc: timestamp = 1417521549
[9741920.273671] bluenoc: NoC is using 16 byte beats
[9741920.273677] bluenoc: Content identifier is c0011eafc0ded00d
[9741920.273787]   alloc kstat_irqs on node 0
[9741920.273802] bluenoc 0000:04:00.0: irq 92 for MSI/MSI-X
[9741920.273821] bluenoc: MSI-X interrupts enabled with IRQ 92
[9741920.273830] bluenoc 0000:04:00.0: setting latency timer to 64
[9741920.273910] bluenoc: /dev/bluenoc_0000:04:00.0 device file created
[9741922.283168] bluenoc: /dev/bluenoc_1 deactivated
[9741922.283219] bluenoc 0000:04:00.0: PCI INT A disabled
[9741924.403026] bluenoc: /dev/bluenoc_2 reactivated
[9741924.403063] bluenoc 0000:83:00.0: PCI INT A -> GSI 64 (level, low) -> IRQ 64
[9741924.403245] bluenoc: revision = 1.0
[9741924.403249] bluenoc: build_version = 34078
[9741924.403253] bluenoc: timestamp = 1417521550
[9741924.403257] bluenoc: NoC is using 16 byte beats
[9741924.403260] bluenoc: Content identifier is c0011eafc0ded00d
[9741924.403369]   alloc kstat_irqs on node 1
[9741924.403386] bluenoc 0000:83:00.0: irq 92 for MSI/MSI-X
[9741924.403409] bluenoc: MSI-X interrupts enabled with IRQ 92
[9741924.403419] bluenoc 0000:83:00.0: setting latency timer to 64
[9741926.418567] bluenoc: /dev/bluenoc_1 reactivated
[9741926.418592] bluenoc 0000:04:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
[9741926.418709] bluenoc: revision = 1.0
[9741926.418713] bluenoc: build_version = 34078
[9741926.418717] bluenoc: timestamp = 1417521549
[9741926.418720] bluenoc: NoC is using 16 byte beats
[9741926.418724] bluenoc: Content identifier is c0011eafc0ded00d
[9741926.418832] bluenoc 0000:04:00.0: irq 93 for MSI/MSI-X
[9741926.418852] bluenoc: MSI-X interrupts enabled with IRQ 93
[9741926.418861] bluenoc 0000:04:00.0: setting latency timer to 64

[keflemin-vs-fpga-4 mem_test_hybrid_vc707_vivado_synplify(22)] ls -l /sys/devices/pci0000:00/0000:00:02.0/
total 0
drwxr-xr-x 3 root root    0 2014-08-11 19:18 0000:00:02.0:pcie01/
drwxr-xr-x 3 root root    0 2014-08-11 19:18 0000:00:02.0:pcie02/
drwxr-xr-x 3 root root    0 2014-12-02 14:38 0000:04:00.0/
-rw-r--r-- 1 root root 4096 2014-09-01 15:01 broken_parity_status
-r--r--r-- 1 root root 4096 2014-08-11 19:18 class
-rw-r--r-- 1 root root 4096 2014-08-11 20:19 config
-r--r--r-- 1 root root 4096 2014-08-11 19:18 device
lrwxrwxrwx 1 root root    0 2014-08-11 19:18 driver -> ../../../bus/pci/drivers/pcieport/
-rw------- 1 root root 4096 2014-09-01 15:01 enable
lrwxrwxrwx 1 root root    0 2014-09-01 15:01 firmware_node -> ../../LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:21/
-r--r--r-- 1 root root 4096 2014-08-11 19:18 irq
-r--r--r-- 1 root root 4096 2014-09-01 15:01 local_cpulist
-r--r--r-- 1 root root 4096 2014-08-11 20:18 local_cpus
-r--r--r-- 1 root root 4096 2014-09-01 15:01 modalias
-rw-r--r-- 1 root root 4096 2014-09-01 15:01 msi_bus
-r--r--r-- 1 root root 4096 2014-09-01 15:01 numa_node
drwxr-xr-x 3 root root    0 2014-08-11 19:18 pci_bus/
drwxr-xr-x 2 root root    0 2014-09-01 15:01 power/
--w--w---- 1 root root 4096 2014-09-01 15:01 remove
--w--w---- 1 root root 4096 2014-12-02 14:38 rescan
--w------- 1 root root 4096 2014-09-01 15:01 reset
-r--r--r-- 1 root root 4096 2014-08-11 19:18 resource
lrwxrwxrwx 1 root root    0 2014-08-11 19:18 subsystem -> ../../../bus/pci/
-r--r--r-- 1 root root 4096 2014-08-11 19:18 subsystem_device
-r--r--r-- 1 root root 4096 2014-08-11 19:18 subsystem_vendor
-rw-r--r-- 1 root root 4096 2014-08-11 19:18 uevent
-r--r--r-- 1 root root 4096 2014-08-11 19:18 vendor

[keflemin-vs-fpga-4 mem_test_hybrid_vc707_vivado_synplify(23)] ls -l /sys/devices/pci0000:00/0000:00:02.0/0000:04:00.0/driver/
total 0
lrwxrwxrwx 1 root root    0 2014-12-03 07:42 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:02.0/0000:04:00.0/
lrwxrwxrwx 1 root root    0 2014-12-03 07:42 0000:83:00.0 -> ../../../../devices/pci0000:80/0000:80:03.0/0000:83:00.0/
--w------- 1 root root 4096 2014-09-01 15:01 bind
lrwxrwxrwx 1 root root    0 2014-09-01 15:01 module -> ../../../../module/bluenoc/
--w------- 1 root root 4096 2014-09-01 15:01 new_id
--w------- 1 root root 4096 2014-09-01 15:01 remove_id
--w------- 1 root root 4096 2014-09-01 15:01 uevent
--w------- 1 root root 4096 2014-09-01 15:01 unbind

Note the presence of bluenoc in /sys/devices/.

Caching PCIE state

On some machines, especially desktop-class machines, reprogramming the FPGA results in the operating system “forgetting” key parameters, such as the PCIE memory mapping. To fix this, LEAP caches its known PCIE configuration and reloads this configuration each time the FPGA is reprogrammed.

To obtain the PCIE configuration, load a golden bitfile (see above), and then

%cat /sys/bus/pci/devices/${bus_id}/config > /usr/share/leap/state/${bus_id}_config

Some versions of Linux may use a different path, but this is the path used by Ubuntu. You can verify that you have the correct device by checking the driver file, which should point to bluenoc.

[keflemin-vs-lin64-2 ~(16)] ls -l /sys/bus/pci/devices/0000:01:00.0/driver/
total 0
lrwxrwxrwx 1 root root    0 Jun  9 19:35 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/
--w------- 1 root root 4096 Jun  9 19:35 bind
lrwxrwxrwx 1 root root    0 Jun  9 19:35 module -> ../../../../module/bluenoc/
--w------- 1 root root 4096 Jun  9 19:35 new_id
--w------- 1 root root 4096 Jun  9 19:35 remove_id
--w------- 1 root root 4096 Feb  3 12:37 uevent
--w------- 1 root root 4096 Jun  9 19:35 unbind

The LEAP supplied board script provides an example of reloading this file at run time.

Troubleshooting

This page contains some common error cases.

You can’t perform that action at this time.