Skip to content
Fibre Channel / FICON HBA implemented on FPGA
Tcl SystemVerilog Verilog C Stata C++ Other
Branch: master
Clone or download
Latest commit 293b9de Nov 5, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
driver PCI is always little endian it seems Oct 29, 2019
images Fejkon picture Oct 6, 2019
ip RD research and usertx/rx adding Nov 3, 2019
utils Timing reports Oct 7, 2019
.gitignore FC State Machine (#14) Nov 1, 2019
Kconfig
LICENSE Create LICENSE Sep 29, 2019
Makefile FC State Machine (#14) Nov 1, 2019
README.md FC State Machine (#14) Nov 1, 2019
config.py
de5net.sdc FC State Machine (#14) Nov 1, 2019
de5net.tcl LED driver IP Nov 1, 2019
defconfig
fejkon.tcl LED driver IP Nov 1, 2019
fejkon_apply_config.tcl Dynamic identity based on active ports Nov 2, 2019
fejkon_pcie.tcl Update documentation and add PCIe instrumentation Oct 7, 2019
fejkon_sfp.tcl I2C SFP port working Oct 6, 2019
msim.do Start of testbench Oct 10, 2019
quartus.tcl FC State Machine (#14) Nov 1, 2019
syscon.tcl Correct last unkn. reg Nov 4, 2019
timing.tcl

README.md

fejkon

fejkon is a basic host bus adapter (HBA) for Fibre Channel (and thus, FICON) implemented on the DE5-Net board. The aim is to provide easy access to add or consume Fibre Channel services, such as FCP or FICON from an ordinary server.

DE5-Net running Fejkon

Fejkon, which was born out of the need for a fake FICON HBA, is a Swedish pun playing on the word "fejk" which translates "fake". It is pronounced similarly to the "FIC" in "FICON".

It is meant to be one of the possible access ports to fikonfarm enabling IBM mainframes to e.g. use Hercules disks natively.

Board

Target board right now is the DE5-Net from Terasic. They are available for $300 - $600 on eBay as of this writing and are capable of implementing 4x 8 Gbit/s Fibre Channel ports on a PCIe Gen 3 x8 port.

There are a few modifications to the board that are recommended.

  • Remove DDR3 SODIMMs, they are not needed and contribute to power drain
  • Add a 100 Ohm resistor to the stock cooler for a better noise level, cooling performance is fine anyway
  • Add a heatsink onto of the LTM4601V

DE5-Net Voltage Regulators

Configuring and Building

FPGA tooling can be a bit stuborn to work with, and if you do not have an FPGA background you might not want to use them.

Fejkon uses Kconfig menu configuration for configuring the board features, and can be configured and built like this:

$ pip3 install menuconfig  # Unless you have menuconfig installed already
$ make menuconfig
$ make

Developing

The recommended flow is using Quartus Platform Designer to make changes.

To build the qsys files needed simply execute make QPATH=/path/to/quartus in the root directory. Then use platform designer to edit fejkon.qsys. When you are done, export the system using "Export System as Platform Designer script (.tcl)" under the "File" menu. If you have updated any subsystems you need to this for those systems as well.

Finally review any changes to the *.tcl files and commit them if they look reasonable.

PCIe specification

The board uses vendor/device ID f1c0:0de5. Mnemonic is FICOn DE5-net.

The PCIe bus has four Base Address Registers (BARs).

BAR 0

PCIe DMA engine. TODO.

BAR 1

PCIe DMA engine. TODO

BAR 2

Accesses need to be 4 byte wide.

Addr Width Part Name Description
0x00000 2 Card Version The constant 0x0DE5
0x00002 1 Card Version Version of the Fejkon card
0x00003 1 Card Port options Number of ports
0x00004 4 Card Git hash Git hash of HDL built
0x00010 1 Card Temprature FPGA Core Temperature (1)
0x00020 4 Card Freq. Gauge PHY effective clock gauge
0x01000 1 Port 0 SFP Status SFP Status Word (2)
0x01040 64 Port 0 SFP Port I2C SFP I2C core (3)
0x02x00 ... Port 1 SFP Port
0x03x00 ... Port 2 SFP Port
0x04x00 ... Port 3 SFP Port
0x10000 512 Port 0 FC XCVR IP Fejkon FC XCVR Core (4)
0x10200 512 Port 0 TX XCVR Mgmt V-Series Transceiver PHY (5)
0x2xxxx ... Port 1 ...
0x3xxxx ... Port 2 ...
0x4xxxx ... Port 3 ...
  1. Details in section below
  2. Details in section below
  3. See "Intel FPGA Avalon I2C (Master) Core" in Embedded Peripherals IP User Guide
  4. Details in section below
  5. See "Custom PHY" in V-Series Transceiver PHY IP Core User Guide

Temperature

See Temperature decoding details in FPGA Temperature Sensor IP Core User Guide

Bit(s) Direction Description
7:0 Read only A/D conv. value
8 Read only A/D complete

SFP Port Status

Bit(s) Direction Description
0 Read only Present
1 Read only Loss of Signal
2 Read only TX Fault
3 Read/Write TX Disable
4:5 Read/Write Rate Select
6 Read/Write I2C Reset

Fejkon FC XCVR Core

Addr Width Name
0x00000 4 Port Status
Port Status
Bit(s) Description Values
0:3 Sync Status 0xf = fully synced
4:7 Pattern detect 0x1 = detected
8:11 Error detect >0 = error(s) detected
12:15 Disparity error >0 = error(s) detected
16 PLL Locked 1 = PLL locked

A fully operational port has status 0x1001f.

BAR 3

PCIe DMA engine descriptors. TODO

MSI Interrupts

Currently fejkon is using multiple MSI interrupts, not MSI-X. MSI-X is a bit more complicated to implement, and the benefits of MSI-X over MSI isn't very documented when you don't need the thousands interrupts that MSI-X offers.

During development it was discovered that e.g. QEMU does not do multiple MSI interrupts by default, and some kernel options are needed as well. The symptom of the platform not being setup correctly is that pci_alloc_irq_vectors(pcidev, 1, irqs, PCI_IRQ_ALL_TYPES); only returns one available vector instead of the requested number. There appears to be other people that have ran into the same issue. In the kernel this support for multiple MSI interrupts seems to be gated by MSI_FLAG_MULTI_PCI_MSI.

To enable the kernel support, enable CONFIG_IRQ_REMAP. On Intel, you should see mentions of DMAR in your dmesg. This is usually enabled by enabling VT-d. The equivalent on AMD seems to be just IOMMU support, but that has not been verified. PCIe hotplug has been known to cause issues, so you might want to disable that.

The configuration for QEMU is something like:

-machine q35,kernel-irqchip=split \
-device intel-iommu,intremap=on,device-iotlb=on \

Since the card only uses MSI interrupts, legacy interrupts are disabled.

So if you are designing something from scratch and have the option to use MSI or MSI-X, this information above should tell you that maybe using MSI-X will be easier. However, it is quite likely MSI-X has its own pitfalls.

Interrupts

Vector Description
0 Card status
1 Port 0 RX DMA
2 Port 0 TX DMA
3 Port 0 SFP status
4 Port 0 SFP I2C
5-8 Port 1 ...
9-12 Port 2 ...
13-16 Port 3 ...

Debugging

Use make syscon to launch Intel's System Console. It will guide you through the debug commands, but here is an example of reading the SFP data from port 1:

=> Fejkon system console initialized

 Master is available at $m

 E.g:
 - master_write_32 $m 0x000e0000 5
 - jtag_debug_reset_system $m
 - sfp 1

% sfp 1
 SFP status: 0x02
 SFP identifier: 0x03
 SFP ext. identifier: 0x04
 SFP connector: 0x07
 SFP vendor: 'JDS UNIPHASE    '
 SFP vendor PN: 'JSM-21S0AA1     '
 SFP vendor SN: 'F44939581059    '

The design has been compiled with instrumentation of some key data buses which you can inspect using make syscon by loading the fejkon.sof and using "Bus Analyzer (Beta)".

If you need to debug a particular component, then using Signal Tap is recommended.

Known Issues

These are some changes you might need to make to build under Quartus 19.1.

Si570

The Si570 on board appears to have the following specs:

Part Number: 570FAB000433DG
Product: Si570
Description: Differential/single-ended; I2C programmable XO; 10-1417 MHz
Frequency A: 100 MHz
I2C Address (Hex Format): 0
Format: LVDS
Supply Voltage: 2.5 V
OE Polarity: OE active high
Temperature Stability / Total Stability: 50 ppm / 61.5 ppm
Frequency Range: 10 - 810 MHz
Operating Temp Range (°C): -40 to +85

MSI limit to 32

For some reason the limit is set to 16 in the stock IP. Change it to 32.

--- /home/bluecmd/intelFPGA/19.1/ip/altera/altera_pcie/altera_pcie_hip_256_avmm/pcie_256_avmm_parameters.tcl.org	2019-10-07 22:32:09.828337496 +0200
+++ /home/bluecmd/intelFPGA/19.1/ip/altera/altera_pcie/altera_pcie_hip_256_avmm/pcie_256_avmm_parameters.tcl	2019-10-07 22:32:18.536266216 +0200
@@ -731,7 +731,7 @@
 
    add_parameter          msi_multi_message_capable_hwtcl string        "4"
    set_parameter_property msi_multi_message_capable_hwtcl DISPLAY_NAME "Number of MSI messages requested"
-   set_parameter_property msi_multi_message_capable_hwtcl ALLOWED_RANGES { "1" "2" "4" "8" "16"}
+   set_parameter_property msi_multi_message_capable_hwtcl ALLOWED_RANGES { "1" "2" "4" "8" "16" "32"}
    set_parameter_property msi_multi_message_capable_hwtcl GROUP $group_name
    set_parameter_property msi_multi_message_capable_hwtcl VISIBLE true
    set_parameter_property msi_multi_message_capable_hwtcl HDL_PARAMETER true

Confused Perl

The temporary LED driver will not generate without this pretty weird hack. This issue appeared in 19.1 and hopefully will be patched by Intel soon.

sudo mkdir -p /tools/perl/5.28.1/linux64/
sudo ln -sf /home/bluecmd/intelFPGA/19.1/quartus/linux64/perl/lib /tools/perl/5.28.1/linux64/

ModelSim Starter Edition

If you're using Intel's ModelSim Starter Edition you might run into issues like the one below:

bluecmd]$ make sim
mkdir -p libraries
(cd libraries; vlib work)
vlog -sv *.sv *.v
Model Technology ModelSim - Intel FPGA Edition vlog 2019.2 Compiler 2019.04 Apr 17 2019
Start time: 13:09:44 on Oct 24,2019
vlog -sv i2c_util.sv si570_ctrl.sv top_tb.sv i2c_master.v 
** Error: (vlog-66) Execution of vlib failed. Please check the error log for more details.
sh: /home/bluecmd/intelFPGA_pro/19.3/modelsim_ase/linuxpe/vlib: No such file or directory
End time: 13:09:44 on Oct 24,2019, Elapsed time: 0:00:00
Errors: 1, Warnings: 0

To fix that, run:

ln -sf ${HOME}/intelFPGA_pro/19.3/modelsim_ase/linuxaloem ${HOME}/intelFPGA_pro/19.3/modelsim_ase/linuxpe 

Possible future work

  • Integrate the MAX1619 sensor

The MAX1619 is an SMBus temperature sensor supported by the Linux kernel. It would allow for more temperature data. However, SMBus seems to be incompatible with the Intel I2C core, and no replacement core seems to be easily avaiable. This means writing an SMBus controller, QEMU model, Linux and driver for it.

An alternative is to re-use Terasic's NIOS drivers and create an enviromental processor that controls not only the temperature sensor but also the fan. However, this breaks the "keep it simple" methodology currently in use.

  • Over-temperature auto-shutdown

The Intel FPGAs do not feature a protective auto-shutdown as Xilinx FPGAs do. It would be useful to enter some kind of low-power mode if an over-temperature condition is detected.

Possibly integrated with the above work.

  • Replace Si570 with soft core CPU

Although writing the Si570 controller was useful, in the end it is less reliable and much more expensive, than using a soft core to implement the equivalent logic. Using a soft core like NIOS, RISC-V, or OpenRISC would make it possible to control clock and reset networks from the firmware of the card instead of wiring it up in Verilog. In the beginning I thought this would end up being simpler, and while it ended up being a very good learning experience if I had to do it again I would stick a soft core CPU on there and let it do temperature, fan, reset, and clock control.

You can’t perform that action at this time.