[ title ]

vpcie: virtual PCIE devices


[ introduction ]

I am an engineer implementing software for PCIE based data acquisition
devices. The devices are designed and implemented in VHDL on XILINX FPGA
by other people working in the same group. I wanted a simple way to test
the drivers I am implementing against their VHDL design. The problem is
that setting up a real hardware device takes a lot of time (FPGA synthesis,
flashing large memories ...). Thus, I ended up with developing a virtual
environment consisting of QEMU, LINUX and GHDL glued alltogether by a
small TCP based protocol. It allows PCIE devices to be implemented as
standard userland processes, answering actual PCIE requests coming from
QEMU. It supports PCIE configuration headers, requests, memory readwrite
operations and MSI. Different abstractions are provided to simplify the
implementation of PCIE devices.


+--------+   +---------------------------+
| LINUX  |   |  PCIE device in C or VHDL |
| QEMU   |   |  PCIE layer               |
| PCIEFW |   |  PCIE_NET layer           |
+----+---+   +------------+--------------+
     |                    |
     +--------------------+
        PCIE forwarding
       TCP based protocol


[ installation ]

[[ QEMU ]]

Instructions to clone the QEMU repo are available at:
http://wiki.qemu.org/Main_Page

The VPCIE qemu/ directory contains patches to apply starting from a given
QEMU git commit. When several patches are provided, the order in which they
must be applied is given by their numeric prefix (ie. 00_pciefw.patch then
01_nvme.patch ...).

For instance, QEMU version 1.4.0 has commit:
03e94e39ce5259efdbdeefa1f249ddb499d57321

Thus, clone QEMU and position to the commit:
03e94e39ce5259efdbdeefa1f249ddb499d57321

Apply the PCIEFW patch:
< qemu/03e94e39ce5259efdbdeefa1f249ddb499d57321/00_pciefw.patch patch -p1

If wanted, apply the NVME patch:
< qemu/03e94e39ce5259efdbdeefa1f249ddb499d57321/01_nvme.patch patch -p1

Then you can build QEMU. A default configuration script is provided:
qemu/03e94e39ce5259efdbdeefa1f249ddb499d57321/do_configure.sh

[[ GHDL ]]

Currently, GHDL has to be compiled from sources. A working repository is at:
http://github.com/peteut/ghdl.git

For information, the revision used is:
42cc43913147b12bf5ae89c6fe64c3977b30018f

Follow the installation instructions available in:
ghdl/README


[ source tree ]

pcie: pcie related development file, both C and VHDL
main: currently contains a minimalistic GHDL main
sbone: a device written in VHDL and the corresponding LINUX driver
dma: simple dma engine, both in C and VHDL. refer to d_in_c/main_dma.c
ebone: how to use EBONE (http://www.ohwr.org/projects/e-bone). It may
not work with the latest EBONE release, as the endpoint changed a bit
since the time of this implementation. It gives a good example anyway.


[ implementing PCIE devices ]

Devices are implemented in userland processes. They communicate with
QEMU using TCP sockets. Currently, only a point to point connection
is possible, but this is to change in a future version. Basically, QEMU
forwards PCIE requests to the device userland process using the PCIEFW
and the corresponding small protocol. The protocol is minimalist and
may lack features (esp. identifiers). Improvements will be made as
needed. Currently, userland process have been implemented in C or VHDL
using the GHDL frontend.

A small PCIE runtime is provided for both C and VHDL. It consists of
several layers:
. callback based network messaging (pcie_net.c),
. PCIE low level requests (pcie.c),
. simple glue and packages for GHDL (pcie_xxx.vhdl).

These layers are made to simplify the development of simple PCIE devices,
so that one can focus on the hardware logic. They have some limitations,
but one can still choose not to use them and directly handle low level
PCIE and protocol details. Note that they are not


[ random thoughts ]

Not implemented with performance in mind, but it works quite well enough
for functional testing purposes. If performances become an issue, a shared
memory could be favored over the current TCP based transport layer if QEMU
and the virtual device are located on the same physical machine.

The VHDL pcie physical layer should be nearer than the XILINX one. Ideally,
it should implement the same interface.


[ acknowledgments ]

This work is being developed on my time at ESRF (ISDD division), in the
context of a fast data acquistion project financed by the CRISP wp12. More
info here:
http://www.esrf.eu
http://www.crisp-fp7.eu/research-programme

This work heavily relies on opensource projects, especially:
http://wiki.qemu.org/Main_Page
http://ghdl.free.fr
http://ygdes.com/GHDL

Keith Busch (keith.busch@gmail.com) for NVME support in QEMU


[ contact ]

fabien.lementec@gmail.com (texane@gmail.com works too)


[ license ]

This software is distributed under the GPLv3 terms:
https://www.gnu.org/licenses/gpl-3.0.fr.html