Repository navigation
Home
Ever wondered how PCIe actually works at Hardware level? Most hardware enthusiasts talking about PCIe specifications and their speed and stuff and what really exists underneath?
Today, I present an open-source implementation of PCIe in RTL (Register Transfer Level): SmolPCIe.
A few open source implementation of PCIe do exist but as far as I now they are not meant for teaching and require a really good PCIe spec knowledge. I'll here, explain the spec from basics and how I implemented them.
SmolPCIe for now contains only the TLP Layer and the DLL implementation is still being worked on, I'll update the repo as soon as its done and verified (will explain more about layers in a sec).
First, we need to understand how multiple devices work together (like a CPU and a GPU) and how each one of them is accessed by the host (CPU or Root Complex).
The host might require to access a device's memory and might require a device to compute something for itself, so to communicate such requirements, protocols like PCI/PCIe exist.
Requirements like accessing the input/output (IO) space or having a certain range of memory address space allotted to a device or conveying errors or messages to the host or vice-versa are a few examples.
There are 3 types of address space which can be accessed by a PCIe device, namely, IO address space, memory address space and configuration address space.
The memory and IO address spaces are allotted to a device based on its requirements, like the device must ask the root or host to allocate it some of it for its operation.
The configuration address space holds the information about the device being connected. Like its functions, its memory and IO space requirements. The Configuration space can be read and written by root through Configuration accesses. In PCI and PCI-X devices, due to memory constrains, the Configuration space was not directly accessible by the host so the address to be accessed was (or still to support legacy or older devices) written on a certain memory location and in the next cycle, the data or be read or written was sent to another memory location.
The Configuration Space also holds the starting address of memory and IO spaces it has been allotted by the software (contained in BARS), bus number and other relevant information. To access a device for memory or IO request, the target address must fall in the range of the device.
The device tree (where a number of devices can be connected with each other and the root) supports switches (bridges for PCI and PCI-X) which allows multiple devices to be connected to the root. The path between a bridge or switch and the device is called a bus. In PCI/PCI-X more than 1 devices could be connected to a single bus. PCIe limited this to a single bus and single device model to increase the frequency and reduce complexity of accessing devices.
So, I'll give a brief introduction about PCI and PCI-X (the predecessors of PCIe).
PCI or Peripheral component interface was a parallel protocol which was used to exchange data between devices which supported this protocol. There were many handshake signals which were used in PCI which could cause problems and errors when anyone of them misfired at any time.
The same applies for PCI-X but had a few enhancements over PCI like increased clock frequency but came at the cost of reduced number of devices which can be connected to the same bus.
I'll not go in further details of these but one must understand why PCIe was introduced, PCI and PCI-X both shared the parallel bus model which limited the clock frequency and were error prone due to many handshake signals between the devices.
PCIe or Peripheral Component Interface Express completely abandoned the parallel bus model and introduced a serial communication model between PCIe devices. Serial communication means transmitting a single bit at a time. PCIe also introduced "messages" to compensate the removal of flags which were used for interrupts and other stuff.
Completions are sent back for appropriate requests which mention the status of the transaction (whether it was successful or had an error or failed).
So, PCIe has mainly 4 types of accesses, Memory Requests, IO requests (for legacy devices), Configuration Requests and Messages. They are further divided into reads and writes except for messages which can only be sent by the Root.
So how does PCIe work and support PCI and PCI-X devices? It introduces a packet-based protocol.
The data to read or written or messages to be read are sent in packets. The packet contains the information about the device the data is meant or to be read from, the data payload for writes, error detection bytes and start and end fields.
The formation of a packet is divided between 3 layers: Transaction Layer, Data Link Layer and Physical Layer. The transaction layer is connected to the core (which interfaces the software) and receives information about the device to be accessed and the data written if it's a write request. The data link layer is connected to the transaction and physical layers and mainly handles error detection and correction by request replays and tracks the current request. The physical layer encodes the information received from the above layers and attaches the start and end bits in PCIe 1.0 and 2.0 and only attaches the start bits for PCIe 3.0 onwards (how it does will be discussed later) and serializes the data for transmission.
The receiver does the exact opposite, the physical layer receives and collects the packet and deserializes them. The data link layer checks for errors in the packets and the transaction layer decodes the main packet.
As mentioned earlier, the transaction layer receives the type the of request and the device to accessed along with the data payload (if applicable) from the core. The TL then collects the device information like the device ID, bus to be targeted, request type etc in a header. The payload is attached along with the header. And option error correcting field is attached too If the target device supports it. This generates and packet called as Transaction Layer Packet (a TLP).
The data link layer collects or buffers multiple TLPs and attaches an LCRC field that is, a link cyclic redundancy code which checks for errors in the receiver side and a sequence number to each TLP.
It notifies the transmitter about the condition and status of the packets received at regular intervals to the transmitter.
The packet received from the DLL is this ready to be serialized. The physical layer used 8b/10b encoding or 128b/130b encoding for serialization (discussing such encoding techniques is out of scope this blog) and attaches some fields for the receiver to know when does a packet start and end.
In PCI/PCI-X multiple devices can be connected to the same bus and those devices could be accessed by the root through bridges. PCIe only allowed a single device in a bus and to connect multiple devices, switches were introduced. To route packets through switches, type 1 headers are used and the switch checks if the target device is connect to it, if it is then the header is converted into type 0 and the packet is delivered.
I'll not discuss more about switches and header details as it'll make this blog much more complicated.
My SmolPCIe implements the TL between a root and a device. It tests memory and IO space initialization using type 0 headers and then verifies requests sent to these address spaces.
The source contains a TLP decoder and encoder, Completer generator and decoder and top modules which wrap them up.
The details and implementation about the project are in this repo:
[repo link]
I'm currently working on the DLL and will update once its done.
Note that this is a very brief overview of what PCIe does and a lot of information was skipped or simplified for the reader to understand the basics. I recommend to read Mindshare's book on PCIe as it covers everything mentioned in much greater details and you can build a small prototype of PCIe too.