TLP package

[beer-belly]

Hi Alex,

Can you provide the TLP packet format used in corundum. I don't know much about this. I feel that the TLP used here is different from what I checked online.Or can you provide a link to the TLP design specification you adopted

Thanks

[alexforencich]

Corundum uses standard PCIe TLPs as described in the PCIe specification, exactly like what you have posted. But, possibly the way they are transferred and the bit numbering is a little bit confusing. I really liked the way the Intel P-Tile PCIe IP core handles TLPs - TLP payload data transferred in little endian over a relatively standard streaming interface, TLP headers transferred over a fixed 128 bit big endian sideband signal - so the generic PCIe interface that Corundum uses is quite similar to this. This solves several thorny issues - first is it improves packing of TLP payloads with maximum-size payloads on wide interfaces, and it makes the variable-size big-endian-byte-order, little-endian-bit-order headers much easier to deal with. This interface can then be adapted to any PCIe IP core with a TLP-level interface using shims. On the big-endian sideband, the bits are simply numbered from 127 (fmt[2]) through 0 (PH[0] for a 64-bit memory read request).

[alexforencich]

Here is a link to the intel P-tile docs that has some diagrams for how the TLPs are packed: https://www.intel.com/content/www/us/en/programmable/documentation/htp1538516890095.html#wzr1554420581590

I'm doing basically the same thing for the headers and payloads, but the handshaking scheme is slightly different. And I'm not currently doing anything with segments, but I made an attempt to set things up so it would not be terribly complicated to implement segments in the future.

[alexforencich]

Oh, I suppose I should mention - if you're looking at an old version of Corundum before the generic PCIe interface was implemented, then you'll want to take a look at the Xilinx documentation: https://www.xilinx.com/products/intellectual-property/pcie4-ultrascale-plus.html#documentation . The Xilinx 7-series gen 3, UltraScale, and UltraScale+ PCIe IP cores all use a non-standard TLP format, which is described in the IP core documentation. In the current version, the DMA engines speak the generic version, then there are shims to adapt to the core-specific version, taking care of all of the packing and re-arranging.

[beer-belly]

Thank you for your reply. I saw the old version of corundum, and the TLP packaging method is consistent with that of the Xilinx 7-series Gen 3. I saw the same DMA design in netfpga before, so I don't understand it. I just downloaded the latest version of corundum, which is consistent with the picture sent yesterday.
I've been doing some SR-IOV related work recently, and there's a question about the requester ID. I want to know why it is all set to 0. Shouldn't it be automatically assigned by the host when it is powered on.

[alexforencich]

Yeah, I mentioned that because I saw you had a fork of Corundum that's over a year old, and if you were looking at that, it would definitely explain the discrepancy.

Requester ID is set to 0 at the moment for two reasons. The bus and device number components are set automatically by the hard IP core in endpoint mode, so you don't need to "turn around" the bus and device numbers form the hard IP core (this does need to be done for Intel devices, however). The function number is set to 0 as currently only function 0 is used. For supporting SR-IOV, I need to add a function number field to the DMA read/write request descriptors and pass this through to the requester ID field in the DMA engine, which would be a relatively straightforward change to implement.

There are several other things that would need to be done to support SR-IOV as well, including potentially adding tuser sidebands on the AXI lite register interface to carry the function number, adding state to the queue handling logic so queues can be associated with different functions, and additional logic to the queue handling logic and other control logic for function-level access control. And then support may be needed in the ingress pipeline and schedulers for performance isolation and queue selection (MAC/IP matching, etc.).

[beer-belly]

I have a simple idea. Put the queue initialization work in PF and allocate the queue according to the number of VF.
For example, I apply for 4 VF and 1024 queues. Allocate queues (0 ~ 255) to vf0 (port0) and queues (256 ~ 512) to VF1 (port1)... Then get the function_id through the top two bits of the queue address to DMA logic, and modify queue management, axil and other related configurations.
Do you think this plan is feasible?

[alexforencich]

TBH, I think this part is peanuts compared to what needs to be added to the ingress pipeline for queue selection and what not. So let's do it right. But that's a fine idea for a prototype/proof-of-concept. The key point is whatever is done to support SR-IOV, I want to make it as scalable and as flexible as possible (large number of VFs, any number of queues assigned to any VF). The PCIe hard IP cores on some FPGAs support up to something like 2K VFs (see Intel P-tile), it would be great if whatever solution we implement in Corundum can scale to (or at least close to) the full capability of the FPGA.

Also, another thing to keep in mind is that I'm going to throw out and rewrite all of the queue management and descriptor fetch logic relatively soon as part of variable-length descriptor support. So if we can pin down whatever functionality is needed for SR-IOV and roll that in during the rewrite, that would be optimal.

Another thing that we may need to think about is possibly provisions for the PF driver to talk to the VF drivers in some way. Not sure what that might look like, but presumably if these drivers have to interface with each other in some way, that will have to be done through the hardware itself and not through some other software channel. I'm not sure how other NICs handle this.

[alexforencich]

Also, I have a TODO item for SR-IOV support: #7 . If this is something you might be interested in working on, please hop on the slack workspace so we can coordinate; I just created a channel for SR-IOV. Possibly there are other people that would like to help with this effort as well.