PCIe Express Generic Bridge enumeration #40720
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
superna9999
requested review from
dcpleung,
jhedberg and
nashif
as code owners
superna9999
force-pushed
the
narmstrong/pcie-bridge
branch
2 times, most recently
from
4430088 to
0de5c7e
Compare
|
Hey guys, Can you kindly approve this change ? I need this PCIE support for ARM |
Contributor
Author
|
Hi @dcpleung @tbursztyka would you have some time to review this PR ? |
superna9999
force-pushed
the
narmstrong/pcie-bridge
branch
2 times, most recently
from
90607e1 to
ff052db
Compare
Contributor
Author
|
@dcpleung do you think it's possible to have this merged before end of merge window close for 3.0.0 ? |
Member
|
Wish to have input from @jhedberg or @tbursztyka first since they will probably be impacted by this. |
jhedberg
approved these changes
Jan 18, 2022
Member
jhedberg
left a comment
jhedberg
left a comment
There was a problem hiding this comment.
Looks fine to me, but I think we still need to double-check that this doesn't break anything on physical (as opposed to just qemu) x86 PCIe-using boards like ehl_crb.
Member
|
Having this tested on actual hardware would be great. |
tbursztyka
approved these changes
Jan 19, 2022
Contributor
Author
|
@dcpleung @jhedberg this code only impacts configs with PCIE_CONTROLLER, so this won't impact x86 platforms. And since there is no non-x86 Zephyr enabled platforms with PCIe and bridges, I can only validate this with Qemu. |
When enumerating bridges, the allocated memory base and limit must be programmed in the bridge config registers. The base is programmed when starting enumerating the bridge and the limit is programmed when all endpoints of the bridge and sub-bridges are enumerated. Add a second callback to get the base of the next allocation in order to fill these base & limit values. Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
Implement callback to get allocation base similar to the pcie_ecam_region_allocate callback. Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
Add config registers defines for the Type 1 bridge endpoints. Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
The Type 1 endpoints has 2 BARs are the same position as the Type 0 BARS 0 & 1, so reuse the generic_pcie_ctrl_type0_enumerate_bars() for both types by passing the number of possible BARs on the endpoint. Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
In order to prepare support for bridges enumeration, split out the actual endpoint enumeration code out of the enumeration loop. Pass a skip_next_func boolean to indicate if the current endpoint is multifunction of not, to continue to next dev or next function. Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
This adds setup of Type 1 bridge endpoints in two steps, first when endpoint is detected and secondly when enumerating the next endpoint. First, the code configures the bus primary & secondary number and 0xff as subordinate to redirect all PCIe messages to this bus. Then memory & I/O base are programmed by getting the current allocation bases. Finally, now right away, we program the subordinate to the max bus number under the bridge, here the same, and the memory & I/O limit, here lower than the base. This doesn't make the bridge totally usable, enumeration would work bus not for nested bridges and BARs wouldn't be accessible. Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
In order to preserve stack, this replaces the single-bus enumeration
loop by a stack based non-recursive pcie hierarchy iteration.
Each stack entry contains a bridge bus enumeration state.
When a bridge endpoint is detected on the current bus, it is
configured and this new bus is pushed on top of the stack in
order to be enumerated at next loop.
When enumeration ends on the bus, the current bus state is
removed from the stack to continue enumeration on the previous
bus.
This enumeration affects a sequential bus number to each new
bus detected in the same order as Linux & U-Boot does.
In this hierarchy:
[0 1 2 ... 31]
| | |
EP | |
| |
[0 1 ... 31] [0 ... 31]
| | |
| EP |
| [0 ... 31]
[0 ... 31] |
| EP
EP
We will get the following BDFs enumeration order:
00:00.0 Endpoint
00:01.0 Bridge => Bus primary 0 secondary 1
01:00.0 Bridge => Bus primary 1 secondary 2
02:00.0 Endpoint
... Bus secondary 2 => subordinate 2
01:01.0 Endpoint
... Bus secondary 1 => subordinate 2
00:02.0 Bridge => Bus primary 0 secondary 3
03:00.0 Bridge => Bus primary 3 secondary 4
04:00.0 Endpoint
... Bus secondary 4 => subordinate 4
... Bus secondary 3 => subordinate 4
The gives the following primary/secondary/subordinate map:
Bus 0 [0 1 2 ... 31]
| | |
| [0:1->2] [0:3->4]
EP 00:00.0 | |
| |
Bus 1 [0 1 ... 31] Bus 3 [0 ... 31]
| | |
| EP 01:01.0 |
[1:2->2] [3:4->4]
| |
| Bus 4 [0 ... 31]
Bus 2 [0 ... 31] |
| EP 04:00.0
EP 02:00.0
Signed-off-by: Neil Armstrong <narmstrong@baylibre.com>
superna9999
force-pushed
the
narmstrong/pcie-bridge
branch
from
ff052db to
82c5f39
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds support for PCIe Bridge & Switches enumeration.
The actual code only supports PCIe devices on the primary root bus, also known as Root Controller Integrated Endpoints for internal PCIe devices without any external facing ports.
This adds support for Type 1 Bridge devices support and exploration of sub-busses when such devices are enumerated.
Usually bridge enumeration is done recursivelly, but this is not an option with a stack constrained OS like Zephyr.
So a non-recursive algorithm was written to enumerate all the possible bridges.
A new PCIe Controller API is needed to get the current allocation base address in order to configure the Bridges before and after enumeration.
Testing has been done with QEMU, but any Bridge or Switch needs MSI support, so was tested using a version with ITS support changes merged for next release.
A branch with ITS is available here:
https://github.com/superna9999/qemu/tree/gicv3-its-v8