ACPI PCIe doesn't work with most devices

[valpackett]

Under both FreeBSD -CURRENT and Fedora 30 with acpi=force, the only device that was recognized by the OS is the ASMedia SATA controller (1B21:0612). All the other things I've tried — Intel igb NIC, Mellanox CX2 NIC, LSI SAS HBA, AMD Radeon GPU — are completely invisible to the OS in ACPI mode (adding debug prints to FreeBSD revealed that vendor/device ID registers come out as ffff).

Linux with devicetree (with host-generic) does recognize the igb NIC.

EFI pci command output: https://gist.github.com/myfreeweb/8b09f1c93ee9572aef01513ba9bf756f — only the assmedia card shows a non-zero command register, and for some reason it's a "Legacy PCI Express Endpoint".

I've tried different firmware: my own upstream build, both upstream and 18.09.4 from wiki, same results.

@semihalf-wojtas-marcin

[valpackett]

Aha! it is the ECAM shift (?)

After changing

#define PCI_ECAM_BASE      0xE0008000

to

#define PCI_ECAM_BASE      0xE0000000

(and PCI_BUS_MAX to 0xfe, idk if that's required)

after changing that, it's recongized, though there are problems with interrupts and stuff, and so far it seems that on the NIC, only TX works but not RX (because of the interrupt? UPD yes, I fixed MSI-X and it works!! yay)

igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1000-0x101f mem 0xc0c20000-0xc0c3ffff,0xc0400000-0xc07fffff,0xc0c44000-0xc0c47fff at device 0.0 on pci0
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: failed to allocate 5 MSI-X vectors, err: 3
igb0: Using a Legacy interrupt
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> mem 0xc0c00000-0xc0c1ffff,0xc0000000-0xc03fffff,0xc0c40000-0xc0c43fff at device 0.1 on pci0
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: failed to allocate 5 MSI-X vectors, err: 3
igb1: Using a Legacy interrupt

[wojtas-marcin]

Hi Greg,

Indeed it's the ecam shift :/ Because the HW multiplies device 0 on each 32 positions of bus 0, a special handling must be added. This is not a problem in DT world (please check "marvell,armada8k-pcie-ecam" compatible in Linux drivers/pci/controller/pci-host-generic.c). However with ACPI, community does not allow to do it. As a result we are able to use endpoints that are enumerated on bdf 0:0:0 in the OS. This is problem for some cards like you mentioned.

Regarding above in FreeBSD:

• I think adding handling for "marvell,armada8k-pcie-ecam" in DT will be super-easy
• With ACPI - I think there are 3 options:
  • add a build flag in edk2 to set config space base and bus count without
  • do above depending on edk2 variable - better, as we'd have single image for all OSs, however more difficult to implement at the moment
  • play with CID/HID like was done with UART recently and override shift/bus count in OS (unfortunately both fields seems to have meaningful values)

[valpackett]

What's interesting is that only the "legacy endpoint" card was successfully duplicated by the HW into the last position (I assume the shift is meant to only expose the last position to the OS?) and literally all other cards I have weren't duplicated..

unfortunately both fields seems to have meaningful values

You mean the current HID and CID?

Name (_CID, "PNP0A03" /* PCI Bus */)  // _CID: Compatible ID

..yeah, I doubt that any OS would support legacy PCI on aarch64 but not Express :D