Pi4 Arm64 USB Device descriptor errors

Getting USB device descriptor errors on boot of arm64 kernel using rpi-4.19.y branch.
All usb ports unusable and unable to detect devices.
Below is the dmesg log.

[    0.206616] usbcore: registered new interface driver usbfs
[    0.206663] usbcore: registered new interface driver hub
[    0.206761] usbcore: registered new device driver usb
[    0.451596] usbcore: registered new interface driver r8152
[    0.451652] usbcore: registered new interface driver lan78xx
[    0.451705] usbcore: registered new interface driver asix
[    0.451752] usbcore: registered new interface driver smsc95xx
[    0.455294] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19
[    0.455307] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.455318] usb usb1: Product: xHCI Host Controller
[    0.455328] usb usb1: Manufacturer: Linux 4.19.58-v8+ xhci-hcd
[    0.455338] usb usb1: SerialNumber: 0000:01:00.0
[    0.456670] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.19
[    0.456682] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.456692] usb usb2: Product: xHCI Host Controller
[    0.456702] usb usb2: Manufacturer: Linux 4.19.58-v8+ xhci-hcd
[    0.456712] usb usb2: SerialNumber: 0000:01:00.0
[    0.483608] dwc_otg fe980000.usb: base=(____ptrval____)
[    0.790156] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[    0.889508] dwc_otg fe980000.usb: DWC OTG Controller
[    0.889533] dwc_otg fe980000.usb: new USB bus registered, assigned bus number 3
[    0.889574] dwc_otg fe980000.usb: irq 21, io mem 0x00000000
[    0.889885] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19
[    0.889897] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.889908] usb usb3: Product: DWC OTG Controller
[    0.889918] usb usb3: Manufacturer: Linux 4.19.58-v8+ dwc_otg_hcd
[    0.889928] usb usb3: SerialNumber: fe980000.usb
[    0.891408] usbcore: registered new interface driver usb-storage
[    0.897443] usbcore: registered new interface driver usbhid
[    0.897451] usbhid: USB HID core driver
[    0.918449] usb 1-1: device descriptor read/64, error 18
[    1.154449] usb 1-1: device descriptor read/64, error 18
[    1.390200] usb 1-1: new high-speed USB device number 3 using xhci_hcd
[    1.518444] usb 1-1: device descriptor read/64, error 18
[    1.754502] usb 1-1: device descriptor read/64, error 18
[    1.862462] usb usb1-port1: attempt power cycle
[    2.518209] usb 1-1: new high-speed USB device number 4 using xhci_hcd
[    2.538978] usb 1-1: device descriptor read/8, error -61
[    2.666644] usb 1-1: device descriptor read/8, error -61
[    2.904871] usb 1-1: new high-speed USB device number 5 using xhci_hcd
[    2.922966] usb 1-1: device descriptor read/8, error -61
[    3.051140] usb 1-1: device descriptor read/8, error -61
[    3.158510] usb usb1-port1: unable to enumerate USB device
[    3.383972] usbcore: registered new interface driver brcmfmac

[Strit]

It's caused by d5dc848.
It should be reverted.

[agherzan]

@Strit that commit is a step forward - I see that is not fixing everything but at least some devices are working with it. You still have the old option to limit ram and things will get back on track.
@pelwell do you have any leads on this? I'm directly interested in making this work.

[pelwell]

It sounds like the DMA bounce buffer support needed by the PCIe block is either not hooked up or not working.

[agherzan]

How does that come into play given that we limit now the dma to 1G?

[pelwell]

My understanding (and this whole area is a bit murky and complicated) is that the DMA zone limits where memory for dma_*_alloc is placed.

PCIe is a bus master, so it is effectively also a DMA controller, but it can only get to the bottom 3GB of RAM. If there is data in the top 1GB to be read from our written to USB then something has to arrange to copy to or from an intermediate buffer before or after the transfer. This is a common requirement, and yet bounce buffers don't seem to have a common implementation - individual subsystems seem to do it differently.

[agherzan]

That would explain it indeed. I found the implementation of the bounce buffers for the pcie controller and saw that the threshold is set to 3G. This basically means that we need to set this to 1G as well, right? At least that is how it sounds to me. But I'm confused how come this works on 32bit as we similarly only limit the zone and the driver bounce threshold default to the same 3G. There is something I still miss in this story.

[pelwell]

[ Again - this is just what I think is going on, I can't guarantee it is correct. ]

Think about what happens when an application needs to send some data to USB: a user-space pointer will get passed to a system call. At this point the driver or framework has a choice - whether to copy the data into an intermediate buffer for transmission or whether to attempt to use it in-place. The DMA Zone helps with the first case because it can guarantee that the intermediate buffer is in a DMA-safe location. It doesn't help (directly) with the second case because those user pages could be anywhere in RAM, so the PCIe driver hooks into the DMA methods for the USB device and (transparently to the user) performs a copy at the point the buffer is mapped for DMA, returning the address of the bounce buffer rather than the user pages.

[agherzan]

That was my understanding as well. Thanks for clarifying @pelwell . So in this case we need to make sure the threshold is indeed 1G. We need to allocate bounce buffers for everything that goes beyond this limit. The first 1G is marked as DMA safe-zone so everything else needs to go through bounce buffers.

[pelwell]

That's still not correct. For PCIe (which means for XHCI USB on the Pi 4), any memory within the first 3GB would be safe to use as a bounce buffer for anything in the last 1GB on a 4GB Pi4. PCIe is a DMA controller, it doesn't need to use the system DMA controller, so the 1GB limit does not apply. However there seems to be only one global DMA_ZONE configuration, so it must be set to the largest value safe for all DMA controllers.

[agherzan]

But that exactly what we do. The DMA_ZONE configuration applies to all DMA controllers (at least it should) and we set it to the first 1GB. The system controller can address that and also the PCIe should work with that limit.

[pelwell]

What I am saying is that a 1GB limit is unnecessarily pessimistic for PCIe as a bounce buffer threshold. Yes it's fine to allocate bounce buffers from memory below 1GB for PCIe as well, but don't require the use of bounce buffers for USB/PCIe accesses to RAM above 1GB - the threshold should be 3GB, or performance will suffer.

[agherzan]

Got it now. It makes sense. Thanks for expanding it.

[agherzan]

@pelwell I think I "found" the problem. The bounce buffers are only compiled when CONFIG_ARM is enabled. They don't compile on arm64.

[agherzan]

I have been bashing my head for a while now and it seems to not be as easy as I thought. The current implementation uses a lot of arm_dma* identifiers (and not only). All these are arm specific. Does anyone know in depth this implementation? arm64 defines a couple of dma_ips structures but I'm not sure what is the way forward here as I'm not familiar with this part of the kernel.