We observed a memory coherent issue on the vfio passthroughed device in TDX with host/guest TEE configuration. So please share some suggestions how I move on. Thanks in advance.
On our device, there is a IH(Interrupt Handler) hardware ring where our device will write interrupt entry information into. The ring is allocated via "dma_alloc_coherent" in kernel, and returned dma address is programmed to IH ring registers for its access. Our device is vfio passthroughed to CVM.
- In the first round driver attach in TDX, the IH ring is working as expected. When hardware interrupts arrive, via the irq handler, we are able to read back the correct interrupt entry data in the IH buffer allocated by dma_alloc_coherent, and start to handle it. Also, at the same time, we have an IH entry counter register from the device, which tells us IH writing to IH buffer is working as expected.
- Next we detach our device, and the IH buffer is released accordingly. CVM is working as usual.
- Attach our device again and with same dma_alloc_coherent calling for IH buffer, the issue occurs. Once hardware interrupt arrives, we observe the data read back from the IH buffer is always 0, while IH counter value from the device register is not 0, which means real device interrupt happens. Also, I polled the data for more than 5 minutes to ensure no race issue between MSI signals and interrupt data, but unluckily, the data read back is still 0. Does it look like cpu always reads from cached memory?
- When creating a non-CC guest VM with the same device passthroughed on the same TEE host platform, no such issue happens in cycles of driver attach/detach.
Also, I tried below methods as below, but the issue persists:
- Instead of dma_alloc_coherent, I tried a combination of dma_alloc_pages and vmap to get an uncached cpu mapping for read, but no luck to fix this issue.
- Flush the memory via clflush_cache_range before IH hardware accesses.
Moreover, when issue happens, there is no outstanding error log/hint in host platform. Please share some thoughts on this.
We observed a memory coherent issue on the vfio passthroughed device in TDX with host/guest TEE configuration. So please share some suggestions how I move on. Thanks in advance.
On our device, there is a IH(Interrupt Handler) hardware ring where our device will write interrupt entry information into. The ring is allocated via "dma_alloc_coherent" in kernel, and returned dma address is programmed to IH ring registers for its access. Our device is vfio passthroughed to CVM.
Also, I tried below methods as below, but the issue persists:
Moreover, when issue happens, there is no outstanding error log/hint in host platform. Please share some thoughts on this.