Transferring very large DMA buffers

[McGr]

Hi Jeremy,
I'm trying to DMA very large buffers (512M byte if I can) into memory. I seem to run into a memory problem for sg_alloc_table if I try to create a DMA buffer > 1Mbyte. I believe I just have too many entries in the sg table.
My AXI DMA core is configured with 23bit addressing for the scatter/gather engine which should allow for individual DMA blocks of up to 8MByte each, so it should be possible to describe the 512 MByte buffer with just 64 chained entries. Looking at the ezdma driver it seems like you are allocating blocks of 4096 bytes for each of the scatterlist entries. Is there a way to allocate 8Mbyte blocks for each of the scatterlist entries so I can reduce the number of them?

Thanks Mike.

[jeremytrimble]

Mike,

I think the reasoning for this is because there's no guarantee that the userspace buffer (which appears to be contiguous inside the userspace process's virtual memory) is actually physically contiguous. In fact, if you have a 512 Mbyte buffer I'm almost sure that it isn't physically-contiguous -- on Zynq systems I've worked with, 4 MB was the maximum physically-contiguous allocation that was possible without special accomodations (e.g. the CMA -- contiguous memory allocator).

4096 bytes is the size of a single page (the smallest unit of contiguous allocation), so allocating buf_size/4096 is kind of assuming worst-case (every page is physically discontiguous). I suspect with a little bit of logic after calling get_user_pages() it would be possible to compute exactly how many scatterlist entries are required (one for each contiguous section), and only allocate that many entries in sg_alloc_table().

[Livius90]

Hi Jeremy,
I'm trying to DMA very large buffers (512M byte if I can) into memory. I seem to run into a memory problem for sg_alloc_table if I try to create a DMA buffer > 1Mbyte. I believe I just have too many entries in the sg table.
My AXI DMA core is configured with 23bit addressing for the scatter/gather engine which should allow for individual DMA blocks of up to 8MByte each, so it should be possible to describe the 512 MByte buffer with just 64 chained entries. Looking at the ezdma driver it seems like you are allocating blocks of 4096 bytes for each of the scatterlist entries. Is there a way to allocate 8Mbyte blocks for each of the scatterlist entries so I can reduce the number of them?

Thanks Mike.

Root cause of your problem is that there is a hard-coded limitation in the Linux kernel driver xilinx_dma.c.

https://github.com/Xilinx/linux-xlnx/blob/xilinx-v2020.2/drivers/dma/xilinx/xilinx_dma.c#L181

#define XILINX_DMA_NUM_DESCS		255

You need to make a patch for Linux kernel and increase this max desc num, for example to 8*255 or more for your 8MB demand and it should works.