ntb_tool cannot dma_alloc_coherent with CMA (over switchtec)

[glimchb]

I also posted question to https://groups.google.com/forum/#!topic/linux-ntb/4zFt8GkykNg

I’m trying to enlarge buffers I use in ntb_tool by using cma:

$ dmesg | grep cma

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable
[    0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable
[    0.000000] Memory: 6057140K/1073389456K available (10252K kernel code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K reserved, 33554432K cma-reserved)

$ grep -i cma /proc/meminfo

CmaTotal:       33554432 kB
CmaFree:        33501788 kB

After loading drivers:

modprobe ntb
insmod /var/tmp/switchtec-kernel/switchtec.ko dyndbg=+pmfl
insmod /var/tmp/switchtec-kernel/ntb_hw_switchtec.ko  max_mw_size=0x400000000 dyndbg=+pmfl
insmod /var/tmp/linux-4.14.17/drivers/ntb/test/ntb_tool.ko dyndbg=+pmfl

I allocate memory (uses dma_alloc_coherent):

echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1

I get this stack:

[  313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 20971520 bytes)
[  313.224546] swiotlb: coherent allocation failed for device 0000:3b:00.1 size=20971520
[  313.224550] Call Trace:
[  313.224560]  dump_stack+0x5c/0x83
[  313.224567]  swiotlb_alloc_coherent+0x14c/0x160
[  313.224572]  tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool]
[  313.224578]  ? perf_trace_mm_compaction_migratepages+0x5a/0x120
[  313.224584]  full_proxy_write+0x5a/0xa0
[  313.224589]  __vfs_write+0x33/0x170
[  313.224594]  ? handle_mm_fault+0xc4/0x1d0
[  313.224599]  ? __audit_syscall_entry+0xaf/0x100
[  313.224601]  vfs_write+0xad/0x1a0
[  313.224603]  SyS_write+0x52/0xc0
[  313.224608]  do_syscall_64+0x61/0x1a0
[  313.224614]  entry_SYSCALL64_slow_path+0x25/0x25
[  313.224616] RIP: 0033:0x7f79ac50e840
[  313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 00007f79ac50e840
[  313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: 0000000000000001
[  313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: 00007f79ace27740
[  313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: 00007f79ac7e2400
[  313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: 0000000000000000

Any help please?

[dmeyergio]

Caveat: This may or may not be helpful. Hopefully it's not a complete non sequitur. All of the following is within the context of running the machine with the IOMMU DISABLED. I ran into difficulty with dma_alloc_coherent with my system that was running with i7 CPUs and an PM8546 switch. I was having the difficulty in spite of using menuconfig to reserve large amounts of CMA memory. In particular, I could have menuconfig settings like this, and still not be able to successfully allocate more than 4 MiB. CONFIG_CMA=y # CONFIG_CMA_DEBUG is not set # CONFIG_CMA_DEBUGFS is not set CONFIG_CMA_AREAS=10 CONFIG_DMA_CMA=y CONFIG_CMA_SIZE_MBYTES=128 CONFIG_CMA_SIZE_SEL_MBYTES=y # CONFIG_CMA_SIZE_SEL_PERCENTAGE is not set # CONFIG_CMA_SIZE_SEL_MIN is not set # CONFIG_CMA_SIZE_SEL_MAX is not set CONFIG_CMA_ALIGNMENT=10 Increasing CMA_SIZE to 256 or 1024 similarly failed. Even 4096 MBYTES, ALIGNMENT 16, and AREAS 100 failed. And then I looked at the kernel code... *gasp* First, I figured out how to get CMA debugfs and debug messages turned on (via menuconfig) and saw that cma_alloc() was actually seeming to return a pointer to the allocated page number: [ 4089.779852] cma: cma_alloc(cma 00000000aa0aa405, count 2048, align 11) [ 4089.780243] cma: cma_alloc(): returned 000000004fef7868 As I followed the kernel flow for cma_alloc() and other tidbits, I came to see that the result for cma_alloc() is eventually checked against the dma_mask value. This made me think that the address associated with the page that cma_alloc() returned was above 32 bits. As an initial test, I explicitly set the DMA mask to 32 bits, with the following call (I put it in ntb_hw_switchtec during the init flow, prior to the first coherent alloc): dma_set_coherent_mask(&stdev->pdev->dev, 0xFFFFFFFF) And the allocations above 4 MiB still failed. The Microsemi switch supports 64-bit DMA, so my next experiment set the mask for 48 bits: dma_set_coherent_mask(&stdev->pdev->dev, 0xFFFFFFFFFFFFULL) This worked fine. I was able to allocated 8 MiB, 16 MiB, and beyond. I was also able to lower the menuconfig values back down to smaller values and still get the allocations that I needed. The way that I am using the hardware it is valid for me to set the DMA mask all the way up to 64 bits (I'm performing memory accesses with 64-bit addresses). If this seems relevant for what you're experiencing, my suggestion would be to set the coherent mask appropriately, along with the menuconfig values, and give it a try. Hopefully this was at least slightly helpful for you. Blessings, Doug

…

On Thu, May 31, 2018 at 9:40 AM, glimchb ***@***.***> wrote: I also posted question to https://groups.google.com/forum/#!topic/linux-ntb/4zFt8GkykNg I’m trying to enlarge buffers I use in ntb_tool by using cma: $ dmesg | grep cma [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable [ 0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable [ 0.000000] Memory: 6057140K/1073389456K available (10252K kernel code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K reserved, 33554432K cma-reserved) grep -i cma /proc/meminfo CmaTotal: 33554432 kB CmaFree: 33501788 kB After loading drivers I allocate memory (uses dma_alloc_coherent): echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1 I get this stack: [ 313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 20971520 bytes) [ 313.224546] swiotlb: coherent allocation failed for device 0000:3b:00.1 size=20971520 [ 313.224550] Call Trace: [ 313.224560] dump_stack+0x5c/0x83 [ 313.224567] swiotlb_alloc_coherent+0x14c/0x160 [ 313.224572] tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool] [ 313.224578] ? perf_trace_mm_compaction_migratepages+0x5a/0x120 [ 313.224584] full_proxy_write+0x5a/0xa0 [ 313.224589] __vfs_write+0x33/0x170 [ 313.224594] ? handle_mm_fault+0xc4/0x1d0 [ 313.224599] ? __audit_syscall_entry+0xaf/0x100 [ 313.224601] vfs_write+0xad/0x1a0 [ 313.224603] SyS_write+0x52/0xc0 [ 313.224608] do_syscall_64+0x61/0x1a0 [ 313.224614] entry_SYSCALL64_slow_path+0x25/0x25 [ 313.224616] RIP: 0033:0x7f79ac50e840 [ 313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 00007f79ac50e840 [ 313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: 0000000000000001 [ 313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: 00007f79ace27740 [ 313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: 00007f79ac7e2400 [ 313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: 0000000000000000 Any help please? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[lsgunth]

On 31/05/18 01:41 PM, dmeyergio wrote: dma_set_coherent_mask(&stdev->pdev->dev, 0xFFFFFFFF) And the allocations above 4 MiB still failed. The Microsemi switch supports 64-bit DMA, so my next experiment set the mask for 48 bits: dma_set_coherent_mask(&stdev->pdev->dev, 0xFFFFFFFFFFFFULL) This worked fine. I was able to allocated 8 MiB, 16 MiB, and beyond.

Ah, thanks Doug. Any chance you can create a patch? I suspect using all 64 bits would be fine. It looks like PCI defaults to 32 for some reason.

CONFIG_CMA_ALIGNMENT=10

Also, you should be careful with this setting. This tells the CMA it can return unaligned buffers for buffers greater than PAGE_SIZE * 2^n. However, the memory windows for switchtec must be aligned! So things may fail if it's set smaller than the size you are allocating. Logan

[glimchb]

BTW, you can use this macro:

dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64));

And thanks a lot Doug from me as well, you explanation was excellent!

…

-- Best regards, Boris

[glimchb]

After trying this, see #15
I get new error that memory address returned by dma_alloc_coherent is not aligned to it's size:

[216492.164593] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec0: enabling link
[216492.164597] ntb_tool:tool_link_event:170: ntb_tool 0000:3b:00.1: link is down speed 0 width 0
[216492.164657] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec1: enabling link
[216492.164660] ntb_tool:tool_link_event:170: ntb_tool 0000:86:00.1: link is down speed 0 width 0
[216492.192305] ntb_hw_switchtec:switchtec_ntb_mw_set_trans:319: switchtec switchtec0: MW 1: part 0 addr 0x000000f883100000 size 0x0000000001400000
[216492.192306] switchtec switchtec0: ERROR: Memory window address is not aligned to it's size!

my config file is like this:

CONFIG_CMA=y
# CONFIG_CMA_DEBUG is not set
CONFIG_CMA_DEBUGFS=y
CONFIG_CMA_AREAS=7
CONFIG_DMA_CMA=y
CONFIG_CMA_SIZE_MBYTES=32
CONFIG_CMA_SIZE_SEL_MBYTES=y
# CONFIG_CMA_SIZE_SEL_PERCENTAGE is not set
# CONFIG_CMA_SIZE_SEL_MIN is not set
# CONFIG_CMA_SIZE_SEL_MAX is not set
CONFIG_CMA_ALIGNMENT=8

Probably the issue is with CMA_ALIGNMENT and CMA_AREAS, right?

Found an answer here: https://github.com/davejiang/linux/wiki/Intel-NTB-Startup-Guide#cma-continous-memory-allocator:

The alignment is page order size, 4096 * 2^N. So an 8M alignment would be 4096 * 2^11.

[lsgunth]

Yes, that is exactly the issue you will see with a low CMA_ALIGNMENT.

Logan

[glimchb]

Find another related commit (putting it here for documentation purposes only)
torvalds/linux@417cf39cfea9c680aa7c278c8d8a0

[glimchb]

Looks like this code never made upstream kernel.
https://elixir.bootlin.com/linux/latest/source/drivers/pci/switch/switchtec.c#L1244

Missing same chunk of masks as in other drivers according to this commit: torvalds/linux@417cf39cfea9c680aa7c278c8d8a0

For example: https://elixir.bootlin.com/linux/latest/source/drivers/ntb/hw/idt/ntb_hw_idt.c#L2429

pci_set_master(pdev);

rc = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
if (rc) {
	rc = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
	if (rc)
		goto err_dma_mask;
	dev_warn(&pdev->dev, "Cannot DMA highmem\n");
}

rc = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
if (rc) {
	rc = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
	if (rc)
		goto err_dma_mask;
	dev_warn(&pdev->dev, "Cannot DMA consistent highmem\n");
}
rc = dma_coerce_mask_and_coherent(&ndev->ntb.dev,
				  dma_get_mask(&pdev->dev));
if (rc)
	goto err_dma_mask;

Can you please facilitate pushing to upstream via NTB or PCI mailing lists?

Thanks,

[lsgunth]

Microsemi is still working on upstreaming a bunch of changes, not sure when it will all land though.

[glimchb]

Hi Boris,
I am happy to let you know that upstreaming process for dma_mask has begun:
https://lkml.org/lkml/2018/11/15/458
Thanks, Paul