increase 'DMA buffer size' to 'swiotlb=8192' by default

[adrelanos]

As per WiFi Realtek RTL8191SEvB Issue and Solution, I suggest increase DMA buffer size to swiotlb=8192 by default. Unless something speaks against that?

Maybe a duplicate of #365?

#365 (comment)

This is really a Xen-specific thing, not ours, closing.

This may be true, but I would argue, that this is a grave usability issue for new users and therefore also Qubes responsibility to deliver a functioning product.

So...

• Has this issue been reported upstream?
• What about increasing the default?

[marmarek]

Do you have a hardware which needs swiotlb=8192? Can you try to omit
that value completely (but only that, iommu=soft keep there)?
The default is 64MB. This means that VM will need that much physically
coherent memory to start. Which can be hard to find after some system
up time (mostly because dynamic memory management, which cause some
fragmentation).
Theoretically the kernel will try to allocate smaller buffer when the
original fails, but that wasn't working well in the past. Can you try
it? If there is no drawbacks on current kernel versions, I think it is
better way than static 16MB (yes, swiotlb=8192 means 16MB...).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Do you have a hardware which needs swiotlb=8192? Can you try to omit
that value completely (but only that, iommu=soft keep there)?
The default is 64MB. This means that VM will need that much physically
coherent memory to start. Which can be hard to find after some system
up time (mostly because dynamic memory management, which cause some
fragmentation).
Theoretically the kernel will try to allocate smaller buffer when the
original fails, but that wasn't working well in the past. Can you try
it? If there is no drawbacks on current kernel versions, I think it is
better way than static 16MB (yes, swiotlb=8192 means 16MB...).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

Do you have a hardware which needs swiotlb=8192?

I got hardware that does not work with the Qubes Q3 RC1 default. See,
WiFi Realtek RTL8191SEvB Issue and Solution:
https://groups.google.com/forum/#!msg/qubes-users/kMGTSwP72aU/BRvTJEbkGBQJ

Can you try to omit that value completely (but only that, iommu=soft keep there)?

Works for me:
qvm-prefs -s sys-net kernelopts "nopat iommu=soft swiotlb=soft"

Works also for me:
qvm-prefs -s sys-net kernelopts "nopat iommu=soft"

[adrelanos]

Do you have a hardware which needs swiotlb=8192?

I got hardware that does not work with the Qubes Q3 RC1 default. See,
WiFi Realtek RTL8191SEvB Issue and Solution:
https://groups.google.com/forum/#!msg/qubes-users/kMGTSwP72aU/BRvTJEbkGBQJ

Can you try to omit that value completely (but only that, iommu=soft keep there)?

Works for me:
qvm-prefs -s sys-net kernelopts "nopat iommu=soft swiotlb=soft"

Works also for me:
qvm-prefs -s sys-net kernelopts "nopat iommu=soft"

[marmarek]

Getting rid of swiotlb=... is not a good idea. When initial swiotlb allocation fails, kernel crashes (panic) before console is initialized, so its really hard to know what actually has happened.
The crash looks like this:

Kernel command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH 3 nopat iommu=soft
PID hash table entries: 2048 (order: 2, 16384 bytes)
xen:swiotlb_xen: Lowering to 32MB
xen:swiotlb_xen: Lowering to 16MB
xen:swiotlb_xen: Lowering to 8MB
fxen:swiotlb_xen: Failed to get contiguous memory for DMA from Xen!
You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
Kernel panic - not syncing: Failed to get contiguous memory for DMA from Xen!
You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
CPU: 0 PID: 0 Comm: swapper Not tainted 3.18.16-3.pvops.qubes.x86_64 #1
 0000000000000000 34c976dd45d0c184 ffffffff81c03d88 ffffffff81752829
 0000000000000000 ffffffff81abd3e5 ffffffff81c03e08 ffffffff8174aafa
 0000000000000018 ffffffff81c03e18 ffffffff81c03db8 34c976dd45d0c184
Call Trace:
 [<ffffffff81752829>] dump_stack+0x46/0x58
 [<ffffffff8174aafa>] panic+0xd0/0x204
 [<ffffffff817451a5>] xen_swiotlb_init+0x415/0x4a0
 [<ffffffff81d53af9>] ? pci_swiotlb_detect_4gb+0x2c/0x2c
 [<ffffffff81d40ba9>] pci_xen_swiotlb_init+0x1c/0x2e
 [<ffffffff81d44474>] pci_iommu_alloc+0x5e/0x6e
 [<ffffffff81d54959>] mem_init+0x10/0x8f
 [<ffffffff81d39f05>] start_kernel+0x261/0x4b9
 [<ffffffff81d39a94>] ? set_init_arg+0x55/0x55
 [<ffffffff81d395ee>] x86_64_start_reservations+0x2a/0x2c
 [<ffffffff81d3cad3>] xen_start_kernel+0x599/0x5a5

Surprisingly a moment later, with swiotlb=8192 (so 16MB - which theoretically was tried), the netvm starts up.

[marmarek]

Getting rid of swiotlb=... is not a good idea. When initial swiotlb allocation fails, kernel crashes (panic) before console is initialized, so its really hard to know what actually has happened.
The crash looks like this:

Kernel command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH 3 nopat iommu=soft
PID hash table entries: 2048 (order: 2, 16384 bytes)
xen:swiotlb_xen: Lowering to 32MB
xen:swiotlb_xen: Lowering to 16MB
xen:swiotlb_xen: Lowering to 8MB
fxen:swiotlb_xen: Failed to get contiguous memory for DMA from Xen!
You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
Kernel panic - not syncing: Failed to get contiguous memory for DMA from Xen!
You either: don't have the permissions, do not have enough free memory under 4GB, or the hypervisor memory is too fragmented! (rc:-12)
CPU: 0 PID: 0 Comm: swapper Not tainted 3.18.16-3.pvops.qubes.x86_64 #1
 0000000000000000 34c976dd45d0c184 ffffffff81c03d88 ffffffff81752829
 0000000000000000 ffffffff81abd3e5 ffffffff81c03e08 ffffffff8174aafa
 0000000000000018 ffffffff81c03e18 ffffffff81c03db8 34c976dd45d0c184
Call Trace:
 [<ffffffff81752829>] dump_stack+0x46/0x58
 [<ffffffff8174aafa>] panic+0xd0/0x204
 [<ffffffff817451a5>] xen_swiotlb_init+0x415/0x4a0
 [<ffffffff81d53af9>] ? pci_swiotlb_detect_4gb+0x2c/0x2c
 [<ffffffff81d40ba9>] pci_xen_swiotlb_init+0x1c/0x2e
 [<ffffffff81d44474>] pci_iommu_alloc+0x5e/0x6e
 [<ffffffff81d54959>] mem_init+0x10/0x8f
 [<ffffffff81d39f05>] start_kernel+0x261/0x4b9
 [<ffffffff81d39a94>] ? set_init_arg+0x55/0x55
 [<ffffffff81d395ee>] x86_64_start_reservations+0x2a/0x2c
 [<ffffffff81d3cad3>] xen_start_kernel+0x599/0x5a5

Surprisingly a moment later, with swiotlb=8192 (so 16MB - which theoretically was tried), the netvm starts up.

[adrelanos]

Alright.

What about a high(er) initial value by default? swiotlb=8192 or so. Or even higher. You tell me.

Any disadvantages by that?

[adrelanos]

Alright.

What about a high(er) initial value by default? swiotlb=8192 or so. Or even higher. You tell me.

Any disadvantages by that?

[nrgaway]

That would be nice having a higher value. I patch every build to have 8192 and direct many people to increase the value whenever hear of setup failing due to lockup, freezes.

While most people having these issues the value of 8192 seems to solve them, but with one user they need to increase it to 16384.

[nrgaway]

That would be nice having a higher value. I patch every build to have 8192 and direct many people to increase the value whenever hear of setup failing due to lockup, freezes.

While most people having these issues the value of 8192 seems to solve them, but with one user they need to increase it to 16384.

[adrelanos]

If there is no significant performance penalty and/or other disadvantage, I'd say let's go with 16384.

[adrelanos]

If there is no significant performance penalty and/or other disadvantage, I'd say let's go with 16384.

[marmarek]

As I've said earlier - if you do not have that much coherent memory
available, netvm will crash early without any simple way to know what
has actually happened. This could be especially a problem if someone
want to start the VM later (VM restart, start different VM with PCI
device etc), because memory can be fragmented badly at that time.
But maybe indeed 8192 will not be too much for default value.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

As I've said earlier - if you do not have that much coherent memory
available, netvm will crash early without any simple way to know what
has actually happened. This could be especially a problem if someone
want to start the VM later (VM restart, start different VM with PCI
device etc), because memory can be fragmented badly at that time.
But maybe indeed 8192 will not be too much for default value.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Details about memory fragmentation: #174

[marmarek]

Details about memory fragmentation: #174