Inclusion into upstream kernel plans? #1
Comments
|
Hi, As you can see the current approach is "all-or-nothing". That is: you either get complete inline behaviour or not. While it greatly improved our IO cache workload (with smaller IO block size), other synthetic benchmarks with larger IO block size (~>2MB) actually sometimes show worse results. Secondly, the current approach is rather x86-specific, but upstream kernel needs something more generic, which would at least run on different architectures. The plan forward is to try to profile specific parts of I would not be surprised, if the proposed here That said - if you found the patches useful we would appreciate if you could share some details about your IO workload. |
|
I'll have to get back to you on that once I actually try it out.. But my most important workload would by ceph disks encrypted with luks/dmcrypt, so each pod in k8s can choose if they need encryption at rest or not for their data (given the huge performance impact it has). |
|
I have initiated the discussion with the upstream. You may follow https://lkml.org/lkml/2020/6/19/1934 |
|
Thanks you for then efforts to improve the performance in the upstream, it's very valuable for all of us :-) :-) :-) |
|
I don't mean to ressurect this issue, but this is for anyone who stumbles across this in the future. As of kernel version 5.9, the changes have been upstreamed. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.9-rc1&id=39d42fa96ba1b7d2544db3f8ed5da8fb0d5cb877 |
|
Its sad its not run-time options (via /proc knob) - but one has to recompile the kernel.. becomes annoying with all the kernel upgrades that happens all the time :( |
|
Well, even with proc support you have to recompile at least once to get the code in the first place. Once you have the kernel with above flags, you can just enable/disable flags in runtime by manipulating |
|
Hi, thank you very much for all your work. Is there a step-by-step howto available to make use of those changes? As an end-user I'm having a bit of a hard time understanding what needs to be done to benefit from them. Like how to specify the new flags while recompiling the kernel? And should we also compile the xtsproxy module from the repo since the |
|
What distribution are you using? Maybe it makes sense just to wait for the upstream changes to get propagated? |
|
Thanks for your quick answer! I'm using Archlinux with kernel 5.9.2, and I ask because I'm just provisioning a new machine with NVMe storage where the encryption speedup would be most appreciated, especially since I'll be transfering a lot of user data for the initial setup.
So you mean that once we have kernel 5.9 we don't need to recompile as indicated in a previous comment? Nothing else is needed but the What I am mainly trying to grasp is: how to make use of the added flags ( Thanks |
I think so.
Yes, you can use the flags independently. It seems you don't need to mess with If you're on Archlinux there is a good chance you already have |
|
Thank you very much, this is exactly the information I was looking for! I'll try that and report the results. Much appreciated. |
|
Preliminary results with ramfs confirm the speed gains: Now I need to find a good IO benchmark example involving NVMe storage to see what the effect is. Note: at first I tried to do the testing it with: The speed gains were exactly the same as in the previous examples, however the two sha256sums of eram0 did not match the one from ram0 (but were consistent together before/after refreshing with the perf parameters). Edit: maybe this results from not using a detached header and overwriting it? From the article:
|
This and the fact that with So by hashing |
|
Ha of course, that makes sense. Thanks |
|
Hi, I have tried doing sudo cryptsetup refresh --perf-no_read_workqueue --perf-no_write_workqueue /dev/mapper/root with a 5.9.x kernel and my system locks up with attached messages. Am I doing this wrong?
|
|
No, you're doing it right. Which exact kernel version are you using? Is it a distro or a self-compiled kernel? |
|
Hi, same error here: I'm running Ubuntu 20.04 with Kernel 5.9.8 and 5.9.10 (from here https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.9.8/amd64/). One system crashed tonight with those scheduling errors. Cause seems to be high i/o load by a btrfs scrub job (scheduled monthly). Reproduced this multiple times since then by forcing some extra i/o-load on the block devices. System was running fine (with this kernel) for some days now. I never observed this behavior when using the default Ubuntu kernel with the two updated modules from cloudflare added. Further testing only on non-production machines / VMs. Some details about my setup (maybe that's relevant): I'm running LVM + dm-crypt + btrfs |
|
It would be very helpful if you could also rebuild the |
|
Happens in 5.9.9 and in 5.10.0-rc6, custom build kernel on gentoo here is the config.. |
|
[Tue Dec 1 10:22:59 2020] BTRFS info (device dm-0): devid 2 device path /dev/mapper/root_1 changed to /dev/dm-1 scanned by udevd (22281) |
|
For the record, no issue so far here on Arch Linux with standard |
|
@evilbulgarian can you post your |
|
fio --bs=4096 --rw=randrw --name=generic_job_name --filename=./test.fio --numjobs=1 --size=50g --thread --direct=1 --runtime=300 --rwmixread=50 --iodepth=256 --ioengine=libaio |
|
If I understand the problem right, the patch below should probably fix it. We will test it on our end, but would appreciate folks here, who reported the problem, to try it out as well, if possible/feasible. |
|
Hi @ignatk, Thanks for patch seems like now there is no lockup but lots of these btrfs errors in output.. [Tue Dec 8 08:44:51 2020] BTRFS warning (device dm-0): direct IO failed ino 3185621 rw 0,0 sector 0x34dd1610 len 4096 err no 10 |
|
Also the performance is worse but seems issue is isolated to the fio job, I do not see errors if not running fio so far ( tried recreating the fio file as well ): before flags: Run status group 0 (all jobs): after flags: Run status group 0 (all jobs): |
|
What about this one? Should help with errors, not sure if it helps with performance |
|
Can you try again (I updated the above comment with the patch - probably a copy-paste error). |
|
@ignatk, ok seems like same errors.. [Tue Dec 8 13:04:24 2020] BTRFS warning (device dm-0): direct IO failed ino 4077816 rw 0,0 sector 0x3660bb80 len 4096 err no 10 |
|
Thanks for the follow up - will try to reproduce on my side then... |
|
Some update: I've been trying hard to reproduce something like above in QEMU setup using ramdisk. I've modified the Linux ramdisk driver to complete read requests in For me to try to dig further is it possible for you to provide:
Thanks. |
|
Thanks for troubleshooting this @ignatk,
grep -A 10 xts /proc/crypto
|
|
I was able to reliably reproduce this in QEMU with only 1 cpu and scaling down While we figure this out, as a workaround, you may try to actually increase |
|
Hi all, I think I've come up with a patch which at least fixes the above on my environment. If you could test it on your side and let me know if it fixes the issues, I would be very grateful |
|
@ignatk thanks for keeping at this! here is result ( looked good at first then got the io error and fio stopped ): [Wed Dec 23 16:55:41 2020] BTRFS error (device dm-0): bdev /dev/mapper/root errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 fio --bs=4096 --rw=randrw --name=generic_job_name --filename=./test.fio --numjobs=1 --size=50g --thread --direct=1 --runtime=300 --rwmixread=50 --iodepth=256 --ioengine=libaio generic_job_name: (groupid=0, jobs=1): err= 5 (file:io_u.c:1787, func=io_u error, error=Input/output error): pid=3169: Wed Dec 23 16:55:42 2020 Run status group 0 (all jobs): |
|
Not sure if it is related - but I found a typo in my patch: a missing |
|
@ignatk re-patched and same error and fio command stops. |
|
Changed the idea a bit and came up with this I've been running this for a while in my repro environment and so far no warnings in dmesg or fio errors |
|
@ignatk nice job! no more errors in dmesg ( but slower fio prob due to something with btrfs not sure ): `❯ fio --bs=4096 --rw=randrw --name=generic_job_name --filename=./test.fio --numjobs=1 --size=50g --thread --direct=1 --runtime=300 --rwmixread=50 --iodepth=256 --ioengine=libaio Run status group 0 (all jobs): ~ 5m 2s ~ 14s Run status group 0 (all jobs): |
|
Thanks for testing! I've submitted the patches to the |
|
I see a second updated set of packages has been submitted here: https://www.redhat.com/archives/dm-devel/2020-December/msg00468.html @ignatk would be great to get these backported to 5.10 if possible, given that's a stable release. |
|
The patches have |
Brilliant, thank you :) |
|
FYI these patches and even some more were merged into upstream and backported to |
|
@ignatk Hi, I just want to throw in a quick THANK YOU for everything you're doing to speed up Linux disk encryption! You're AWESOME! <3 |
|
Interesting. When this flags enabled guest OSes in virtualbox became extremely unstable. May be we still missing something with btrfs bug. |
|
Might be since I've recently got a hard lockup when machine with these flags enabled started heavily swapping. I am waiting whether this will occur again. |
|
A long time ago i noticed strange VBox behavior when it tries to write to memory buffer while it in process of writing. But i may be mistaken. I got several direct-io errors about data changed "in-flight" during write. May be this is connected. |
Hi guys,
This is a fantastic optimization on a very important part of the system.. Who doesn't use LUKS today :)
I haven't been able to find any mentioning of this patch on any of the kernel mailinglists :(
Whats the plans, for including this in the upstream kernel, so everyone can benefit?
The text was updated successfully, but these errors were encountered: