Add aarch64 asm accelerated versions of crypto algorihtms to kernel

[cotequeiroz]

This series adds armv8 Crypto Extensions accelerated versions of crypto algorithms to the following targets:

• bcm4908
• layerscape/armv8_64b
• mvebu/cortexa53
• mvebu/cortexa72
• rockchip/armv8
• sunxi/cortexa53
  Also:
• octeontx was missing the CE optimized CRC T10 algorihtm, so it was added here.
• bcm27xx/bcm2710 and bcm27xx/bcm2711 don't have the crypto extensions, so the SIMD neon versions were added instead.

mvebu/cortex{a53,a72} configs were given a refresh, although they are not strictly required.

For most targets, no algorithms were added if the regular C version was not included. I've made an exception for mvebu: I've mirrored the algorithms that the 32-bit cortex-a9 has. I found it rather odd that the asm config symbols were set at the target, not subtarget config, even though they only apply to the 32-bit a9. So, i mimicked what 86e27f1 had done to cortex-a9 and added the same algorithms to the 64-bit targets. I have not gone so far as to move the 32bit symbols to the cortexa9 subtarget, but I do think it is a good idea. I think I'm already doing too much in one PR, so I'm leaving this one out.

noob alert: sunxi/cortexa53 config-5.15 was missing the CONFIG_SUN50I_H616_CCU symbol, so I added it as a n. IIUC, it is a new H616 SOC, which--I'm guessing--is not supported by openwrt yet (or else someone would have already fixed it). Of course, if setting this to y is enough to add support for it, let me know, and I'll happily change this.

I've already done this to mediatek/mt7622 in #9698 because I have access to the hardware. The speedup was 16 times for aes-128-ctr, 17x for aes-128-gcm, and 9 times for sha256.
The targets supporting CE should be confirmed here or in #9738. I've compiled-tested the changes for all targets, but I don't have any of them to run-test.

[cotequeiroz]

Should anybody wish to benchmark this, install kmod-crypto-test, then insert the module, passing at least mode >=200 (for speed tests), and optionally, sec= # of seconds to run. Mode=200 will clock different block-mode-AES; 211=AES-GCM; 212=AES-CCM. Then check results with dmesg. Look in crypto/tcrypt.c for the full list.
Here's a diff --side of some tests on Linksys E8450 (mediatek/mt7622):

insmod /lib/modules/5.10.110/tcrypt.ko mode=211 sec=1; dmesg | tail -n39 | cut -b17-
testing speed of rfc4309(ccm(aes)) (rfc4309(ccm_base(ctr(aes-generic),cbcmac(aes-generic   |    testing speed of rfc4309(ccm(aes)) (rfc4309(ccm-aes-ce)) encryption
test 0 (152 bit key, 16 byte blocks): 250328 operations in 1 seconds (4005248 bytes)       |    test 0 (152 bit key, 16 byte blocks): 611586 operations in 1 seconds (9785376 bytes)
test 1 (152 bit key, 64 byte blocks): 169145 operations in 1 seconds (10825280 bytes)      |    test 1 (152 bit key, 64 byte blocks): 561872 operations in 1 seconds (35959808 bytes)
test 2 (152 bit key, 256 byte blocks): 73909 operations in 1 seconds (18920704 bytes)      |    test 2 (152 bit key, 256 byte blocks): 437846 operations in 1 seconds (112088576 bytes)
test 3 (152 bit key, 512 byte blocks): 41855 operations in 1 seconds (21429760 bytes)      |    test 3 (152 bit key, 512 byte blocks): 336445 operations in 1 seconds (172259840 bytes)
test 4 (152 bit key, 1024 byte blocks): 22499 operations in 1 seconds (23038976 bytes)     |    test 4 (152 bit key, 1024 byte blocks): 230860 operations in 1 seconds (236400640 bytes)
test 5 (152 bit key, 2048 byte blocks): 11725 operations in 1 seconds (24012800 bytes)     |    test 5 (152 bit key, 2048 byte blocks): 141403 operations in 1 seconds (289593344 bytes)
test 6 (152 bit key, 4096 byte blocks): 5977 operations in 1 seconds (24481792 bytes)      |    test 6 (152 bit key, 4096 byte blocks): 80131 operations in 1 seconds (328216576 bytes)
test 7 (152 bit key, 8192 byte blocks): 3006 operations in 1 seconds (24625152 bytes)      |    test 7 (152 bit key, 8192 byte blocks): 42172 operations in 1 seconds (345473024 bytes)

testing speed of rfc4309(ccm(aes)) (rfc4309(ccm_base(ctr(aes-generic),cbcmac(aes-generic   |    testing speed of rfc4309(ccm(aes)) (rfc4309(ccm-aes-ce)) decryption
test 0 (152 bit key, 16 byte blocks): 249427 operations in 1 seconds (3990832 bytes)       |    test 0 (152 bit key, 16 byte blocks): 628663 operations in 1 seconds (10058608 bytes)
test 1 (152 bit key, 64 byte blocks): 166321 operations in 1 seconds (10644544 bytes)      |    test 1 (152 bit key, 64 byte blocks): 576070 operations in 1 seconds (36868480 bytes)
test 2 (152 bit key, 256 byte blocks): 73068 operations in 1 seconds (18705408 bytes)      |    test 2 (152 bit key, 256 byte blocks): 445653 operations in 1 seconds (114087168 bytes)
test 3 (152 bit key, 512 byte blocks): 41782 operations in 1 seconds (21392384 bytes)      |    test 3 (152 bit key, 512 byte blocks): 343782 operations in 1 seconds (176016384 bytes)
test 4 (152 bit key, 1024 byte blocks): 22532 operations in 1 seconds (23072768 bytes)     |    test 4 (152 bit key, 1024 byte blocks): 232829 operations in 1 seconds (238416896 bytes)
test 5 (152 bit key, 2048 byte blocks): 11686 operations in 1 seconds (23932928 bytes)     |    test 5 (152 bit key, 2048 byte blocks): 142051 operations in 1 seconds (290920448 bytes)
test 6 (152 bit key, 4096 byte blocks): 1946 operations in 1 seconds (7970816 bytes)       |    test 6 (152 bit key, 4096 byte blocks): 80403 operations in 1 seconds (329330688 bytes)
test 7 (152 bit key, 8192 byte blocks): 3034 operations in 1 seconds (24854528 bytes)      |    test 7 (152 bit key, 8192 byte blocks): 42213 operations in 1 seconds (345808896 bytes)

insmod /lib/modules/5.10.110/tcrypt.ko mode=211 sec=1; dmesg | tail -n71 | cut -b17-
testing speed of rfc4106(gcm(aes)) (rfc4106(gcm_base(ctr(aes-generic),ghash-generic))) e   |    testing speed of rfc4106(gcm(aes)) (rfc4106(gcm-aes-ce)) encryption
test 0 (160 bit key, 16 byte blocks): 251562 operations in 1 seconds (4024992 bytes)       |    test 0 (160 bit key, 16 byte blocks): 683770 operations in 1 seconds (10940320 bytes)
test 1 (160 bit key, 64 byte blocks): 165619 operations in 1 seconds (10599616 bytes)      |    test 1 (160 bit key, 64 byte blocks): 695458 operations in 1 seconds (44509312 bytes)
test 2 (160 bit key, 256 byte blocks): 69906 operations in 1 seconds (17895936 bytes)      |    test 2 (160 bit key, 256 byte blocks): 570068 operations in 1 seconds (145937408 bytes)
test 3 (160 bit key, 512 byte blocks): 39631 operations in 1 seconds (20291072 bytes)      |    test 3 (160 bit key, 512 byte blocks): 461817 operations in 1 seconds (236450304 bytes)
test 4 (160 bit key, 1024 byte blocks): 21332 operations in 1 seconds (21843968 bytes)     |    test 4 (160 bit key, 1024 byte blocks): 335709 operations in 1 seconds (343766016 bytes)
test 5 (160 bit key, 2048 byte blocks): 11093 operations in 1 seconds (22718464 bytes)     |    test 5 (160 bit key, 2048 byte blocks): 215831 operations in 1 seconds (442021888 bytes)
test 6 (160 bit key, 4096 byte blocks): 5644 operations in 1 seconds (23117824 bytes)      |    test 6 (160 bit key, 4096 byte blocks): 126955 operations in 1 seconds (520007680 bytes)
test 7 (160 bit key, 8192 byte blocks): 2845 operations in 1 seconds (23306240 bytes)      |    test 7 (160 bit key, 8192 byte blocks): 67925 operations in 1 seconds (556441600 bytes)

testing speed of gcm(aes) (gcm_base(ctr(aes-generic),ghash-generic)) encryption            |    testing speed of gcm(aes) (gcm-aes-ce) encryption
test 0 (128 bit key, 16 byte blocks): 278652 operations in 1 seconds (4458432 bytes)       |    test 0 (128 bit key, 16 byte blocks): 886581 operations in 1 seconds (14185296 bytes)
test 1 (128 bit key, 64 byte blocks): 176314 operations in 1 seconds (11284096 bytes)      |    test 1 (128 bit key, 64 byte blocks): 897542 operations in 1 seconds (57442688 bytes)
test 2 (128 bit key, 256 byte blocks): 71876 operations in 1 seconds (18400256 bytes)      |    test 2 (128 bit key, 256 byte blocks): 708778 operations in 1 seconds (181447168 bytes)
test 3 (128 bit key, 512 byte blocks): 40429 operations in 1 seconds (20699648 bytes)      |    test 3 (128 bit key, 512 byte blocks): 546289 operations in 1 seconds (279699968 bytes)
test 4 (128 bit key, 1024 byte blocks): 21591 operations in 1 seconds (22109184 bytes)     |    test 4 (128 bit key, 1024 byte blocks): 376733 operations in 1 seconds (385774592 bytes)
test 5 (128 bit key, 2048 byte blocks): 11137 operations in 1 seconds (22808576 bytes)     |    test 5 (128 bit key, 2048 byte blocks): 232520 operations in 1 seconds (476200960 bytes)
test 6 (128 bit key, 4096 byte blocks): 1833 operations in 1 seconds (7507968 bytes)       |    test 6 (128 bit key, 4096 byte blocks): 132747 operations in 1 seconds (543731712 bytes)
test 7 (128 bit key, 8192 byte blocks): 2829 operations in 1 seconds (23175168 bytes)      |    test 7 (128 bit key, 8192 byte blocks): 69269 operations in 1 seconds (567451648 bytes)
test 8 (192 bit key, 16 byte blocks): 259526 operations in 1 seconds (4152416 bytes)       |    test 8 (192 bit key, 16 byte blocks): 867538 operations in 1 seconds (13880608 bytes)
test 9 (192 bit key, 64 byte blocks): 164037 operations in 1 seconds (10498368 bytes)      |    test 9 (192 bit key, 64 byte blocks): 887118 operations in 1 seconds (56775552 bytes)
test 10 (192 bit key, 256 byte blocks): 67611 operations in 1 seconds (17308416 bytes)     |    test 10 (192 bit key, 256 byte blocks): 673290 operations in 1 seconds (172362240 bytes)
test 11 (192 bit key, 512 byte blocks): 37562 operations in 1 seconds (19231744 bytes)     |    test 11 (192 bit key, 512 byte blocks): 510950 operations in 1 seconds (261606400 bytes)
test 12 (192 bit key, 1024 byte blocks): 20127 operations in 1 seconds (20610048 bytes)    |    test 12 (192 bit key, 1024 byte blocks): 344668 operations in 1 seconds (352940032 bytes
test 13 (192 bit key, 2048 byte blocks): 10352 operations in 1 seconds (21200896 bytes)    |    test 13 (192 bit key, 2048 byte blocks): 210356 operations in 1 seconds (430809088 bytes
test 14 (192 bit key, 4096 byte blocks): 5271 operations in 1 seconds (21590016 bytes)     |    test 14 (192 bit key, 4096 byte blocks): 118086 operations in 1 seconds (483680256 bytes
test 15 (192 bit key, 8192 byte blocks): 2630 operations in 1 seconds (21544960 bytes)     |    test 15 (192 bit key, 8192 byte blocks): 61443 operations in 1 seconds (503341056 bytes)
test 16 (256 bit key, 16 byte blocks): 246923 operations in 1 seconds (3950768 bytes)      |    test 16 (256 bit key, 16 byte blocks): 859009 operations in 1 seconds (13744144 bytes)
test 17 (256 bit key, 64 byte blocks): 157134 operations in 1 seconds (10056576 bytes)     |    test 17 (256 bit key, 64 byte blocks): 877824 operations in 1 seconds (56180736 bytes)
test 18 (256 bit key, 256 byte blocks): 63300 operations in 1 seconds (16204800 bytes)     |    test 18 (256 bit key, 256 byte blocks): 659758 operations in 1 seconds (168898048 bytes)
test 19 (256 bit key, 512 byte blocks): 35534 operations in 1 seconds (18193408 bytes)     |    test 19 (256 bit key, 512 byte blocks): 496413 operations in 1 seconds (254163456 bytes)
test 20 (256 bit key, 1024 byte blocks): 18875 operations in 1 seconds (19328000 bytes)    |    test 20 (256 bit key, 1024 byte blocks): 333591 operations in 1 seconds (341597184 bytes
test 21 (256 bit key, 2048 byte blocks): 9704 operations in 1 seconds (19873792 bytes)     |    test 21 (256 bit key, 2048 byte blocks): 200661 operations in 1 seconds (410953728 bytes
test 22 (256 bit key, 4096 byte blocks): 4946 operations in 1 seconds (20258816 bytes)     |    test 22 (256 bit key, 4096 byte blocks): 112025 operations in 1 seconds (458854400 bytes
test 23 (256 bit key, 8192 byte blocks): 2468 operations in 1 seconds (20217856 bytes)     |    test 23 (256 bit key, 8192 byte blocks): 58886 operations in 1 seconds (482394112 bytes)

testing speed of rfc4106(gcm(aes)) (rfc4106(gcm_base(ctr(aes-generic),ghash-generic))) d   |    testing speed of rfc4106(gcm(aes)) (rfc4106(gcm-aes-ce)) decryption
test 0 (160 bit key, 16 byte blocks): 255620 operations in 1 seconds (4089920 bytes)       |    test 0 (160 bit key, 16 byte blocks): 689930 operations in 1 seconds (11038880 bytes)
test 1 (160 bit key, 64 byte blocks): 163401 operations in 1 seconds (10457664 bytes)      |    test 1 (160 bit key, 64 byte blocks): 701022 operations in 1 seconds (44865408 bytes)
test 2 (160 bit key, 256 byte blocks): 70453 operations in 1 seconds (18035968 bytes)      |    test 2 (160 bit key, 256 byte blocks): 570270 operations in 1 seconds (145989120 bytes)
test 3 (160 bit key, 512 byte blocks): 39955 operations in 1 seconds (20456960 bytes)      |    test 3 (160 bit key, 512 byte blocks): 461019 operations in 1 seconds (236041728 bytes)
test 4 (160 bit key, 1024 byte blocks): 21365 operations in 1 seconds (21877760 bytes)     |    test 4 (160 bit key, 1024 byte blocks): 333832 operations in 1 seconds (341843968 bytes)
test 5 (160 bit key, 2048 byte blocks): 11086 operations in 1 seconds (22704128 bytes)     |    test 5 (160 bit key, 2048 byte blocks): 214256 operations in 1 seconds (438796288 bytes)
test 6 (160 bit key, 4096 byte blocks): 5653 operations in 1 seconds (23154688 bytes)      |    test 6 (160 bit key, 4096 byte blocks): 125934 operations in 1 seconds (515825664 bytes)
test 7 (160 bit key, 8192 byte blocks): 2848 operations in 1 seconds (23330816 bytes)      |    test 7 (160 bit key, 8192 byte blocks): 67274 operations in 1 seconds (551108608 bytes)

testing speed of gcm(aes) (gcm_base(ctr(aes-generic),ghash-generic)) decryption            |    testing speed of gcm(aes) (gcm-aes-ce) decryption
test 0 (128 bit key, 16 byte blocks): 278471 operations in 1 seconds (4455536 bytes)       |    test 0 (128 bit key, 16 byte blocks): 880623 operations in 1 seconds (14089968 bytes)
test 1 (128 bit key, 64 byte blocks): 174423 operations in 1 seconds (11163072 bytes)      |    test 1 (128 bit key, 64 byte blocks): 896621 operations in 1 seconds (57383744 bytes)
test 2 (128 bit key, 256 byte blocks): 72218 operations in 1 seconds (18487808 bytes)      |    test 2 (128 bit key, 256 byte blocks): 697911 operations in 1 seconds (178665216 bytes)
test 3 (128 bit key, 512 byte blocks): 40533 operations in 1 seconds (20752896 bytes)      |    test 3 (128 bit key, 512 byte blocks): 539460 operations in 1 seconds (276203520 bytes)
test 4 (128 bit key, 1024 byte blocks): 21584 operations in 1 seconds (22102016 bytes)     |    test 4 (128 bit key, 1024 byte blocks): 372198 operations in 1 seconds (381130752 bytes)
test 5 (128 bit key, 2048 byte blocks): 11139 operations in 1 seconds (22812672 bytes)     |    test 5 (128 bit key, 2048 byte blocks): 230050 operations in 1 seconds (471142400 bytes)
test 6 (128 bit key, 4096 byte blocks): 5703 operations in 1 seconds (23359488 bytes)      |    test 6 (128 bit key, 4096 byte blocks): 130938 operations in 1 seconds (536322048 bytes)
test 7 (128 bit key, 8192 byte blocks): 2848 operations in 1 seconds (23330816 bytes)      |    test 7 (128 bit key, 8192 byte blocks): 68716 operations in 1 seconds (562921472 bytes)
test 8 (192 bit key, 16 byte blocks): 258390 operations in 1 seconds (4134240 bytes)       |    test 8 (192 bit key, 16 byte blocks): 861758 operations in 1 seconds (13788128 bytes)
test 9 (192 bit key, 64 byte blocks): 163196 operations in 1 seconds (10444544 bytes)      |    test 9 (192 bit key, 64 byte blocks): 880134 operations in 1 seconds (56328576 bytes)
test 10 (192 bit key, 256 byte blocks): 67191 operations in 1 seconds (17200896 bytes)     |    test 10 (192 bit key, 256 byte blocks): 669823 operations in 1 seconds (171474688 bytes)
test 11 (192 bit key, 512 byte blocks): 37800 operations in 1 seconds (19353600 bytes)     |    test 11 (192 bit key, 512 byte blocks): 507910 operations in 1 seconds (260049920 bytes)
test 12 (192 bit key, 1024 byte blocks): 19967 operations in 1 seconds (20446208 bytes)    |    test 12 (192 bit key, 1024 byte blocks): 343543 operations in 1 seconds (351788032 bytes
test 13 (192 bit key, 2048 byte blocks): 10365 operations in 1 seconds (21227520 bytes)    |    test 13 (192 bit key, 2048 byte blocks): 207871 operations in 1 seconds (425719808 bytes
test 14 (192 bit key, 4096 byte blocks): 5263 operations in 1 seconds (21557248 bytes)     |    test 14 (192 bit key, 4096 byte blocks): 117027 operations in 1 seconds (479342592 bytes
test 15 (192 bit key, 8192 byte blocks): 2651 operations in 1 seconds (21716992 bytes)     |    test 15 (192 bit key, 8192 byte blocks): 61024 operations in 1 seconds (499908608 bytes)
test 16 (256 bit key, 16 byte blocks): 248719 operations in 1 seconds (3979504 bytes)      |    test 16 (256 bit key, 16 byte blocks): 856307 operations in 1 seconds (13700912 bytes)
test 17 (256 bit key, 64 byte blocks): 154868 operations in 1 seconds (9911552 bytes)      |    test 17 (256 bit key, 64 byte blocks): 874444 operations in 1 seconds (55964416 bytes)
test 18 (256 bit key, 256 byte blocks): 63540 operations in 1 seconds (16266240 bytes)     |    test 18 (256 bit key, 256 byte blocks): 657464 operations in 1 seconds (168310784 bytes)
test 19 (256 bit key, 512 byte blocks): 35522 operations in 1 seconds (18187264 bytes)     |    test 19 (256 bit key, 512 byte blocks): 493316 operations in 1 seconds (252577792 bytes)
test 20 (256 bit key, 1024 byte blocks): 6160 operations in 1 seconds (6307840 bytes)      |    test 20 (256 bit key, 1024 byte blocks): 329940 operations in 1 seconds (337858560 bytes
test 21 (256 bit key, 2048 byte blocks): 9631 operations in 1 seconds (19724288 bytes)     |    test 21 (256 bit key, 2048 byte blocks): 199171 operations in 1 seconds (407902208 bytes
test 22 (256 bit key, 4096 byte blocks): 4978 operations in 1 seconds (20389888 bytes)     |    test 22 (256 bit key, 4096 byte blocks): 111230 operations in 1 seconds (455598080 bytes
test 23 (256 bit key, 8192 byte blocks): 2477 operations in 1 seconds (20291584 bytes)     |    test 23 (256 bit key, 8192 byte blocks): 57975 operations in 1 seconds (474931200 bytes)

[abajk]

I think it would be better to build these modules as packages instead of permanently compiling them into the kernel. Compiling them permanently into the kernel results in increased size. For example, ghash is built this way.

[cotequeiroz]

For example, ghash is built this way.

I know. That was introduced by 285df63, in Feb 2020. Then in Dec 2021, fa3690f made the generic version built-in, and mac80211 will happily use it instead of the optimized version.

Most of the crypto api was switched to being built in by c718212, earlier in June, 2021, because:

This reduces the flash space impact, since built-in code is much smaller than a bunch of kernel modules on squashfs

This is basically what I'm mimicking here. I looked at all of generic drivers that were already built in (not necessarily by the above commits), and selected their asm version where available. The exception was mvebu, where I mimicked the 32-bit behavior instead. I did so because the 32-bit selection was not done in the subtarget config, but in the target one--which should apply to both 32 and 64-bit SoC. I wasn't sure this would be the best course of action, but if I owned one of them, I would build them in.

[cotequeiroz]

I'm marking this ready for review. Kernel detects CE support before enabling the drivers, so this should not break anything. I've rebased and compile-tested this today.

[Ansuel]

@cotequeiroz only concern of this is size increase... aside from this i think it should be safe to enable this...

@ynezz wonder if you can spin this on a github action just to make sure the additional config doesn't cause some strange thing?

[cotequeiroz]

It's a 20k increase for most compressed target images, and 30k for mvebu, which adds the most algorithms.

[Ansuel]

@cotequeiroz are you aware of space constrains for the kernel partition? Wonder if we should use kmods and set them as default packages?

I just read the reason of why it would be a bad idea to use mods... think will just merge this

[Ansuel]

Anyway i'm running this under https://github.com/Ansuel/openwrt/runs/6730322372 and then merge if everything goes well

[Ansuel]

Thanks, merged to master! We still have to wait for the wolfssl pr tho...