Kernel 5.8 breaks rtl8812au, here's a patch to fix it.

[muncrief]

Kernel 5.8-rc1 breaks the rtl8812au driver so I created a patch that appears to fix it.

I used a patch for the rtl8723bs I found at "https://patchwork.kernel.org/patch/11509497" as an example, so it's a fairly blind patch. I'm not a WiFi developer and didn't spend any time to understand the hardware/software, I just created the patch and lightly tested it on kernel versions 5.4.46, 5.7.2, and 5.8.0-rc1 and it appears to work fine. And by "lightly tested" I mean I simply connected to 2.4 and 5 GHz networks, ran speed tests, and looked for errors with dmesg.

The two issues were a structure name conflict and a change in the Management frame registration API. Kernel 5.8 now defines a structure named sha256_state and so does the driver, so I renamed the driver structure to sha256_state_pvt. I then did my best to account for the API change.

Hopefully this will help until the real developers have time to address the issue.

rtl8812au-5_8.patch.txt

[kubrickfr]

I confirm this works with 5.8.0-rc2-1-drm-tip-git-g24b806b0a1dd and 5.7.4-arch1-1.
Not tested thoroughly but at least it builds.

[gordboy]

I'm also looking at something very like this for my RTL8812AU driver. Someone has submited a pull request and I am trying things out.

gordboy/rtl8812au-5.6.4.2@101d8de

The names are slightly different but the scheme is more or less the same. Will report back when I have got this in shape.
The submitter has got the kernel version numbers wrong, but obviously that can be fixed.

[gordboy]

My repo now fixed for kernel 5.8, with scheme very similar to the one outlined above.