AP mode fails with 'timed out to flush queue 3'

[henkv1]

When I put my 8811cu and/or my 8812bu adapter in AP mode, the client is able to connect. But when it generates any traffic the connection drops.
On the AP I see the following message in the kernel log:
[ 1858.393929] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3

This is my setup:
Raspberry Pi 3 with Arch Linux Arm 6.1.34-1-rpi-ARCH, aarch64. Latest rtw88 driver from this repository. Bridged ethernet. hostapd v2.10 and WPA2 authentication.

[brianwitte]

I have an 8811cu and an 8812bu that should be arriving tomorrow evening and I can try to reproduce after they arrive. I will speak to @lwfinger later today and mull over this with him unless he starts working the issue before then and responds here.

I will update you as soon as I can. @henkv1

Thank you for letting us know.

[lwfinger]

First of all, I would like to introduce @brianwitte to this list. I recently placed a request on the Linux wireless mailing list for someone to co-administer the GitHub repos and to be trained when I am no longer able to do it. Brian stepped up, and I am pleased that he did.

@henkv1 - How are you setting up the AP? I would like to repeat what you are doing using an 8821ce to see if the problem is with the 8821c chip, or just with the USB driver.

[henkv1]

@lwfinger, @brianwitte I use this guide as a base for my setup. It works fine with the Realtek drivers in that repository (but those drivers are out-of-kernel and not fully compliant). So I tried to use the rtw88 driver instead, but without luck, so far.

[Samt43]

@henkv1 : I tried without success to create a high performance access point in 5Ghz using an 8822bu. As 8812bu is near the same hardware, you may have the same issue : #112
Please tell me if you observe the same issues too :)

[lwfinger]

Note that 8812bu is Wifi 4 (802.11n) hardware, and 8822bu is Wifi 5 (802.11ac). They are quite different.

[henkv1]

8812bu is capable of 802.11ac. When I use channel 36, the AP is visible, but I observe a lot of reconnects and the following error message in the kernel log of the AP:

[27801.522737] rtw_8822bu 1-1.5:1.0: timed out to flush queue 3

The AP is not visible when I use channel 40.

[henkv1]

Hi,
I tried again today with the new code, but I still observe the same issue with both rtw_8822bu and rtw_8821cu:
[ 90.197464] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 99.230889] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 109.574949] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 112.623039] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 115.399006] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 118.467021] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 128.711035] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 137.715211] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 140.499128] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 151.063147] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 160.051124] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 168.163595] rtw_8821cu 1-1.4:1.0: error beacon valid
[ 168.164284] rtw_8821cu 1-1.4:1.0: failed to download drv rsvd page
[ 168.406758] rtw_8821cu 1-1.4:1.0: error beacon valid
[ 168.407547] rtw_8821cu 1-1.4:1.0: failed to download drv rsvd page
[ 168.716231] rtw_8821cu 1-1.4:1.0: error beacon valid
[ 168.716903] rtw_8821cu 1-1.4:1.0: failed to download drv rsvd page
[ 168.859163] rtw_8821cu 1-1.4:1.0: timed out to flush queue 3
[ 169.267239] rtw_8821cu 1-1.4:1.0: error beacon valid
[ 169.267971] rtw_8821cu 1-1.4:1.0: failed to download drv rsvd page

[ 324.869779] rtw_8822bu 1-1.4:1.0: timed out to flush queue 3
[ 329.569671] rtw_8822bu 1-1.4:1.0: timed out to flush queue 3

[andreamerello]

I have two different rtl8822bu USB dongles. Here I get the same error ("timed out to flush queue 3") as soon as I shut down hostapd, and I have similar symptoms (client connects, but connection drops).

After some investigations, it seems that this is due to the rtl8822bu not sending beacons at all (I guess that the network can still be discovered due to probe responses).

In my understanding the driver uploads a beacon template to the dongle in the so-called "reserved pages", ad its firmware is in charge to periodically sending it over the air, but it seems this mechanism isn't working here. In my case the reserved pages are apparently uploaded fine (i.e. no complaints - no "failed to download drv rsvd page" / "error beacon valid" messages). The rest of traffic seems to flow normally.

I'm looking in this issue more in deep, but so far I have had no luck (I may also post on the kernel ML, but I need to collect and reorder some information first).

[lwfinger]

Do not post to the kernel ML, but rather to the wireless ML as described in README.md.

You are giving too much credit to the firmware. The beacons are generally triggered by hostapd. Only the contents of the beacon frame are stored.

I suspect the fault lies with hostapd. Have you tried making an AP with NetworkManager?

A quick fix would be to kill hostapd in a script that immediately unloads and reloads rtw88_8822bu.

[Samt43]

@andreamerello : Is 2.4Ghz working for you ? On my side, i only had the issue on 5ghz #112
I think you are right, there is definitly a hardware/firmware misconfiguration resulting on beacon not sent correctly. I couldn't find the root cause on my side unfortunatly ! At least it's reproductible on rtl8822bu chips it seems !

[henkv1]

I just found out that increasing the beacon interval in hostapd fixes the connection issues for me. Both 2.4 and 5 GHz are working with my rt8812bu adapter.
So things are getting better, but I still get the following errors in the kernel log:

[177058.808983] rtw_8822bu 1-1.1:1.0: timed out to flush queue 3
[177162.426150] rtw_8822bu 1-1.1:1.0: error beacon valid
[177162.426536] rtw_8822bu 1-1.1:1.0: failed to download drv rsvd page
[177162.556602] rtw_8822bu 1-1.1:1.0: error beacon valid
[177162.556979] rtw_8822bu 1-1.1:1.0: failed to download drv rsvd page

And this one:

[176854.124015] ------------[ cut here ]------------
[176854.124035] write RF mode table fail
[176854.124091] WARNING: CPU: 1 PID: 159403 at /home/hette/test/rtw88/rtw8822b.c:823 rtw8822b_config_trx_mode.constprop.0+0x7a4/0x7b0 [rtw_8822b]
[176854.124134] Modules linked in: rtw_8822bu(O) rtw_8822b(O) rtw_usb(O) rtw_core(O) ctr aes_arm64 aes_generic libaes ccm 8021q garp nft_masq nft_limit nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables nfnetlink veth tun rpcsec_gss_krb5 bridge xt_tcpudp stp llc iptable_filter iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ath9k_htc ath9k_common ath9k_hw ath mac80211 libarc4 cfg80211 rfkill rpivid_hevc(C) bcm2835_isp(C) bcm2835_codec(C) bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) videobuf2_dma_contig v4l2_mem2mem videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 raspberrypi_hwmon videobuf2_common i2c_brcmstb videodev vc_sm_cma(C) snd_bcm2835(C) mc nvmem_rmem uio_pdrv_genirq uio sch_fq_codel dm_multipath fuse dm_mod ip_tables x_tables ipv6 vc4 snd_soc_hdmi_codec snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer snd drm_display_helper drm_dma_helper drm_kms_helper v3d syscopyarea sysfillrect drm_shmem_helper sysimgblt fb_sys_fops
[176854.124398]  gpu_sched cec drm drm_panel_orientation_quirks backlight [last unloaded: rtw_usb(O)]
[176854.124423] CPU: 1 PID: 159403 Comm: iw Tainted: G        WC O       6.1.55-1-rpi-ARCH #1
[176854.124433] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[176854.124438] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[176854.124447] pc : rtw8822b_config_trx_mode.constprop.0+0x7a4/0x7b0 [rtw_8822b]
[176854.124466] lr : rtw8822b_config_trx_mode.constprop.0+0x7a4/0x7b0 [rtw_8822b]
[176854.124482] sp : ffffffc00a0636c0
[176854.124486] x29: ffffffc00a0636c0 x28: ffffff801c691f00 x27: ffffff8047f97880
[176854.124501] x26: 0000000000000003 x25: 0000000000000001 x24: ffffffe28702f268
[176854.124515] x23: 0000000000000003 x22: ffffffe28702f010 x21: 0000000000000000
[176854.124528] x20: 00000000000aeaea x19: ffffff800a962020 x18: 0000000000000000
[176854.124541] x17: 0000000000000000 x16: ffffffe292889490 x15: 0000000000000000
[176854.124554] x14: 00aa008a3579c268 x13: 6c69616620656c62 x12: 61742065646f6d20
[176854.124566] x11: fffffffffffedb28 x10: ffffffe293bae6e8 x9 : ffffffe2928fb164
[176854.124579] x8 : 00000000ffffefff x7 : ffffffe293bae6e8 x6 : c0000000fffff000
[176854.124592] x5 : 000000000000bff4 x4 : 0000000000000000 x3 : 0000000000000027
[176854.124604] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffff801c691f00
[176854.124617] Call trace:
[176854.124622]  rtw8822b_config_trx_mode.constprop.0+0x7a4/0x7b0 [rtw_8822b]
[176854.124639]  rtw8822b_set_antenna+0x74/0xc4 [rtw_8822b]
[176854.124656]  rtw_ops_set_antenna+0x64/0x94 [rtw_core]
[176854.124702]  ieee80211_set_antenna+0x48/0x120 [mac80211]
[176854.124874]  nl80211_set_wiphy+0x3f0/0xa20 [cfg80211]
[176854.125035]  genl_family_rcv_msg_doit+0xe4/0x154
[176854.125052]  genl_rcv_msg+0x130/0x264
[176854.125061]  netlink_rcv_skb+0x64/0x130
[176854.125070]  genl_rcv+0x40/0x5c
[176854.125079]  netlink_unicast+0x304/0x36c
[176854.125087]  netlink_sendmsg+0x1c4/0x434
[176854.125095]  sock_sendmsg+0x5c/0x6c
[176854.125104]  ____sys_sendmsg+0x260/0x280
[176854.125111]  ___sys_sendmsg+0xb4/0x110
[176854.125119]  __sys_sendmsg+0x8c/0xf0
[176854.125127]  __arm64_sys_sendmsg+0x2c/0x40
[176854.125135]  invoke_syscall+0x50/0x120
[176854.125146]  el0_svc_common.constprop.0+0x104/0x124
[176854.125157]  do_el0_svc+0x34/0xd0
[176854.125166]  el0_svc+0x30/0x9c
[176854.125176]  el0t_64_sync_handler+0xf4/0x120
[176854.125185]  el0t_64_sync+0x18c/0x190
[176854.125193] ---[ end trace 0000000000000000 ]---

[andreamerello]

On my side, neither 2.4GHz / 5GHz work. And configuring the network from network manager rather than hostapd didn't make any difference.

So far, what I've understood is that from the driver side, the AP mode is enabled and the beacon is downloaded to the firmware.

I've also tried to hack with some registers that are involved in beaconing process, but no luck.

Finally I've noticed that the beacon TX descriptor seems incomplete to me i.e. the PCI driver does fill up a lot of extra stuff wrt the USB driver (e.g. rate, hw sequence number, broadcast flag, etc). I've tried to do the same in the USB code, but nothing changed.

I will get back to do experimenting on this soon, and I will also try setting modify beacon interval.

[andreamerello]

Tweaking beacon interval, as well as other hostapd parameters which I thought could affect beacon itself (e.g. IEs) didn't make any difference here.

I see some commits that let me speculate me that AP mode worked correctly in some scenarios. I suspect that it is OK for PCIe cards, while I wonder whether is there any USB dongle that has no problem with it. If I could have more information on this, I could try to narrow down my experiments considering what's different in the code that supports working vs not-working cards.

[henkv1]

When I create the AP, 5GHz, channel 36. The AP is visible and I am able to connect. But when I generate some traffic, the connection drops and I see the following errors in the kernel log:

[  575.407785] rtw_8821cu 1-1.3:1.0: error beacon valid
[  575.415985] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  575.608729] rtw_8821cu 1-1.3:1.0: error beacon valid
[  575.616602] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  576.447244] rtw_8821cu 1-1.3:1.0: error beacon valid
[  576.456195] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  576.653218] rtw_8821cu 1-1.3:1.0: error beacon valid
[  576.661055] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  606.648608] rtw_8821cu 1-1.3:1.0: error beacon valid
[  606.656633] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  606.850490] rtw_8821cu 1-1.3:1.0: error beacon valid
[  606.858246] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  607.051187] rtw_8821cu 1-1.3:1.0: error beacon valid
[  607.058839] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  607.252374] rtw_8821cu 1-1.3:1.0: error beacon valid
[  607.259920] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  608.749121] rtw_8821cu 1-1.3:1.0: error beacon valid
[  608.771110] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  608.989341] rtw_8821cu 1-1.3:1.0: error beacon valid
[  609.011029] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  609.226526] rtw_8821cu 1-1.3:1.0: error beacon valid
[  609.234180] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  609.422156] rtw_8821cu 1-1.3:1.0: error beacon valid
[  609.429659] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page
[  649.943000] rtw_8821cu 1-1.3:1.0: timed out to flush queue 1
[  650.143029] rtw_8821cu 1-1.3:1.0: timed out to flush queue 3
[  650.339540] rtw_8821cu 1-1.3:1.0: error beacon valid
[  650.347078] rtw_8821cu 1-1.3:1.0: failed to download drv rsvd page