Alternate ath9k and ath10k radio reset for deaf nodes #857
Conversation
Experimental test to see if we can reset a deaf radio by leaving and rejoining the adhoc network rather than using scan. A scan, especially if we have to do both active and passive, essentially mutes the radio to AREDN traffic for 10-20 seconds, which isn't good. If the radio is completely deaf then it doesnt matter, but particularly on the 9K radios we do this when things are looking a bit dodgy, though not deaf. This approach is much, much, quicker, but needs to be tested to see if it has the same un-deafing result.
|
What monitoring daemon will be required? |
|
They already exist. This just changes how they do the reset. |
|
The nodes I currently see issues with are all ath10K radios. But I'll definitely test. |
|
a more complicated fix, and should work. This used to be "/usr/sbin/iw $iface scan freq $freq passive"-- a passive scan on a single channel, possibly quicker and less risk of further symptoms than a chip reset. Wouldn't this be simple to get the job done? |
|
#response_container_BBPPID{font-family: initial; font-size:initial; color: initial;} I do exactly what Joe propose with two old 900Mhz Rockets and it works without loosing links. Prior we where setting Ping timeout power cycle on the switch, pinging the opposite node to detect link drop. Jean-Michel Vien - VA2XJM Président, Association des Radioamateurs de ***@***.*** - Cell:819-860-9680Ligne Association: (873) 476-0132 #822956 From: ***@***.***: June 1, 2023 19:18To: ***@***.*** to: ***@***.***: ***@***.***: Re: [aredn/aredn] Alternate ath9k and ath10k radio reset for deaf nodes (PR #857)
a more complicated fix, and should work. This used to be "/usr/sbin/iw $iface scan freq $freq passive"-- a passive scan on a single channel, possibly quicker and less risk of further symptoms than a chip reset. Wouldn't this be simple to get the job done?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
|
Not any less risky to do this rather than a single frequency scan as the scan does the same chip reset - but this doesnt do anything else so wont tie the radio up unnecessarily. |
|
Got it, the reset being the lowest impactful change or risk of symptoms is the best approach. I champion moving this PR forward. Capturing some history here. I had come to the conclusion that this behavior would not be fixed and perpetuated in ath9k/10k/... The logic being that it is a desirable behavior for a client of an AP, to become deaf to other neighbors (by raising the noise floor artificially), such that the remaining signal of the AP is less SNR, but no longer contending with these neighbor signals to decode the bits. In this scenario, the client is only attempting to communicate with the AP<->internet. for adhoc or IBSS networks, where we expect to communicate (OSLR) to all neighbors, this is problematic -- to zero in on a strong neighbor. Note, another possible option is to turn off this "noise immunity" functionality. The limited testing I did at the time, showed using noise immunity improved AREDN long distance usage, and issues went away. Interestingly, see DD-WRT notes -- which combined with AREDN experience shows results all over the board :) : "its generally recommended to leave this disabled, only enable if you are an advanced user, are diagnosing various wireless issues, or it fixes a specific issue you were having. Especially if you have multiple Qualcomm Atheros routers connected to each other in any way, its highly recommended to have noise immunity enabled, or disabled on all routers, but not mixed. There has been some reports over the years that disabling noise immunity has helped stabilize the WLAN in terms of throughput &/or reducing dropouts, disabling noise immunity could also result in great or unchanged close range performance, but horrible or no throughput whatsoever, at medium ~ far range, so experiment with this setting. There is also some cases where enabling noise immunity gives abnormally low TX/RX rates & throughput, or noise immunity is simply too aggressive even in low noise, in this case, disable the feature." |
|
Im most interested in what different factors lead to a node going deaf. Tim can any properties be tracked with prometheus telemetry to identify the scenarios that cause this? I haven't had the problem lately, but im always willing to break things!!! |
|
The prometheus information contains lots of information which might be relevant, such as neighbor SNR and such like .. but was is specifically significant in causing the deafness I dont know (though others might). Also, I would hope you've not seen the problem recently as there have been mitigations to deal with the problem for many years. I think I made them more effective but also more disruptive (not great) so I'm hoping this change will improve that. But it would be good to have this up on any node which had a history of problems so we can validate this some more. |
|
The ptp link that I was running into deafness problems with has not returned! Im working to get one end of it back up (node died). Yes I will work to Update nodes to a FW capable of reporting Prometheus. |
There is a well known issue with the ath9k chip set that it will occasionally go deaf. The usual response to this problem is, when detected, to run a wifi scan which will reset the chip and resolve the problem. However, this takes quite a lot of time - 20+ seconds when scanning the 5GHz band - and during this time the radio is unresponsive to normal traffic, causing drop outs in anyone sending traffic over the node, and potentially causing the mesh to start to reconfigure itself as OLSR traffic is no longer sent.
While I've been unable to find a real solution to this problem (there are various proposals but none confirmed as working) it is generally agreed that a wifi chip reset is all that's required rather than the full wifi scan. Using the included patch, we can now do this from our monitoring daemons when problems are detected. In testing this proved as effective and much faster.
I've also moved the ath10k monitor to use a similar approach, although I've not seen a problem with this driver in recent nighty builds (a node I have which connects across town had previous had a deaf problem with the ath10k driver).
This PR includes the this patch - https://patchwork.kernel.org/project/linux-wireless/patch/20210914192515.9273-2-linus.luessing@c0d3.blue/ - which gives user level access to resetting the ath9k chip (there is already a way to do this on the ath10k).