Atomic channel data updates

[pkendall64]

Investigation

Investigating #3347, in Full resolution modes the channel data each channel is updated atomically as they are 32bit values aligned in an array. There is a multi-update problem with hybrid mode, which the PR addresses (see 1 below).

The code that takes the channels and converts them to SBUS output operates on each channel individually and each channel is read only one time in the code that builds the SBUS frame. So each channel is read atomically. The SBUS frame is only sent to the serial port once it is fully built.

Unless of course the compiler is generating multiple reads on the channelData array while forming the SBUS frame! Which is what it's doing.

Solution(s)

1. Change the OTA channel generation so each channel is only changed once, i.e. not using the slot for intermediate data.
2. In the serial output driver, we create a local copy of the global ChannelData and pass the pointer to the local copy down to the serial protocol drivers which may/do access each channel slot multiple times.
   Fixes SBUS Periodic glitch every15-20 seconds #3347

[CapnBry]

I've run into another problem with the direct ChannelData access used everywhere

When the data comes in and gets loaded into ChannelData TXModuleEndpoint::RcPacketToChannelsData(), there is a callback that then loads the headtracking data onto it. There's a brief bit of time between where one sets the channel values and the other overrides it. Sometimes the RC OTA packet is sent between these two, meaning that the channel values flicker back to the handset data for a packet, then back to the headtracker value for the next packet.

If I just hack into the source that the TXModuleEndpoint::RcPacketToChannelsData() makes a local copy, then calls the callback with that array, then copies the value into the channeldata array after overriding, I never see the flicker back, indicating that sometimes this concurrency issue is happening.

I'm testing with the headtrack data being fed in from the main tx_main loop to take the backpack out of the equation, putting the headtrack data in CH0 so it is sent every packet, and then just having the RX print a debug line if CH0 isn't the headtrack value. At 500Hz or 100Full I very rarely see one but at 50Hz I see one maybe every 10 seconds on average.

This needs to be addressed as well, but the maybe bigger issue is why the TOCK is going off right as the channel data is coming in from the handset. The sync is usually ~100us before transmit time but it seems there's enough jitter in either the handset or us that we're managing to transmit right as the data is arriving so our actual end-to-end latency may be fairly poor if our ETX sync is so bad.

[CapnBry]

I've added 29fe04f which has a new callback in Handset to allow overrides of the channel values before they are published. PPM and CRSF handsets are updated to operate on a local copy of the channels, which are then overridden, and then the RCDataReceived() method is called to "publish" them. Also note that PPM handset would update the arm flag before publishing and CRSF would update the arm flag after publishing, so I've made this consistent so they both update the arm flag then publish. I'm not sure which way is the most correct (perhaps the other way?) but it should be one or the other.

The changes to devBackpack.cpp are pretty rough, but this is rewritten in #3373 so I didn't want to spend much time on it only to cut it back out again after this is merged.

Let me know if you'd prefer I revert my changes and submit them another way. I didn't want to have to paste such a large diff here.

[pkendall64]

I've added 29fe04f which has a new callback in Handset to allow overrides of the channel values before they are published. PPM and CRSF handsets are updated to operate on a local copy of the channels, which are then overridden, and then the RCDataReceived() method is called to "publish" them. Also note that PPM handset would update the arm flag before publishing and CRSF would update the arm flag after publishing, so I've made this consistent so they both update the arm flag then publish. I'm not sure which way is the most correct (perhaps the other way?) but it should be one or the other.

The changes to devBackpack.cpp are pretty rough, but this is rewritten in #3373 so I didn't want to spend much time on it only to cut it back out again after this is merged.

Let me know if you'd prefer I revert my changes and submit them another way. I didn't want to have to paste such a large diff here.

All good. I'm ok with this since it's effectively temporary till #3373 is merged.

[CapnBry]

Code looks good and tested on Nomad TX / BAYCK Dual RX at 1000Hz (both dual core MCU). I don't have an SBUS tester I trust enough to see if one frame has a corrupted value, but for sure the old code had possible concurrency problems.