T-BEAM rebooting, but only when airborne

[buzz53]

Apologies if this is not appropriate for a bug report, but there doesn't seem to be anywhere else to discuss this excellent project.

I have two T-BEAM units, both behave the same. Also, the behaviour is identical with rc5 pre-loaded from Liligo or if I recompile and load the "new" rc5 from github. It seems to happen with both legacy (UK) and PAW protocols, although I am mainly testing with PAW.

On the ground the units operate perfectly and will run for 48+ hours with no problems. This is with both units interacting and with a tablet connected, so I think it should be a thorough test. However, when I go flying, they always reboot frequently - usually every 5 to 20 minutes. I am confident there is no problem with vibration (it happens even when held in a passenger's hand) or power supply (it is using the local battery, same as for ground testing).

I have modified the OLED display to report the result of "rtc_get_reset_reason(0)". Normally this is 1 but following reboot it is 12 (S/W RESET). BTW one unit has no display so this is not a factor.

This seems very odd. The only differences I can think of is that when airborne the GPS velocity is non-zero and also it will receive transmissions from many other types of equipment, perhaps this is triggering something? I will investigate further but wonder if anyone else has seen this problem?

Thank you,
Alan

[buzz53]

Looking at this a bit more, it seems that the behaviour would be consistent with the unit executing a "Save settings and reboot" via the web interface. This causes Reset Reason = 12. I have now added a non-volatile counter to increment and display whenever this restart code executes, so will soon know for certain. But how can this be happening? On my early flights I had a browser on my tablet running to monitor the status, so you could perhaps imagine something odd happening, but I'm 99% sure that on later flights, and especially yesterday, there was no browser active, just SkyDemon.

[buzz53]

Oops, didn't mean to close the issue.

[lyusupov]

Alan, despite all the SoftRF units are more or less equal - almost every client's environment (a set of installed applications) is different. Your case is not first one where client's software is suspected to abuse SoftRF operation. Inspect your tablet on presence of any web apps which may cache and then make an attempt to re-sync (in background) all the web pages that you've recently accessed in your browser.

Other options could be:

• use another tablet, it would be best if it has factory configuration with minimum built-in apps except your SkyDemon ;
• use Bluetooth (if able) instead of the Wi-FI to connect your SkyDemon over NMEA or GDL90 (valid for current SoftRF GitHub source code).

[Tjeerdm]

There is a memory leak in Hex2Bin which is called for every received packet in ParseData.
It allocates an object of type String which is never de-allocated.
When receiving one packet every second the system reboots after some 70 minutes. (freertos calls abort())
Moving the line fo.raw = Hex2Bin(...) inside the next block fixes it as long as nmea_p is not set.

[lyusupov]

@Tjeerdm
In order to detect any memory leaks, WebUI status page has "Free memory" field.
When a leak happens - the value gradually decreases over the time.
Please, provide a sequence of screenshots (taken at, say, 10 minutes interval) where I could see the free memory value. And specify conditions when the screenshots had been taken at.
It would be nice if you will turn "Private" messages on and capture the log of serial data output (at 38400 baud) all the way from boot down to the self-restart.
There could be some RF noise sources nearby that floods the SoftRF input.
I don't see the leak you've mentioned in my testing environment. But I can admit that this could happen under certain curcumstances.

[lyusupov]

You either provide proper reproducing instructions or close this ticket, please.

I see no issues in my 24 hours stress test with average 14 packets per second Rx ratio of "legacy" traffic.
I also did not notice any memory leaks there.

[buzz53]

I'm sorry for the delay but I have been working away from home. I did agree that the most likely cause was accidental triggering of the "Save and reboot" function, so I added a diagnostic to increment a counter when this happens. This works correctly when testing: if I deliberately request a "Save and reboot" then my diagnostics show this counter has incremented and the "Reset Reason" given is 12. I spent a long time trying to make this happen accidentally, logging in and out with the browser, running my second unit as a target etc etc. but it worked perfectly for 48 hours.

Today I finally went flying again. I made sure that my phone was in flight mode and there was no browser active on my tablet. It was connected over WiFi to Skydemon using GDL90. The SoftRF worked for about 15 minutes (and received genuine PAW targets). Then without warning it rebooted, showing Reset Reason = 12 but the "Save and reboot" counter had not changed. This happened about 6 or 7 times in my 40 minutes flying. As I said before I'm sure it is not a power issue (it's using a battery) and there is no vibration. It is very strange and I'm not sure what to try next. I think I will try disabling things until it works correctly.

Alan

[lyusupov]

@buzz53

Alan, if I would face with the situation that you have, I would:

1. at first, try to find a way to reproduce the reboot on the ground (say, on your homebase airport), near PAW devices, OGN-R ground stations and other RF noise sources ;
2. after that, I would switch SoftRF onto some other RF protocol and frequency (with littlle or no traffic, say, OGNTP) to see if the issue is P3I protocol-specific ;
3. if the issue remains active regardless of protocol setting, I would switch from Wi-Fi connection onto USB cable (or Bluetooth) to see if the issue is associated with SoftRF's Wi-Fi AP operation.

[buzz53]

Hi Linar, thanks for the suggestions. I had previously found the same problem with both legacy and P3I protocols. Here's some more information after two flights today. I set P3I protocol (there are very few legacy targets at this time of year). I turned off Bluetooth, NMEA, GDL90 and Dump1090 outputs. I set my phone and tablet to flight mode. In other words there was no onboard connection to the SoftRF at all, I monitored only the OLED.

We received a few dozen P3I packets but after about 20 minutes it rebooted as usual, and did so another three times. The second flight was just the same. Back home, it is working perfectly of course! Next time I propose to fit a dummy load in place of the antenna.
Alan

[buzz53]

More experiments! I flew today firstly with my own SoftRF but this time with a dummy load on the 868 MHz antenna. As before, there was no WiFi or Bluetooth connection. Same result as before: it rebooted 3 times in 20 minutes. I then collected two new T-beam units which had just been delivered to a friend, and operated one of those on the flight home. This time it was set for legacy mode and I connected using GDL90 from Skydemon. Once again, it rebooted at least three times in a short flight. That's now three units that behave the same. A common factor is that all of these tests have been in the same aircraft although I can't see how that would matter. I do have a mode C transponder but the antenna is external and well away from the SoftRF. Next flight I will try to use a different aircraft but meantime it would be nice to hear from anyone who has actually done any flight tests of this hardware. ideally in a powered aircraft at 150 kts.
Alan

[lyusupov]

@buzz53
Alan, more than 60 of the T-Beam boards were delivered to UK pilots and aviators since June last year.
I suppose you guys can create a community somewhere like uk.rec.aviation.soaring newsgroup or FLYER forum. You could meet, discuss and resolve this issue locally. Then, please, share the results here, with other 300+ SoftRF users worldwide.

GitHub tickets does not seem to be the best way to discuss general topics, like particular EMI cases.
You may consider to switch onto SoftRF's Gitter channel instead.
The link ( a "badge" like this one ) is on the topmost page of the SoftRF project's github site.
Feel yourself free to invite other UK pilots (SoftRF users) on this channel.

[buzz53]

Linar, I agree this is not the best place, but I did not know about Gitter. I'll try there.
Thanks again, Alan

[lyusupov]

@buzz53

Alan,

Same result as before: it rebooted 3 times in 20 minutes...

< ... skipped ... >

... it would be nice to hear from anyone who has actually done any flight tests of this hardware. ideally in a powered aircraft at 150 kts

See the pictures below.

Resolution: unable to reproduce.