Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sighting: Stratux drop out #631

Closed
3 tasks
jzeevi opened this issue Jul 5, 2017 · 89 comments
Closed
3 tasks

Sighting: Stratux drop out #631

jzeevi opened this issue Jul 5, 2017 · 89 comments

Comments

@jzeevi
Copy link

jzeevi commented Jul 5, 2017

  1. Stratux version:1.3b3

  2. Stratux config:

    SDR

    • single
    • [ X] dual

    GPS

    • [X ] yes
    • no
      type: GPYes

    AHRS

    • [ X] yes
    • no

    power source: Airplane (USB Port)

    usb cable: um, yes?

  3. EFB app and version: (e.g., WingX Pro7 8.6.2) FF, beta

    EFB platform: (e.g., iOS 9.2) 10.3.x (latest, whatever version that is)

    EFB hardware: (e.g., iPad Mini 2) iPad Mini 2

  4. Description of your issue: A sighting: while on a set of long flights (2-3 hours each) over this last weekend, I saw FF showing ADS-B towers (i.e. connected to Stratux), but NO ADS-B data. Speed went to zero, position went to zero, the FF airplane went to a blue pulsing dot, "remaining" data was missing in the navlog. In one instance, the data came back by itself within a minute or so. In another case, I rebooted the Stratux (power cycle) after 2-3 minutes. When it re-occured, I simply powered off the stratux and utilized the iPad GPS for FF data (at this point, I was 2.5 hours out from my destination. In total over three days, I flew 18+ hours). Earlier, I had checked Stratux temperature and seen that it was around 51 degrees C. Note: in my plane (Aircoupe) the GPS has sky view through the canopy.

If possible, enable "Replay Logs", reproduce the problem, and provide a copy of the logs in http://192.168.10.1/logs/stratux/ and http://192.168.10.1/logs/stratux.log.

@Ergonomicmike
Copy link

This just happened to me today, although mine might not be a representative sample. (I've sh'd up to an Eric build, and then back down to a Chris build. ) Also, my Stratux has been on for about 4 hours in the house (AC set for 83) and is quite warm in a Chris box, with no fan and copper foil shield around to make it a little oven. Am also getting "stuttering" of the web UI, as if packets aren't making it thru or are being delayed or sent real slow. https://vid.me/bGy2d

@daveholland86
Copy link

I also had mine drop out completely, no traffic no GPS but the box was still running and wifi on v1.3b2
c6e96b5. I was flying my checkride so I didnt trouble shoot it but it really threw me off my game (Instrument check), still passed though. I have since updated to the new b3 but have not flown since.

@jpuetthoff
Copy link

daveholland86, you got to use stratux during the instrument checkride? Ipad?

@daveholland86
Copy link

Yes I was using Foreflight with electronic terminal procedures.

@jpuetthoff
Copy link

Any limitation what you cannot use the ipad/ foreflight for? Doing my IFR training, and i like my ipad. My panel mount certified gps is not a moving map.

@daveholland86
Copy link

No. Its only to be used to for situational awareness, so the gps approaches will be loaded in the the panel mount gps and nav from the VOR dial. You can verify your position on the ipad but its not official. I used it for flying my enroute, and approach and departure plates and keeping track where I was on the approach. Its good to see the aircraft on the leg segment and get the stepdown altitudes.

@cyoung
Copy link
Owner

cyoung commented Aug 22, 2017

I've confirmed this issue last weekend. On a 1.25hr flight, ForeFlight showed "ADS-B Disconnected" 3-4 times. Uptime on Stratux (gdl90_daemon) showed 55min+ at the end and no crashes/errors. It happened on my iPad Mini 2, but my passenger had ForeFlight on his iPad Mini 4 and experienced zero disconnects.

Tagging this as a ForeFlight issue. It popped up in ForeFlight 9.2.

@cyoung
Copy link
Owner

cyoung commented Aug 22, 2017

@daveholland86 - which iPad were you using?

@pmaag
Copy link
Contributor

pmaag commented Aug 22, 2017

@cyoung , I wonder if this is hardware/iPad performance related. I also have a Mini 2 and see the same behavior. Do you have Hide Distant Traffic enabled? I am wondering if we hit a certain volume of ADS-B messages that just overwhelms the Mini 2.

My next flight I am going to try and hide Distant Traffic to see if I see the same disconnect messages. Theory is that hiding distant traffic might free up some resources on the iPad.

@cyoung
Copy link
Owner

cyoung commented Aug 22, 2017

Do not have "Hide Distant Traffic" enabled.

Thanks for your report, so far all the confirmed disconnects are on Mini 2's.

@jpuetthoff
Copy link

jpuetthoff commented Aug 22, 2017 via email

@pmaag
Copy link
Contributor

pmaag commented Aug 22, 2017

I am flying in a plane tomorrow that has a Stratus 2s. If the Mini 2 performance theory is accurate, I should see these disconnect messages tomorrow during that flight. Obviously the Stratus could be doing some device-side throttling, but will be an interesting test.

I'll report back after the flight tomorrow.

@cyoung
Copy link
Owner

cyoung commented Aug 22, 2017

@jpuetthoff - thanks for the report. Which Mini do you have? On the back of the iPad it should say "Model A____".

@jpuetthoff
Copy link

jpuetthoff commented Aug 22, 2017 via email

@pmaag
Copy link
Contributor

pmaag commented Aug 22, 2017

I am not sure this is a Foreflight issue. See the attached packet capture taken on the Stratux during an "ADS-B Disconnected" event. Here is my summary:

Foreflight reported the disconnect at 08-21-2017 22:01:14Z

Packet #10 in the capture is the last UDP packet sent from Stratux to the Foreflight client.
Packet #13 is the Stratux attempting to ping the iPad. Never receives a reply.
Packet #72 is the Stratux attempting to ping the iPad. Never receives a reply.
Packet #91, the iPad sends a broadcast UDP packet with DST port of 50113, content: "i want to play - ffm - udp" Appears to be some sort of heartbeat.
Packet #156 is the Stratux attempting to ping the iPad. Never receives a reply.
Packet #219 is the Stratux attempting to ping the iPad. Never receives a reply.
Packet #239 another iPad heartbeat (same as packet #91) to broadcast, so we know the iPad is on the network.
Packet #291 another Stratux ping to the iPad, no response received.
Packet #355 another Stratux ping to the iPad, no response received.
Packet #375 another iPad heartbeat to broadcast, iPad definitely on the network.
Packet #426 Stratux ICMP to iPad, this ICMP is replied to by the iPad.
Packet #435 Stratux starts sending UDP once again to the iPad.

So from the looks of it, the iPad will stop replying to ICMP messages during times of high CPU usage/performance issues? This causes the Stratux "heartbeat" to the iPad to drop, which stops the flow of UDP. Once the iPad starts replying to ICMP once again, things begin to flow correctly.

Suggestion: Use ARP instead of ICMP to test device reachability. Usually ICMP is the first protocol to be dropped once CPU usage gets high. ARP is normally not deprioritized and can simply be checked by issuing an ARP request and looking for the entry within the Stratux's ARP table.

Thoughts?

ads-b_disconnected_filtered.zip

@daveholland86
Copy link

Sorry, I just got notification today. I have a Mini 2 64gb.

@pmaag
Copy link
Contributor

pmaag commented Aug 25, 2017

@cyoung - I have been working on the code for this fix and have something working. Looking for some direction on how you want to proceed, as there are some trade-offs (at least for iOS clients):

Here is how my code currently is working (in the following order):

  1. If client has issued an ICMP Unreachable to UDP 4000 within the past 5 seconds, immediately report as sleeping.

else

  1. If client has sent a FF heartbeat broadcast in the past 10 seconds, report as awake.

else

  1. If client has not responded to ICMP request and has no MAC address in ARP table, report as sleeping.

else

  1. If client has responded to ICMP request within the past 30 seconds, and has a MAC address in ARP table, report as awake.

#2 & #4 are essentially the "second chance" for the older hardware that might be resource constrained at the ICMP layer but still active within the EFB.

The downside to #4 is that when a client is physically put to sleep (press of the power button), iOS has strange/inconsistent behavior. In my testing the iPad sometimes responds to ICMP (very intermittent and high latency), but always responds to ARP. This results in the client flapping between awake and sleep, since #4 marks the client as awake but then the client immediately issues ICMP unreachables when the stratux starts up the GDL90 stream (FF is not running). The flap is just a second, so the client will stay in sleep for 4 out of 5 seconds, but the flapping does occur.

Part of me thinks we should just have #2 and remove the check in #4, however other EFBs might not send a FF heartbeat (they likely don't), so we could see issues with those clients disconnecting when/if they start dropping ICMP due to hardware performance.

Any suggestions are appreciated. I am taking my code flying tonight and tomorrow and I definitely think it will fix the "ADS-B Disconnected" issue, however I don't sleep my iPad during flights so I am not sure how the flapping would affect me.

@pmaag
Copy link
Contributor

pmaag commented Aug 27, 2017

Just an FYI, flew for 4.4 the past two days with my updated code described above and received zero "ADS-B Disconnected" errors. Normally I would have seen at least 3-8 instances during that length of time.

@rgutlon
Copy link

rgutlon commented Aug 28, 2017

Just want to add my name to the list. Flew this Saturday on (2) 2 hour legs and for the first time started seeing the "ADS-B Disconnected" message in FF. These were longer flights than usual. Distant traffic hidden. However, I did have the Radar (Composite) Layer active (which is usually off). UI behavior as others indicated. At the time running 1.4r1 (just upgraded to r2) on a Mini 2 (32GB).

@AMNegron
Copy link

Throwing my name in. 1.4r2 / iPad Mini / FPG.
Flight 1.0 hour. Stratus shows running 1.5
Counted 8 disconnects in flight.

@mrdoornbos
Copy link

Still an issue yesterday. Since I was flying it was hard to troubleshoot the source of the problem.

@cyoung
Copy link
Owner

cyoung commented Oct 7, 2017

@mrdoornbos - which app? iPad Mini/Mini2?

@JohnOCFII
Copy link

I flew for about 90 minutes yesterday. I saw between 8 and 10 disconnects. iPad Pro 10.5 with iOS 11.0.2. I didn't see any disconnects for about 20 minutes into the flight, which made me wonder if it was heat related, even though I have the PWM controlled fan.

@jpuetthoff
Copy link

jpuetthoff commented Oct 7, 2017 via email

@rgutlon
Copy link

rgutlon commented Oct 9, 2017

Noticed this post on FF Facebook page under the topic "We're testing iOS 11.0.2 with ForeFlight" and it made me wonder if this actually a FF issue ...
https://www.facebook.com/pg/foreflight/posts/

I've experienced connectivity issues between Stratus 2S and Foreflight. I'm running the latest iOS. 2 hour night flight ... iPad showed the wifi connection was still active and online. Foreflight's device page showed Stratus connected and sharing data but the map said zero towers at 3500ft. GPS accuracy 5m. (iPad) No traffic on the map. If I killed the wifi from the settings and turned it back on, everything came back alive: 3 towers, lots of air targets, 1m accuracy. Had to do this three times in mid air. This happened at least once with the previous version of iOS. Only three flights so far with stratus.

@mrdoornbos
Copy link

@cyoung iPad Mini 4, Foreflight 9.4.1, iOS 11.0.2, Stratux 1.4r2

I'll grab it out of the plane on my way home and look at the logs.

@mrdoornbos
Copy link

I'll try it with a Trial version of Garmin Pilot just for fun on the same iPad to see if I see anything different.

@sethwebster
Copy link

Confirmed happening a couple of days ago on iPad Pro 10", v1.4r2, FF, 9.4.2.

@jpuetthoff
Copy link

jpuetthoff commented Oct 25, 2017 via email

@mrdoornbos
Copy link

This happened to me over and over on a 3 hours flight yesterday. Upgraded to 9.4.3 this morning and will test again on my return flight tomorrow.

@cyoung
Copy link
Owner

cyoung commented Oct 26, 2017

Summary: confirmed bug in ForeFlight 9.2+ was fixed in 9.4.3. Confirmed that it affected Stratux, Scout (officially supported by ForeFlight), and FreeFlight (officially supported by ForeFlight) from Jul 5th-Oct 25th. We didn't make any changes in Stratux during this time period that affected the issue.

I've reached out to try and establish some beta testing program so that Stratux users can participate to some extent in testing, or at least have someone's ear when there appear to be issues. Alas, such open communication with ForeFlight does not appear to be a possibility.

@cyoung cyoung closed this as completed Oct 26, 2017
@Nokomis449
Copy link

At some point they're bound to look in a mirror and notice the egg on their face. I use an EFB that supports Stratux, and can't help but chuckle at the denial going on over at FF. That Emperor ain't got no clothes on!

@flyinslow
Copy link

flyinslow commented Oct 27, 2017

I flew about an hour and a half last night with 9.4.3. The disconnection alerts were no longer popping up, but I never received traffic targets, and connection to towers was minimal at best. Fortunately, they're working on the issue.

@mrdoornbos
Copy link

3 hours today on my return trip Washington DC and Asheville NC with no disconnects after going to 9.4.3.

Can confirm it appears to be resolved.

@jzeevi
Copy link
Author

jzeevi commented Oct 28, 2017 via email

@captainkw
Copy link

Hi All, thanks @cyoung for the great open source Stratux! I just bought a pre-made one on Amazon and ran into the "ADS-Disconnected" problem on my ForeFlight repeatedly this month. My Stratux is with AHRS and was updated from v1.4r1 image to the latest dangerzone (http://updates.stratux.me/builds/dangerzone/update-stratux-v1.4r3-3b57564490.sh) update, on ForeFlight 9.4.3, and iOS 11.0.3, running on iPad 4 Mini 128GB version without WiFi.

I am not convinced this is 100% fixed yet as of today. Just came back with 2 flights this weekend (10/28 and 10/29) from Yosemite. Flight was following these VORs on v23: CZQ, EFH, GMN, 9,500 feet, then RNAV approach to SMO from DARTS intersection. The entire 2 hour flight the "ADS-B Disconnected" message keeps on popping up, at least 20 time during my flight. Any help would be greatly appreciated. Like earlier commenters, it does seem to disconnect when there is heavier amount of traffic. If I reboot the Stratux then it's good for about 10-15 minutes without disconnects, but then it would start disconnecting again after that.

@Nokomis449
Copy link

When you connect to the Stratux and look at the Status page at 192.168.10.1, how much disk space does it say is free?

@jzeevi
Copy link
Author

jzeevi commented Oct 30, 2017 via email

@Nokomis449
Copy link

"you" = @captainkw. I had a similar issue to his several versions back where Stratux would drop out, lose towers, etc. (I do NOT use ff). I eventually tracked it down to my SD card being nearly full. When I deleted the substantial log files that I had been accumulating, the issue went away. I believe the current releases accumulate some log AHRS log files even with logging turned off in the user interface, and if the user doesn't expand the SD card the card can fill up - especially when we (I'm guilty of it) leave it running at home as a "gee whiz, look at this!" novelty.

@captainkw
Copy link

captainkw commented Oct 31, 2017

Good pointers @Nokomis449, it seems like I did not expand the SD card to full capacity, even though I did not have AHRS logs on.
screen shot 2017-10-31 at 12 44 54 am

Either way, I followed the raspberry pi instructions here to ssh and used the sudo raspi-config config utility to expand my SD card to full size (16GB):
https://github.com/cyoung/stratux/wiki/SSH-into-Stratux
https://coderwall.com/p/mhj8jw/raspbian-how-to-resize-the-root-partition-to-fill-sd-card

It looks like it's fully expanded now:
screen shot 2017-10-31 at 12 59 28 am

It won't be a week or two before I am able to fly with the Stratux unit again, but if you don't hear from me again it can probably be assumed this issued is fixed. Thanks all again for the help.

@jzeevi
Copy link
Author

jzeevi commented Oct 31, 2017 via email

@Nokomis449
Copy link

Nokomis449 commented Oct 31, 2017

Now that you can ssh in to the Pi, log in and execute sudo rm -f /var/log/stratux*

This will delete every file in the /var/log/ directory that starts with "stratux". None of these files are necessary (it won't hurt anything to delete them) and they will be recreated the next time logging starts up. Otherwise you'll just continue to append log info to the existing files and continue to slowly eat up disk space.

@cyoung
Copy link
Owner

cyoung commented Oct 31, 2017

Looks like he had 170MB or so of free space, so I don't think that free space was causing the issue.

Another way to delete logs is to hit the version number a few times in rapid succession. Then a "Developer" tab will show up with "delete logfiles" functions.

@rgutlon
Copy link

rgutlon commented Oct 31, 2017

Chris - is there any benefit / downside to resizing the partition?

@cyoung
Copy link
Owner

cyoung commented Oct 31, 2017

Earlier versions of the AHRS added a lot of logging which caused free space to get used up quick. Recent .sh updates should remove the logs and don't have that issue so much. That should be sufficient, IMO. Expanding the partition is good for development setups. We had an auto-expanding release earlier on and it caused a higher rate of SD card failures.

@Nokomis449
Copy link

Nokomis449 commented Oct 31, 2017

When I had my issue, I had a little over 100MB of room left; that's why I think it might be the problem. This guy here
http://www.iflygps.com/Support/Forum/forumid/17/threadid/29528/scope/posts
bought a pre-assembled Stratux and ran into an issue that was described as a similar problem. Rather than fix it himself, he opted to get a new SD card. Two weeks later with the new card, he had the issue again. His screens say he has <50MB of space left, but the symptoms are similar. That's why I think it's disk space related. It'll be interesting to see if @captainkw 's issue is resolved by just expanding the partition.

@Nokomis449
Copy link

I didn't know an .sh update cleared the logs - that might explain why we don't see more of this issue.

@cyoung
Copy link
Owner

cyoung commented Oct 31, 2017

Ah, ok.

It does as of just a couple weeks ago. It's not in any of the release .sh updates but in the nightly builds.

http://updates.stratux.me/builds

@Nokomis449
Copy link

It might be a good idea to somehow get the word out that if you see a steady blinking green light, you should check the Status page for an explanation. I don't think very many owners even know about 192.168.10.1, especially with all the pre-built units being shipped.

@mikehaire
Copy link

I finally got a chance to fly this weekend after returning from vacation. Flew two flights, 1.0 & 1.4 hrs respectively. I had one "disconnect" on each of the flights that required me to reboot FF. So, it seems to be significantly improved but possibly still not 100%. Whats everyone else seeing since the 9.4.3 update??

@cyoung
Copy link
Owner

cyoung commented Nov 13, 2017

Seems to have settled quite a bit, haven't heard further reports since the update.

@shempferd
Copy link

Not sure if this is the same problem but I flew about 8 hours using avare on my phone and true flight on a cheetah. I noticed I lost signal to both units probably 5 times. I reset the stratux by unplugging each time and it came back every time.

@rgutlon
Copy link

rgutlon commented Nov 15, 2017

Interesting - flew 3.3 hrs on Sunday.
Two iPad Minis
My iPad had been updated to 11.1 and FF 9.4.3 - no disconnect messages
Co-Pilots iPad was still on 10.X and FF 9.4.3 - constant disconnect messages
Would have thought FF fix was backward compatible but the issue we all reported must be tied to the OS

@captainkw
Copy link

captainkw commented Nov 20, 2017

Looking back on this thread, looks like there are some people still having issues with specific iPad iOS versions and FF versions. I am happy to report that I flew 5.6 hours this weekend out of the busy SMO area to PAO and back, and had zero drops/disconnects from my Foreflight. My configuration:

  1. Pre-built Stratux from Everlasting Concepts: https://www.amazon.com/gp/product/B01LZTZ06C
  2. iPad 4 Mini 128GB, running iOS 11.1, FF 9.4.3
  3. Stratux Image: https://github.com/cyoung/stratux/releases/download/v1.4r3/stratux-v1.4r3-4de2420803.img.zip
  4. Dangerzone .sh for AHRS: http://updates.stratux.me/builds/dangerzone/update-stratux-v1.4r3-3b57564490.sh
  5. I also followed the following instructions to use the sudo raspi-config config utility to expand my SD card to full size (16GB):
    https://github.com/cyoung/stratux/wiki/SSH-into-Stratux
    https://coderwall.com/p/mhj8jw/raspbian-how-to-resize-the-root-partition-to-fill-sd-card

I saw there is a new dangerzone .sh on 11/18/2017 but I didn't use it as it came out after my trip began. Now that my setup is stable, is there a reason to update? What's new?

@cyoung
Copy link
Owner

cyoung commented Nov 21, 2017

@captainkw - commit log for those updates here. Biggest change was turning off the GPS LED for night ops.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests