Memory leak kills some Stratux services after 2.5 hours

[ssokol]

I've done another all-day test, this time with 0.3b4 stock. It was running with a single R820T2 and a RY835AI. UAT, GPS and AHRS were enabled.

The system was started at 13:05 and continued to log until 18:17, so just over five hours. The Stratux web UI stopped updating the "Uptime" value after 2 hours, 25 minutes which roughly correlates with the time that ForeFlight stopped displaying GPS and ADS-B information and lost the "FreeFlight" data feed. The "Stratux" data feed remained and continued to provide attitude and altitude until the experiment was terminated.

As seems to be the case when this occurs, memory usage was very high. I was able to log in and capture memory usage, CPU, etc. (see below) but was unable to successfully kill the gen_gdl90 process. Attempting to do so - first by attaching to the screen session and issuing a Ctr+c, then using the kill command - left the process defunct but still resident in memory. I attempted a clean shutdown using "poweroff" but the system hung during the shutdown.

Stratux.log is available here: https://gist.github.com/ssokol/34a6eca6b9d0016640c0

The other Stratux logs are available if you need them.

[bradanlane]

We may be able to narrow this down a bit. I ran Stratux for 6hrs non-stop with two SDRs but no GPS/AHRS. WebUI available the entire time. I have logs if there is anything to look for.

On Sep 24, 2015, at 6:26 PM, Steven Sokol notifications@github.com wrote:

I've done another all-day test, this time with 0.3b4 stock. It was running with a single R820T2 and a RY835AI. UAT, GPS and AHRS were enabled.

The system was started at 13:05 and continued to log until 18:17, so just over five hours. The Stratux web UI stopped updating the "Uptime" value after 2 hours, 25 minutes which roughly correlates with the time that ForeFlight stopped displaying GPS and ADS-B information and lost the "FreeFlight" data feed. The "Stratux" data feed remained and continued to provide attitude and altitude until the experiment was terminated.

As seems to be the case when this occurs, memory usage was very high. I was able to log in and capture memory usage, CPU, etc. (see below) but was unable to successfully kill the gen_gdl90 process. Attempting to do so - first by attaching to the screen session and issuing a Ctr+c, then using the kill command - left the process defunct but still resident in memory. I attempted a clean shutdown using "poweroff" but the system hung during the shutdown.

Stratux.log is available here: https://gist.github.com/ssokol/34a6eca6b9d0016640c0

The other Stratux logs are available if you need them.

—
Reply to this email directly or view it on GitHub.

[cyoung]

Thanks @ssokol and @bradanlane.

For future reference, memprof:

 func dumpHeapProfileAndExit() {
         t := time.NewTicker(10 * time.Minute)
         <-t.C
         f, err := os.Create("/tmp/gen_gdl90.mem.prof")
         if err != nil {
                 log.Printf("err: %s\n", err.Error())
                 return
         }
         pprof.WriteHeapProfile(f)
         f.Close()
         os.Exit(1)
 }

Then just threw "go dumpHeapProfileAndExit()" in main().

 apt-get --fix-missing install graphviz
 go tool pprof --pdf gen_gdl90 /tmp/gen_gdl90.mem.prof >/var/www/memprof.pdf

Before:

After: