[0.3b2] gen_gdl90 defunct process

[n4ohj]

I tend to leave the Stratux running. Consistently I have noticed the web interface hangs and the system stops responding to foreflight if it has been left running long enough. I believe this time the web interface showed about 17hours before it stopped responding.

1898 pts/0 Zl+ 1007:15 [gen_gdl90] <defunct>

pi@raspberrypi ~ $ free -m
total used free shared buffers cached
Mem: 927 914 13 0 6 22
-/+ buffers/cache: 885 41
Swap: 99 21 78

Granted I will not sit in an airplane for 17 hours with my stratux, but it might indicate a lower level problem.

I have not rebooted the Stratux, if there is any other detail that would be helpful.

[AlexSzlavik]

Anything interesting in dmesg, /var/log/messages or /var/log/stratux.log ?

[n4ohj]

I see a number of references in kern.log like this:

kern.log:Sep 17 20:22:01 raspberrypi kernel: [17520.741910] INFO: task gen_gdl90:2377 blocked for more than 120 seconds.
kern.log:Sep 17 20:22:01 raspberrypi kernel: [17520.741951] gen_gdl90 D 80521498 0 2377 1868 0x00000000
kern.log:Sep 17 20:24:01 raspberrypi kernel: [17640.741681] INFO: task gen_gdl90:2377 blocked for more than 120 seconds.
kern.log:Sep 17 20:24:01 raspberrypi kernel: [17640.741721] gen_gdl90 D 80521498 0 2377 1868 0x00000000

In Stratux.log, I see many pages of message queue overflows:
2015/09/17 00:39:56 192.168.10.10:43211 - message queue overflow.
2015/09/17 00:39:56 192.168.10.10:4000 - message queue overflow.
2015/09/17 00:39:56 192.168.10.11:43211 - message queue overflow.
2015/09/17 00:39:56 192.168.10.10:43211 - message queue overflow.
2015/09/17 00:39:56 192.168.10.11:43211 - message queue overflow.

I don't know the mechanism for aging out client devices, but I don't think both devices (iphone and ipad) were still connected during this period, which may have resulted in the queue overflow.

Those are the only items that look interesting to me.

[cyoung]

It's using ICMP pings to determine if the clients are responding, if not then it should not be queueing messages for them. Sounds like it may have run out of resources. Are you able to replicate this at all? If so, can you check "free -m; df -h", and other messages in dmesg?

[ssokol]

FWIW - I tried this as an experiment yesterday. I left a system running the v0.3b3 image all day and overnight. By mid-afternoon ForeFlight had stopped seeing the "FreeFlight" device but the AHRS continued to work. This morning both the "FreeFlight" and the "Stratux" devices were gone. When I tried to SSH in to take a look I was able to connect but kept getting memory allocation errors whenever the shell tried to fork any other process. Pretty much a sure sign of a memory leak.

I'm running the experiment again today and will monitor the memory usage periodically.

[cyoung]

Try with latest code - referenced commit (post-v0.3b3) above improves memory usage and there was a further commit that writes some vital stats (including memory usage) to /var/log/stratux.log.

[n4ohj]

I will give it a shot. Thank you.

On Wed, Sep 23, 2015 at 7:36 AM, cyoung notifications@github.com wrote:

Try with latest code - referenced commit (post-v0.3b3) above improves
memory usage and there was a further commit that writes some vital stats
(including memory usage) to /var/log/stratux.log.

—
Reply to this email directly or view it on GitHub
#48 (comment).

[cyoung]

Anyone want to throw memprof on it before trying to reproduce it?

[ssokol]

Having some kind of problem compiling current code. Output of make is:

root@raspberrypi:~/stratux# make
GOOS=linux GOARCH=arm GOARM=7 go build -ldflags " -X main.stratuxVersion=`git describe --abbrev=0 --tags` -X main.stratuxBuild=`git log -n 1 --pretty=%H`" main/gen_gdl90.go main/traffic.go main/ry835ai.go main/network.go main/managementinterface.go main/sdr.go
# command-line-arguments
usage: 5l [options] main.5
  -1    use alternate profiling code
  -B info
        define ELF NT_GNU_BUILD_ID note
  -C    check Go calls to C code
  -D addr
        data address
  -E sym
        entry symbol
  -G    debug pseudo-ops
  -I interp
        set ELF interp
  -L dir
        add dir to library path
  -H head
        header type
  -K    add stack underflow checks
  -M    disable software div/mod
  -O    print pc-line tables
  -Q    debug byte-register code gen
  -P    debug code generation
  -R rnd
        address rounding
  -S    check type signatures
  -T addr
        text address
  -V    print version and exit
  -W    disassemble input
  -X name value
        define string data
  -Z    clear stack frame on entry
  -a    disassemble output
  -c    dump call graph
  -d    disable dynamic executable
  -extld ld
        linker to run in external mode
  -extldflags ldflags
        flags for external linker
  -f    ignore version mismatch
  -g    disable go package data checks
  -installsuffix suffix
        pkg directory suffix
  -k sym
        set field tracking symbol
  -linkmode mode
        set link mode (internal, external, auto)
  -n    dump symbol table
  -o outfile
        set output file
  -r dir1:dir2:...
        set ELF dynamic linker search path
  -race
        enable race detector
  -s    disable symbol table
  -shared
        generate shared object (implies -linkmode external)
  -tmpdir dir
        leave temporary files in this directory
  -u    reject unsafe packages
  -v    print link trace
  -w    disable DWARF generation
Makefile:7: recipe for target 'all' failed
make: *** [all] Error 2

[cyoung2]

That has to do win the -ldflags arg -- think it might have to do with an
older go version. I've got it going on 1.5.1. Check path also

On Wednesday, September 23, 2015, Steven Sokol notifications@github.com
wrote:

Having some kind of problem compiling current code. Output of make is:

root@raspberrypi:~/stratux# make
GOOS=linux GOARCH=arm GOARM=7 go build -ldflags " -X main.stratuxVersion=git describe --abbrev=0 --tags -X main.stratuxBuild=git log -n 1 --pretty=%H" main/gen_gdl90.go main/traffic.go main/ry835ai.go main/network.go main/managementinterface.go main/sdr.go

command-line-arguments

usage: 5l [options] main.5
-1 use alternate profiling code
-B info
define ELF NT_GNU_BUILD_ID note
-C check Go calls to C code
-D addr
data address
-E sym
entry symbol
-G debug pseudo-ops
-I interp
set ELF interp
-L dir
add dir to library path
-H head
header type
-K add stack underflow checks
-M disable software div/mod
-O print pc-line tables
-Q debug byte-register code gen
-P debug code generation
-R rnd
address rounding
-S check type signatures
-T addr
text address
-V print version and exit
-W disassemble input
-X name value
define string data
-Z clear stack frame on entry
-a disassemble output
-c dump call graph
-d disable dynamic executable
-extld ld
linker to run in external mode
-extldflags ldflags
flags for external linker
-f ignore version mismatch
-g disable go package data checks
-installsuffix suffix
pkg directory suffix
-k sym
set field tracking symbol
-linkmode mode
set link mode (internal, external, auto)
-n dump symbol table
-o outfile
set output file
-r dir1:dir2:...
set ELF dynamic linker search path
-race
enable race detector
-s disable symbol table
-shared
generate shared object (implies -linkmode external)
-tmpdir dir
leave temporary files in this directory
-u reject unsafe packages
-v print link trace
-w disable DWARF generation
Makefile:7: recipe for target 'all' failed
make: *** [all] Error 2

—
Reply to this email directly or view it on GitHub
#48 (comment).

[cyoung]

https://github.com/cyoung/stratux/releases/tag/v0.3b4

memprof if able

[cyoung]

Any further info @ssokol?

[ssokol]

Oops. Created a new issue with logs, screen-shots, etc.
#60

On Thu, Sep 24, 2015 at 4:38 PM, cyoung notifications@github.com wrote:

Any further info @ssokol https://github.com/ssokol?

—
Reply to this email directly or view it on GitHub
#48 (comment).

Steven Sokol
408 Camelot Drive
Liberty, MO 64068

mobile: +1 816-806-8844
fax: +1 816-817-0441