panic: runtime error: index out of range #148

allaire · 2018-05-30T21:18:27Z

I'm getting the following error on Ubuntu 16.04 using upstart. Everything was working correctly until recently after (I think) an apt upgrade. Any hints? When I comment my .inq file below I can start it successfully.

[Unit]
PartOf=agendrix-web.target

[Service]
User=agendrix
Environment=PORT=%i
Environment="PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$HOME/.rbenv/plugins/ruby_build/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"
...
ExecStart=/bin/bash -lc 'exec bundle exec puma'
ExecReload=/bin/kill -USR1 $MAINPID
Restart=always
StandardInput=null
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n
KillMode=mixed
TimeoutStopSec=5

check service agendrix-web@5000.service
  if memory:total_rss > 2g for 4 cycles then alert, reload

May 31 12:31:08 app-01 inspeqtor[3078]: Copyright © 2018 Contributed Systems LLC
May 31 12:31:08 app-01 inspeqtor[3078]: Licensed under the GNU Public License 3.0
May 31 12:31:08 app-01 inspeqtor[3078]: Want more? Upgrade to Inspeqtor Pro for more features and support.
May 31 12:31:08 app-01 inspeqtor[3078]: See http://contribsys.com/inspeqtor for details.
May 31 12:31:08 app-01 inspeqtor[3078]: I 2018-05-31T12:31:08.794821Z 3078 Detected upstart in /etc/init
May 31 12:31:08 app-01 inspeqtor[3078]: I 2018-05-31T12:31:08.794857Z 3078 Detected systemd in /etc/systemd
May 31 12:31:08 app-01 inspeqtor[3078]: I 2018-05-31T12:31:08.794945Z 3078 Detected init.d in /etc/init.d/
May 31 12:31:08 app-01 inspeqtor[3078]: I 2018-05-31T12:31:08.804974Z 3078 Expose now available at port 4677
May 31 12:31:08 app-01 inspeqtor[3078]: I 2018-05-31T12:31:08.809090Z 3078 Found systemd/agendrix-web@5000.service with status Up/1357
May 31 12:31:08 app-01 inspeqtor[3078]: panic: runtime error: index out of range
May 31 12:31:08 app-01 inspeqtor[3078]: goroutine 15 [running]:
May 31 12:31:08 app-01 inspeqtor[3078]: panic(0x764f60, 0xc42000e060)
May 31 12:31:08 app-01 inspeqtor[3078]:         /usr/local/Cellar/go/1.7.1/libexec/src/runtime/panic.go:500 +0x1a1
May 31 12:31:08 app-01 inspeqtor[3078]: github.com/mperham/inspeqtor/metrics.(*processStorage).captureVM(0xc42000abe0, 0x54d, 0xc4200b3201, 0x0)
May 31 12:31:08 app-01 inspeqtor[3078]:         /Users/mike/src/github.com/mperham/inspeqtor/metrics/process.go:361 +0x2fc
May 31 12:31:08 app-01 inspeqtor[3078]: github.com/mperham/inspeqtor/metrics.(*processStorage).Collect(0xc42000abe0, 0x54d, 0x7c4a34, 0x2)
May 31 12:31:08 app-01 inspeqtor[3078]:         /Users/mike/src/github.com/mperham/inspeqtor/metrics/process.go:226 +0x8b
May 31 12:31:08 app-01 inspeqtor[3078]: github.com/mperham/inspeqtor.(*Service).Collect(0xc4200d65d0, 0x0, 0xc4200ccc30)
May 31 12:31:08 app-01 inspeqtor[3078]:         /Users/mike/src/github.com/mperham/inspeqtor/types.go:148 +0x21d
May 31 12:31:08 app-01 inspeqtor[3078]: created by github.com/mperham/inspeqtor.(*Inspeqtor).scan
May 31 12:31:08 app-01 inspeqtor[3078]:         /Users/mike/src/github.com/mperham/inspeqtor/inspeqtor.go:295 +0x217
May 31 12:31:08 app-01 systemd[1]: inspeqtor.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 31 12:31:08 app-01 systemd[1]: inspeqtor.service: Unit entered failed state.
May 31 12:31:08 app-01 systemd[1]: inspeqtor.service: Failed with result 'exit-code'.
May 31 12:31:10 app-01 systemd[1]: inspeqtor.service: Service hold-off time over, scheduling restart.
May 31 12:31:10 app-01 systemd[1]: Stopped Inspeqtor.
May 31 12:31:10 app-01 systemd[1]: inspeqtor.service: Start request repeated too quickly.
May 31 12:31:10 app-01 systemd[1]: Failed to start Inspeqtor.

The text was updated successfully, but these errors were encountered:

allaire · 2018-05-31T12:44:12Z

@gmartsenkov are you having the same issue?

I'm unsure if this could have been caused by some kind of recent OS upgrade. It was working fine prior to running apt upgrade on my servers.

mperham · 2018-05-31T20:21:57Z

The code is reading /proc/<pid>/status and looking for the VmRSS line. That error would occur if there was an empty line in the output. Can you show me what /proc/1357/status looks like?

allaire · 2018-05-31T20:26:37Z

@mperham Hi Mike, here you go:

Name:	ruby
State:	S (sleeping)
Tgid:	1357
Ngid:	0
Pid:	1357
PPid:	1
TracerPid:	0
Uid:	1001	1001	1001	1001
Gid:	1002	1002	1002	1002
FDSize:	256
Groups:	1001 1002
NStgid:	1357
NSpid:	1357
NSpgid:	1357
NSsid:	1357
VmPeak:	  200632 kB
VmSize:	  140688 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	   17856 kB
VmRSS:	   17856 kB
VmData:	   78352 kB
VmStk:	    8188 kB
VmExe:	    3188 kB
VmLib:	    6704 kB
VmPTE:	     156 kB
VmPMD:	      12 kB
VmSwap:	       0 kB
HugetlbPages:	       0 kB
Threads:	3
SigQ:	0/7930
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	00000001c2317e4f
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000003fffffffff
CapAmb:	0000000000000000
Seccomp:	0

Speculation_Store_Bypass:	vulnerable
Cpus_allowed:	7fff
Cpus_allowed_list:	0-14
Mems_allowed:	00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	13074
nonvoluntary_ctxt_switches:	415

mperham · 2018-05-31T20:27:22Z

Yep, there's the blank line.

allaire · 2018-05-31T20:28:05Z

Oh I see you made a fix, awesome! My ansible role is using apt to install. Are you going to cut a release as well?

Thanks so much!

mperham · 2018-05-31T20:29:44Z

I'm working on #146 and will release 2.0 with the Pro functionality sometime soon.

allaire · 2018-05-31T20:32:00Z

Great, and in the meantime, is there a way to patch the issue on my side?

allaire · 2018-05-31T21:09:11Z

@mperham So I compiled by hand and now I'm getting this:

May 31 21:07:22 app-01 inspeqtor[16443]: panic: runtime error: index out of range
May 31 21:07:22 app-01 inspeqtor[16443]: goroutine 15 [running]:
May 31 21:07:22 app-01 inspeqtor[16443]: github.com/mperham/inspeqtor/metrics.totalRssCollector(0x54d, 0xc42001c7d0, 0x0, 0x0)
May 31 21:07:22 app-01 inspeqtor[16443]:         /Users/allaire/.go/src/github.com/mperham/inspeqtor/metrics/process.go:158 +0x785
May 31 21:07:22 app-01 inspeqtor[16443]: github.com/mperham/inspeqtor/metrics.(*processStorage).Collect(0xc42001c7d0, 0x54d, 0x848dc0, 0xc420065ef0)
May 31 21:07:22 app-01 inspeqtor[16443]:         /Users/allaire/.go/src/github.com/mperham/inspeqtor/metrics/process.go:238 +0x12f
May 31 21:07:22 app-01 inspeqtor[16443]: github.com/mperham/inspeqtor.(*Service).Collect(0xc420065ef0, 0x847900, 0xc4200532a0)
May 31 21:07:22 app-01 inspeqtor[16443]:         /Users/allaire/.go/src/github.com/mperham/inspeqtor/types.go:148 +0x1a1
May 31 21:07:22 app-01 inspeqtor[16443]: created by github.com/mperham/inspeqtor.(*Inspeqtor).scan
May 31 21:07:22 app-01 inspeqtor[16443]:         /Users/allaire/.go/src/github.com/mperham/inspeqtor/inspeqtor.go:293 +0x186
May 31 21:07:22 app-01 systemd[1]: inspeqtor.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 31 21:07:22 app-01 systemd[1]: inspeqtor.service: Unit entered failed state.
May 31 21:07:22 app-01 systemd[1]: inspeqtor.service: Failed with result 'exit-code'.
May 31 21:07:23 app-01 systemd[1]: inspeqtor.service: Service hold-off time over, scheduling restart.
May 31 21:07:23 app-01 systemd[1]: Stopped Inspeqtor.
May 31 21:07:23 app-01 systemd[1]: inspeqtor.service: Start request repeated too quickly.

allaire · 2018-05-31T21:15:41Z

So I'm no Go expert (never used it before), but I took your fix:

if len(line) == 0 {
  continue
}

and I copied it where I found for _, line := range lines { in process.go, and now it works. 🤷‍♂️

mperham · 2018-05-31T21:51:42Z

Good call, updated.

gmartsenkov · 2018-06-01T11:18:30Z

I was getting a similar error, but for nginx - basically, if the status endpoint does not exist inspeqtor fails to start with a vague error, had to dig in through the code to figure it out

mperham · 2018-06-01T19:18:38Z

Any idea why /status wouldn’t exist?

…

On Jun 1, 2018, at 04:18, Georgi Martsenkov ***@***.***> wrote: I was getting a similar error, but for nginx - basically, if the status endpoint does not exist inspeqtor fails to start with a vague error, had to dig in through the code to figure it out — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

allaire · 2018-06-01T19:26:53Z

@mperham AFAIK ngx_http_stub_status_module is not built/setup by default.

http://nginx.org/en/docs/http/ngx_http_stub_status_module.html
https://www.datadoghq.com/blog/how-to-collect-nginx-metrics/

Maybe @gmartsenkov thought that nginx exposed this endpoint by default.

mperham · 2018-06-01T19:32:24Z

Oh, I was thinking of /etc/<pid>/status. Nevermind. Didn't realize you were talking about nginx-specific metric collection. That's a separate issue, not this one.

felixbuenemann · 2018-06-19T18:16:05Z

I was also running into this with inspeqtor-1.0.1-1 on Ubuntu Xenial and Linux 4.4.0-128-generic.

Compiling Inspeqtor from master and replacing the binary on the server fixed the problem.

kesha-antonov · 2018-06-19T20:31:47Z

+1 have same issue

kesha-antonov · 2018-06-19T20:35:31Z

sudo service inspeqtor status
● inspeqtor.service - Inspeqtor
   Loaded: loaded (/lib/systemd/system/inspeqtor.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Tue 2018-06-19 20:31:37 UTC; 2s ago
  Process: 19189 ExecStart=/usr/bin/inspeqtor -c /etc/inspeqtor -s /var/run/inspeqtor.sock (code=exited, status=2)
  Process: 19186 ExecStartPre=/bin/rm -f /var/run/inspeqtor.sock (code=exited, status=0/SUCCESS)
 Main PID: 19189 (code=exited, status=2)

Jun 19 20:31:36 ip-172-31-20-112 inspeqtor[19189]:         /Users/mike/src/github.com/mperham/inspeqtor/types.go:148 +0x21d
Jun 19 20:31:36 ip-172-31-20-112 inspeqtor[19189]: created by github.com/mperham/inspeqtor.(*Inspeqtor).scan
Jun 19 20:31:36 ip-172-31-20-112 inspeqtor[19189]:         /Users/mike/src/github.com/mperham/inspeqtor/inspeqtor.go:295 +0x217
Jun 19 20:31:36 ip-172-31-20-112 systemd[1]: inspeqtor.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 19 20:31:36 ip-172-31-20-112 systemd[1]: inspeqtor.service: Unit entered failed state.
Jun 19 20:31:36 ip-172-31-20-112 systemd[1]: inspeqtor.service: Failed with result 'exit-code'.
Jun 19 20:31:37 ip-172-31-20-112 systemd[1]: inspeqtor.service: Service hold-off time over, scheduling restart.
Jun 19 20:31:37 ip-172-31-20-112 systemd[1]: Stopped Inspeqtor.
Jun 19 20:31:37 ip-172-31-20-112 systemd[1]: inspeqtor.service: Start request repeated too quickly.
Jun 19 20:31:37 ip-172-31-20-112 systemd[1]: Failed to start Inspeqtor.

kesha-antonov · 2018-06-19T20:35:50Z

How to fix it?

kesha-antonov · 2018-06-19T20:38:22Z

inspeqtor -v
Inspeqtor 1.0.1

felixbuenemann · 2018-06-19T20:59:11Z

How to fix it?

Install golang and make
Clone the inspeqtor repo
Run make
Copy inspeqtor to /usr/bin/inspeqtor
Restart inspeqtor service

Or wait for a new release…

mperham · 2018-06-19T21:19:12Z

Master is now 2.0.0-rc. If someone wants to rebuild, smoke test it on their server and give me a thumbs up, I'll release 2.0.0.

felixbuenemann · 2018-06-20T09:50:20Z

I've installed the latest build from master on a server, I'll see how it goes.

mperham closed this as completed in 803bb96 May 31, 2018

mperham added a commit that referenced this issue May 31, 2018

Continuation of #148

42e9f59

allaire mentioned this issue Jun 28, 2018

Getting a swap alert when swap is at 0 #150

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic: runtime error: index out of range #148

panic: runtime error: index out of range #148

allaire commented May 30, 2018 •

edited

Loading

allaire commented May 31, 2018 •

edited

Loading

mperham commented May 31, 2018

allaire commented May 31, 2018 •

edited

Loading

mperham commented May 31, 2018

allaire commented May 31, 2018

mperham commented May 31, 2018

allaire commented May 31, 2018

allaire commented May 31, 2018

allaire commented May 31, 2018 •

edited by mperham

Loading

mperham commented May 31, 2018

gmartsenkov commented Jun 1, 2018

mperham commented Jun 1, 2018 via email

allaire commented Jun 1, 2018 •

edited

Loading

mperham commented Jun 1, 2018 •

edited

Loading

felixbuenemann commented Jun 19, 2018

kesha-antonov commented Jun 19, 2018

kesha-antonov commented Jun 19, 2018

kesha-antonov commented Jun 19, 2018

kesha-antonov commented Jun 19, 2018

felixbuenemann commented Jun 19, 2018

mperham commented Jun 19, 2018

felixbuenemann commented Jun 20, 2018

panic: runtime error: index out of range #148

panic: runtime error: index out of range #148

Comments

allaire commented May 30, 2018 • edited Loading

allaire commented May 31, 2018 • edited Loading

mperham commented May 31, 2018

allaire commented May 31, 2018 • edited Loading

mperham commented May 31, 2018

allaire commented May 31, 2018

mperham commented May 31, 2018

allaire commented May 31, 2018

allaire commented May 31, 2018

allaire commented May 31, 2018 • edited by mperham Loading

mperham commented May 31, 2018

gmartsenkov commented Jun 1, 2018

mperham commented Jun 1, 2018 via email

allaire commented Jun 1, 2018 • edited Loading

mperham commented Jun 1, 2018 • edited Loading

felixbuenemann commented Jun 19, 2018

kesha-antonov commented Jun 19, 2018

kesha-antonov commented Jun 19, 2018

kesha-antonov commented Jun 19, 2018

kesha-antonov commented Jun 19, 2018

felixbuenemann commented Jun 19, 2018

mperham commented Jun 19, 2018

felixbuenemann commented Jun 20, 2018

allaire commented May 30, 2018 •

edited

Loading

allaire commented May 31, 2018 •

edited

Loading

allaire commented May 31, 2018 •

edited

Loading

allaire commented May 31, 2018 •

edited by mperham

Loading

allaire commented Jun 1, 2018 •

edited

Loading

mperham commented Jun 1, 2018 •

edited

Loading