Panic in EC2 discovery

[kanand2]

What did you do? I restarted the ./prometheus

What did you expect to see? I had outage in monitoring.

What did you see instead? Under which circumstances?

Environment AWS Cloud

• System information: Linux 3.10.0-514.21.1.el7.x86_64 x86_64

• Prometheus version: prometheus-1.7.0.linux-amd64

• Alertmanager version: alertmanager-0.7.0.linux-amd64

• Logs:

time="2017-07-27T18:05:11-05:00" level=info msg="Done checkpointing in-memory metrics and chunks in 6.292170375s." source="persistence.go:665" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xa56329]
goroutine 163482690 [running]:
github.com/prometheus/prometheus/discovery/ec2.(*Discovery).refresh.func2(0xc492c8a620, 0x1, 0x0)
        /go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:195 +0x619
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages.func1(0x18f9580, 0xc492c8a620, 0x1, 0xc492c8a640)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6785 +0x49
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).EachPage(0xc9108caa80, 0xc64903fb80, 0x2, 0x2)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request/request_pagination.go:98 +0x90
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages(0xc88d12a450, 0x0, 0xc64903fca0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6786 +0x124
github.com/prometheus/prometheus/discovery/ec2.(*Discovery).refresh(0xc57914a340, 0xc57c076630, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:201 +0x319
github.com/prometheus/prometheus/discovery/ec2.(*Discovery).Run(0xc57914a340, 0x7f6e94c82310, 0xc4dff1b180, 0xc755fefe60)
        /go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:116 +0x1db
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
        /go/src/github.com/prometheus/prometheus/discovery/discovery.go:249 +0x283
time="2017-07-27T18:10:46-05:00" level=info msg="Starting prometheus (version=1.7.0, branch=master, revision=bfa37c8ee39d11078662dce16c162a61dccf616c)" source="main.go:88"
time="2017-07-27T18:10:46-05:00" level=info msg="Build context (go=go1.8.3, user=root@7a6329cc02bb, date=20170607-09:43:48)" source="main.go:89"
time="2017-07-27T18:10:46-05:00" level=info msg="Host details (Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat Apr 22 02:41:35 EDT 2017 x86_64 coaf-prommon (none))" source="main.go:90"
time="2017-07-27T18:10:46-05:00" level=info msg="Loading configuration file prommon.yml" source="main.go:252"
time="2017-07-27T18:10:46-05:00" level=info msg="Loading series map and head chunks..." source="storage.go:428"
time="2017-07-27T18:10:50-05:00" level=warning msg="Persistence layer appears dirty." source="persistence.go:846"
time="2017-07-27T18:10:50-05:00" level=warning msg="Starting crash recovery. Prometheus is inoperational until complete." source="crashrecovery.go:40"
time="2017-07-27T18:10:50-05:00" level=warning msg="To avoid crash recovery in the future, shut down Prometheus with SIGTERM or a HTTP POST to /-/quit." source="crashrecovery.go:41"
time="2017-07-27T18:10:50-05:00" level=info msg="Scanning files." source="crashrecovery.go:55"

[brian-brazil]

@beorn7

[fabxc]

Edited your initial issue for readability.

[beorn7]

That's in the ec2 SD code. Not really my construction site…

[danielfm]

I also just seen this in Prometheus v1.8.0.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xa25091]

goroutine 187 [running]:
github.com/prometheus/prometheus/discovery/ec2.(*Discovery).refresh.func2(0xc442e45080, 0x1, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:204 +0x551
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages.func1(0x1879aa0, 0xc442e45080, 0x1, 0xc4b99efd40)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6785 +0x49
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).EachPage(0xc4c0b94e00, 0xc77f4d3b60, 0x2, 0x2)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request/request_pagination.go:98 +0xa1
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages(0xc45a6d0898, 0x0, 0xc77f4d3ca0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6784 +0x11a
github.com/prometheus/prometheus/discovery/ec2.(*Discovery).refresh(0xc45ac12a00, 0xc4d6560ba0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:163 +0x3c8
github.com/prometheus/prometheus/discovery/ec2.(*Discovery).Run(0xc45ac12a00, 0x7f3cfff03598, 0xc4671222c0, 0xc4671d4120)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:119 +0x219
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:249 +0x26f

[grobie]

A server at SoundCloud also just crashed with this.

2017-11-03_06:11:31.91585 panic: runtime error: invalid memory address or nil pointer dereference
2017-11-03_06:11:31.91588 [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xa25091]
2017-11-03_06:11:31.91588 
2017-11-03_06:11:31.91589 goroutine 479 [running]:
2017-11-03_06:11:31.91590 github.com/prometheus/prometheus/discovery/ec2.(*Discovery).refresh.func2(0xc4735c5fa0, 0x1, 0x0)
2017-11-03_06:11:31.91591       /go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:204 +0x551
2017-11-03_06:11:31.91591 github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages.func1(0x1879aa0, 0xc4735c5fa0, 0x1, 0xc495118060)
2017-11-03_06:11:31.91592       /go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6785 +0x49
2017-11-03_06:11:31.91593 github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).EachPage(0xc4ae8b2700, 0xc4a88ffb60, 0x2, 0x2)
2017-11-03_06:11:31.91593       /go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request/request_pagination.go:98 +0xa1
2017-11-03_06:11:31.91594 github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages(0xc48427e3f8, 0x0, 0xc48752bca0, 0x0, 0x0)
2017-11-03_06:11:31.91595       /go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6784 +0x11a
2017-11-03_06:11:31.91596 github.com/prometheus/prometheus/discovery/ec2.(*Discovery).refresh(0xc4403fd6d0, 0xc5143daae0, 0x0, 0x0)
2017-11-03_06:11:31.91596       /go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:163 +0x3c8
2017-11-03_06:11:31.91597 github.com/prometheus/prometheus/discovery/ec2.(*Discovery).Run(0xc4403fd6d0, 0x7f21e342a340, 0xc4684f57c0, 0xc45219dce0)
2017-11-03_06:11:31.91597       /go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:119 +0x219
2017-11-03_06:11:31.91598 created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
2017-11-03_06:11:31.91598       /go/src/github.com/prometheus/prometheus/discovery/discovery.go:249 +0x26f

[beorn7]

@mjibson @federicobaldo @dominikschulz @mtanda have touched this code. Any help appreciated.

[dominikschulz]

The code is making several assumptions about pointers not being nil, not sure if we should check all of them against nil.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.