Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

service panic because of race conditions #13674

Closed
rvdwijngaard opened this issue Oct 15, 2018 · 5 comments

Comments

Projects
None yet
3 participants
@rvdwijngaard
Copy link

commented Oct 15, 2018

Grafana v5.3.0 (commit: c613a31)
Datasource: Cloudwatch
OS : linux

I have upgraded my server to the latest version. The service now panics with a random frequency. I cannot find a pattern yet. In the logs I get the following stack trace:

fatal error: concurrent map writes
goroutine 12697 [running]:
runtime.throw(0x1523ebe, 0x15)
/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc0009c7ea0 sp=0xc0009c7e70 pc=0x4353a2
runtime.mapassign_faststr(0x12ec080, 0xc00134c7b0, 0xc00121f0c0, 0x19, 0x0)
/usr/local/go/src/runtime/map_faststr.go:275 +0x3bf fp=0xc0009c7f08 sp=0xc0009c7ea0 pc=0x41b93f
github.com/grafana/grafana/pkg/tsdb/cloudwatch.(*CloudWatchExecutor).executeTimeSeriesQuery.func1(0x8, 0x1590fa0)
/go/src/github.com/grafana/grafana/pkg/tsdb/cloudwatch/cloudwatch.go:138 +0x1a2 fp=0xc0009c7f88 sp=0xc0009c7f08 pc=0xe8c272
github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00134c810, 0xc001356380)
/go/src/github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57 fp=0xc0009c7fd0 sp=0xc0009c7f88 pc=0xb2f347
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0009c7fd8 sp=0xc0009c7fd0 pc=0x465221
created by github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup.(*Group).Go
/go/src/github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66
@marefr

This comment has been minimized.

Copy link
Member

commented Oct 15, 2018

@rvdwijngaard Please include full dashboard JSON if you think that's related.

@marefr

This comment has been minimized.

Copy link
Member

commented Oct 15, 2018

@mtanda seems related to this change, but the only way I think above can happen would be if there are multiple queries with the same RefId, right?

@mtanda

This comment has been minimized.

Copy link
Collaborator

commented Oct 15, 2018

@marefr I guess the same RefId is not a problem.
I should protect map by mutex. It's a common mistake about golang map...

@marefr marefr added type/bug and removed needs more detail labels Oct 15, 2018

@marefr

This comment has been minimized.

Copy link
Member

commented Oct 15, 2018

@mtanda yes you're right. But then this is not a new problem, since the same line of code existed before v5.3.

@mtanda

This comment has been minimized.

Copy link
Collaborator

commented Oct 16, 2018

@marefr The line is derived from another PR.
It is released at 5.3 beta.

https://github.com/grafana/grafana/pull/11624/files#diff-ed5cc68bef51197743c5863b078db25bR124

#13684 will fix this issue, please check.

@marefr marefr modified the milestones: 5.3.1, 5.3.2 Oct 16, 2018

marefr added a commit that referenced this issue Oct 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.