Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] occasional crashes in v1.3.7: "panic: sync: WaitGroup is reused before previous Wait has returned" #9109

Closed
skladd opened this issue Nov 14, 2017 · 1 comment · Fixed by #9141
Assignees
Labels

Comments

@skladd
Copy link
Contributor

skladd commented Nov 14, 2017

InfluxDB: version 1.3.7 (from linux_armhf release tarball)
OS: Linux 4.1.41 on armv7l

I experienced these crashes several times, also for older versions. There are points written all the time, also deletes are performed regularly by cron jobs and scripts.

[I] 2017-11-13T08:58:56Z Snapshot for path /var/lib/influxdb/data/_internal/monitor/32 written in 1.303492817s engine=tsm1
[I] 2017-11-13T08:58:57Z beginning level 1 compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-11-13T08:58:57Z compacting level 1 group (0) /var/lib/influxdb/data/_internal/monitor/32/000000010-000000001.tsm (#0) engine=tsm1
[I] 2017-11-13T08:58:57Z compacting level 1 group (0) /var/lib/influxdb/data/_internal/monitor/32/000000011-000000001.tsm (#1) engine=tsm1
[I] 2017-11-13T08:59:00Z compacted level 1 group (0) into /var/lib/influxdb/data/_internal/monitor/32/000000011-000000002.tsm.tmp (#0) engine=tsm1
[I] 2017-11-13T08:59:00Z compacted level 1 2 files into 1 files in 2.476008924s engine=tsm1
[I] 2017-11-13T09:01:34Z retention policy shard deletion check commencing service=retention
[I] 2017-11-13T09:31:34Z retention policy shard deletion check commencing service=retention
panic: sync: WaitGroup is reused before previous Wait has returned

goroutine 288 [running]:
sync.(*WaitGroup).Wait(0x114d2018)
	/usr/local/go/src/sync/waitgroup.go:133 +0x100
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).disableLevelCompactions(0x114d2000, 0x11228100)
	/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:272 +0x68
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).SetCompactionsEnabled(0x114d2000, 0x8eb200)
	/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:207 +0x60
github.com/influxdata/influxdb/tsdb.(*Shard).SetCompactionsEnabled(0x11228100, 0x10d62d00)
	/go/src/github.com/influxdata/influxdb/tsdb/shard.go:424 +0x48
github.com/influxdata/influxdb/tsdb.(*Store).monitorShards(0x10d561c0)
	/go/src/github.com/influxdata/influxdb/tsdb/store.go:1247 +0x1d8
created by github.com/influxdata/influxdb/tsdb.(*Store).Open
	/go/src/github.com/influxdata/influxdb/tsdb/store.go:150 +0x21c
[I] 2017-11-14T02:01:23Z InfluxDB starting, version 1.3.7, branch 1.3, commit 2d474a3089bcfce6b472779be9470a1f0ef3d5e4
[I] 2017-11-14T02:01:23Z Go version go1.8.4, GOMAXPROCS set to 2

I changed some settings to reduce the memory footprint, in case they do matter:

query-log-enabled = false
cache-max-memory-size = 16777216 # 16 MB instead of 1048576000
reporting-disabled = true
logging-enabled = false
cache-snapshot-memory-size = 8388608 # 8 MB instead of 26214400
max-concurrent-queries = 50
query-timeout = "60s"
log-queries-after = "30s"
max-select-series = 1024
log-enabled = false
max-connection-limit = 20
@skladd skladd changed the title [Bug] occasional crashs in v1.3.7: "panic: sync: WaitGroup is reused before previous Wait has returned" [Bug] occasional crashes in v1.3.7: "panic: sync: WaitGroup is reused before previous Wait has returned" Nov 14, 2017
@ghost ghost assigned jwilder Nov 20, 2017
@ghost ghost added review and removed proposed labels Nov 20, 2017
@ghost ghost removed the review label Nov 22, 2017
@skladd
Copy link
Contributor Author

skladd commented Dec 4, 2017

Thank you very much for fixing.
Is version 1.5.0 to be released soon?
Could you otherwise please integrate the fix in a 1.4.x release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants