Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.12.2 cannot restart influxdb after a no space left on device #7488

Closed
Lahorde opened this issue Oct 19, 2016 · 3 comments
Closed

0.12.2 cannot restart influxdb after a no space left on device #7488

Lahorde opened this issue Oct 19, 2016 · 3 comments

Comments

@Lahorde
Copy link

Lahorde commented Oct 19, 2016

I did not get any space left on infludb storage disk. So I did some cleaning. A big influxdb log file was stored in /var/log/ , I deleted it and configured logrotate to rotate it.

But now I cannot restart influxdb :

[tsm1wal] 2016/10/19 22:56:09 tsm1 WAL writing to /var/lib/influxdb/wal/teleinfo/default/106
[filestore] 2016/10/19 22:56:09 /var/lib/influxdb/data/teleinfo/default/106/000000004-000000003.tsm (#0) opened in 202.291µs
[cacheloader] 2016/10/19 22:56:09 reading file /var/lib/influxdb/wal/teleinfo/default/106/_05149.wal, size 0
[tsm1] 2016/10/19 22:56:09 /var/lib/influxdb/data/teleinfo/default/106 database index loaded in 188.801µs
[store] 2016/10/19 22:56:09 /var/lib/influxdb/data/teleinfo/default/106 opened in 4.095078ms
[cacheloader] 2016/10/19 22:56:11 reading file /var/lib/influxdb/wal/_internal/monitor/116/_05145.wal, size 0
[tsm1] 2016/10/19 22:56:11 /var/lib/influxdb/data/_internal/monitor/116 database index loaded in 89.703179ms
[store] 2016/10/19 22:56:11 /var/lib/influxdb/data/_internal/monitor/116 opened in 2.141215739s
[cacheloader] 2016/10/19 22:56:17 reading file /var/lib/influxdb/wal/home_data/default/109/_00002.wal, size 10485777
[cacheloader] 2016/10/19 22:56:24 reading file /var/lib/influxdb/wal/home_data/default/109/_00003.wal, size 1613771
[run] 2016/10/19 22:56:25 InfluxDB starting, version 0.12.2, branch unknown, commit 383332daed5595926c235f250b11433f67229c35
[run] 2016/10/19 22:56:25 Go version go1.6.2, GOMAXPROCS set to 4
[run] 2016/10/19 22:56:25 Using configuration at: /etc/influxdb/influxdb.conf
[store] 2016/10/19 22:56:25 Using data dir: /var/lib/influxdb/data
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/teleinfo/default/106
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/103
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/105
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/teleinfo/default/106/000000004-000000003.tsm (#0) opened in 226.77µs
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/teleinfo/default/12
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/teleinfo/default/106/_05150.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/teleinfo/default/106 database index loaded in 319.112µs
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/teleinfo/default/12/000000005-000000002.tsm (#0) opened in 188.02µs
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/teleinfo/default/106 opened in 6.894539ms
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/103/000000003-000000002.tsm (#0) opened in 1.553532ms
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/105/000000003-000000002.tsm (#0) opened in 2.2416ms
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/teleinfo/default/12/_05171.wal, size 0
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/103/_05157.wal, size 0
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/105/_05157.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/teleinfo/default/12 database index loaded in 198.071µs
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/teleinfo/default/12 opened in 8.589998ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/104
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/112
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/104/000000003-000000002.tsm (#0) opened in 1.041503ms
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/104/_05157.wal, size 0
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/112/000000002-000000002.tsm (#0) opened in 2.43139ms
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/112/_00009.wal, size 10492819
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/103 database index loaded in 139.67265ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/103 opened in 149.514358ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/107
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/107/000000004-000000003.tsm (#0) opened in 7.196152ms
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/107/_05159.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/105 database index loaded in 158.929766ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/105 opened in 168.520487ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/108
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/108/000000004-000000003.tsm (#0) opened in 961.348µs
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/108/_05159.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/104 database index loaded in 153.91209ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/104 opened in 172.189942ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/111
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/111/000000004-000000003.tsm (#0) opened in 1.200565ms
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/111/_05159.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/108 database index loaded in 123.31713ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/108 opened in 128.25465ms
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/107 database index loaded in 132.755663ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/102
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/102/000000003-000000002.tsm (#0) opened in 1.111139ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/107 opened in 151.52747ms
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/102/_05157.wal, size 0
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/114
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/114/_00001.wal, size 13386
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/111 database index loaded in 141.448941ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/111 opened in 156.587646ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/113
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/113/_00001.wal, size 13007
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/114/_05147.wal, size 0
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/113/_05147.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/102 database index loaded in 136.198245ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/102 opened in 142.916067ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/100
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/home_data/default/100/000000001-000000001.tsm (#0) opened in 246.144µs
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/home_data/default/100/_05153.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/home_data/default/100 database index loaded in 1.103847ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/home_data/default/100 opened in 6.563552ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/2
[filestore] 2016/10/19 22:56:25 /var/lib/influxdb/data/home_data/default/2/000000002-000000002.tsm (#0) opened in 38.232048ms
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/home_data/default/2/_05171.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/114 database index loaded in 132.654258ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/114 opened in 193.104705ms
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/home_data/default/2 database index loaded in 719.735µs
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/home_data/default/2 opened in 49.616402ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/116
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/_internal/monitor/116/_00001.wal, size 1174993
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/109
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/home_data/default/109/_00001.wal, size 10485804
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/113 database index loaded in 144.270694ms
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/_internal/monitor/113 opened in 197.642696ms
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:25 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/115
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/home_data/default/115/_00001.wal, size 109736
[cacheloader] 2016/10/19 22:56:25 reading file /var/lib/influxdb/wal/home_data/default/115/_05146.wal, size 0
[tsm1] 2016/10/19 22:56:25 /var/lib/influxdb/data/home_data/default/115 database index loaded in 393.539µs
[store] 2016/10/19 22:56:25 /var/lib/influxdb/data/home_data/default/115 opened in 135.852518ms
[cacheloader] 2016/10/19 22:56:27 reading file /var/lib/influxdb/wal/_internal/monitor/116/_05146.wal, size 0
[tsm1] 2016/10/19 22:56:27 /var/lib/influxdb/data/_internal/monitor/116 database index loaded in 90.398225ms
[store] 2016/10/19 22:56:27 /var/lib/influxdb/data/_internal/monitor/116 opened in 2.01444974s
[cacheloader] 2016/10/19 22:56:34 reading file /var/lib/influxdb/wal/home_data/default/109/_00002.wal, size 10485777
[cacheloader] 2016/10/19 22:56:41 reading file /var/lib/influxdb/wal/_internal/monitor/112/_00010.wal, size 10498277
[cacheloader] 2016/10/19 22:56:41 reading file /var/lib/influxdb/wal/home_data/default/109/_00003.wal, size 1613771
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/home_data/default/109/_00004.wal, size 4044
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/home_data/default/109/_00005.wal, size 4044
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/home_data/default/109/_00006.wal, size 36864
panic: runtime error: makeslice: len out of range

goroutine 49 [running]:
panic(0x60cc20, 0x134e1458)
    /usr/lib/go/src/runtime/panic.go:481 +0x330
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WriteWALEntry).UnmarshalBinary(0x134e1450, 0x13a9c000, 0x3b, 0x100000, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:564 +0x2cc
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WALSegmentReader).Next(0x10db4c44, 0x131dc400)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:804 +0x714
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load.func1(0x10db4c88, 0x10db4d14, 0x10f800c0, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:444 +0x338
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load(0x10db4d14, 0x10f800c0, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:428 +0xac
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).reloadCache(0x112bca10, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:663 +0x1b4
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).Open(0x112bca10, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:159 +0x174
github.com/influxdata/influxdb/tsdb.(*Shard).Open.func1(0x113fe000, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/shard.go:149 +0x1d0
github.com/influxdata/influxdb/tsdb.(*Shard).Open(0x113fe000, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/shard.go:129 +0x28
github.com/influxdata/influxdb/tsdb.(*Store).loadShards.func1(0x10d5c988, 0x10d1ccf0, 0x10d90080, 0x10d11fc0, 0x10d848f7, 0x9, 0x10d771f1, 0x7, 0x10d77349, 0x3)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/store.go:162 +0x51c
created by github.com/influxdata/influxdb/tsdb.(*Store).loadShards
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/store.go:170 +0x5ec
[run] 2016/10/19 22:56:43 InfluxDB starting, version 0.12.2, branch unknown, commit 383332daed5595926c235f250b11433f67229c35
[run] 2016/10/19 22:56:43 Go version go1.6.2, GOMAXPROCS set to 4
[run] 2016/10/19 22:56:43 Using configuration at: /etc/influxdb/influxdb.conf
[store] 2016/10/19 22:56:43 Using data dir: /var/lib/influxdb/data
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/2
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/100
[filestore] 2016/10/19 22:56:43 /var/lib/influxdb/data/home_data/default/2/000000002-000000002.tsm (#0) opened in 251.925µs
[filestore] 2016/10/19 22:56:43 /var/lib/influxdb/data/home_data/default/100/000000001-000000001.tsm (#0) opened in 166.614µs
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/home_data/default/2/_05172.wal, size 0
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/home_data/default/100/_05154.wal, size 0
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/109
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/115
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/home_data/default/109/_00001.wal, size 10485804
[tsm1] 2016/10/19 22:56:43 /var/lib/influxdb/data/home_data/default/2 database index loaded in 1.275356ms
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/home_data/default/115/_00001.wal, size 109736
[store] 2016/10/19 22:56:43 /var/lib/influxdb/data/home_data/default/2 opened in 7.136569ms
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1] 2016/10/19 22:56:43 /var/lib/influxdb/data/home_data/default/100 database index loaded in 3.451123ms
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/116
[store] 2016/10/19 22:56:43 /var/lib/influxdb/data/home_data/default/100 opened in 10.528474ms
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/teleinfo/default/106
[filestore] 2016/10/19 22:56:43 /var/lib/influxdb/data/teleinfo/default/106/000000004-000000003.tsm (#0) opened in 250.727µs
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/teleinfo/default/106/_05151.wal, size 0
[tsm1] 2016/10/19 22:56:43 /var/lib/influxdb/data/teleinfo/default/106 database index loaded in 3.832059ms
[store] 2016/10/19 22:56:43 /var/lib/influxdb/data/teleinfo/default/106 opened in 17.066401ms
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/_internal/monitor/116/_00001.wal, size 1174993
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/teleinfo/default/12
[filestore] 2016/10/19 22:56:43 /var/lib/influxdb/data/teleinfo/default/12/000000005-000000002.tsm (#0) opened in 14.364231ms
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/teleinfo/default/12/_05172.wal, size 0
[tsm1] 2016/10/19 22:56:43 /var/lib/influxdb/data/teleinfo/default/12 database index loaded in 339.373µs
[store] 2016/10/19 22:56:43 /var/lib/influxdb/data/teleinfo/default/12 opened in 43.989406ms
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/102
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/home_data/default/115/_05147.wal, size 0
[tsm1] 2016/10/19 22:56:43 /var/lib/influxdb/data/home_data/default/115 database index loaded in 407.394µs
[store] 2016/10/19 22:56:43 /var/lib/influxdb/data/home_data/default/115 opened in 212.835824ms
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:43 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/103
[filestore] 2016/10/19 22:56:43 /var/lib/influxdb/data/_internal/monitor/103/000000003-000000002.tsm (#0) opened in 965.15µs
[cacheloader] 2016/10/19 22:56:43 reading file /var/lib/influxdb/wal/_internal/monitor/103/_05158.wal, size 0
[filestore] 2016/10/19 22:56:43 /var/lib/influxdb/data/_internal/monitor/102/000000003-000000002.tsm (#0) opened in 13.866369ms
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/102/_05158.wal, size 0
[tsm1] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/103 database index loaded in 167.996164ms
[store] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/103 opened in 250.687145ms
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/108
[filestore] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/108/000000004-000000003.tsm (#0) opened in 910.567µs
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/108/_05160.wal, size 0
[tsm1] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/102 database index loaded in 152.126318ms
[store] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/102 opened in 403.28674ms
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/111
[filestore] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/111/000000004-000000003.tsm (#0) opened in 991.921µs
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/111/_05160.wal, size 0
[tsm1] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/108 database index loaded in 117.295239ms
[store] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/108 opened in 122.8714ms
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/112
[filestore] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/112/000000002-000000002.tsm (#0) opened in 928.587µs
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/112/_00009.wal, size 10492819
[tsm1] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/111 database index loaded in 123.327231ms
[store] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/111 opened in 129.536462ms
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/104
[filestore] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/104/000000003-000000002.tsm (#0) opened in 1.106972ms
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/104/_05158.wal, size 0
[tsm1] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/104 database index loaded in 95.397357ms
[store] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/104 opened in 102.289293ms
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/113
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/113/_00001.wal, size 13007
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/113/_05148.wal, size 0
[tsm1] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/113 database index loaded in 85.060028ms
[store] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/113 opened in 174.128676ms
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/105
[filestore] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/105/000000003-000000002.tsm (#0) opened in 1.073222ms
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/105/_05158.wal, size 0
[tsm1] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/105 database index loaded in 102.533614ms
[store] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/105 opened in 107.965871ms
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/114
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/114/_00001.wal, size 13386
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/114/_05148.wal, size 0
[tsm1] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/114 database index loaded in 136.062202ms
[store] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/114 opened in 165.910448ms
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:44 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/107
[filestore] 2016/10/19 22:56:44 /var/lib/influxdb/data/_internal/monitor/107/000000004-000000003.tsm (#0) opened in 1.557854ms
[cacheloader] 2016/10/19 22:56:44 reading file /var/lib/influxdb/wal/_internal/monitor/107/_05160.wal, size 0
[tsm1] 2016/10/19 22:56:45 /var/lib/influxdb/data/_internal/monitor/107 database index loaded in 160.459077ms
[store] 2016/10/19 22:56:45 /var/lib/influxdb/data/_internal/monitor/107 opened in 167.02216ms
[cacheloader] 2016/10/19 22:56:45 reading file /var/lib/influxdb/wal/_internal/monitor/116/_05147.wal, size 0
[tsm1] 2016/10/19 22:56:45 /var/lib/influxdb/data/_internal/monitor/116 database index loaded in 45.470334ms
[store] 2016/10/19 22:56:45 /var/lib/influxdb/data/_internal/monitor/116 opened in 1.961951445s
[cacheloader] 2016/10/19 22:56:50 reading file /var/lib/influxdb/wal/home_data/default/109/_00002.wal, size 10485777
[cacheloader] 2016/10/19 22:56:57 reading file /var/lib/influxdb/wal/home_data/default/109/_00003.wal, size 1613771
[cacheloader] 2016/10/19 22:56:57 reading file /var/lib/influxdb/wal/_internal/monitor/112/_00010.wal, size 10498277
[cacheloader] 2016/10/19 22:56:58 reading file /var/lib/influxdb/wal/home_data/default/109/_00004.wal, size 4044
[cacheloader] 2016/10/19 22:56:58 reading file /var/lib/influxdb/wal/home_data/default/109/_00005.wal, size 4044
[cacheloader] 2016/10/19 22:56:58 reading file /var/lib/influxdb/wal/home_data/default/109/_00006.wal, size 36864
panic: runtime error: makeslice: len out of range

goroutine 24 [running]:
panic(0x60cc20, 0x11bbb640)
    /usr/lib/go/src/runtime/panic.go:481 +0x330
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WriteWALEntry).UnmarshalBinary(0x11bbb638, 0x1229c000, 0x3b, 0x100000, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:564 +0x2cc
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WALSegmentReader).Next(0x10c31c44, 0x115c1f00)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:804 +0x714
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load.func1(0x10c31c88, 0x10c31d14, 0x10d9a060, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:444 +0x338
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load(0x10c31d14, 0x10d9a060, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:428 +0xac
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).reloadCache(0x10daa000, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:663 +0x1b4
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).Open(0x10daa000, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:159 +0x174
github.com/influxdata/influxdb/tsdb.(*Shard).Open.func1(0x10d86000, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/shard.go:149 +0x1d0
github.com/influxdata/influxdb/tsdb.(*Shard).Open(0x10d86000, 0x0, 0x0)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/shard.go:129 +0x28
github.com/influxdata/influxdb/tsdb.(*Store).loadShards.func1(0x10d1b208, 0x10d38000, 0x10d30d40, 0x10d30c80, 0x10d35e57, 0x9, 0x10d36f81, 0x7, 0x10d370d9, 0x3)
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/store.go:162 +0x51c
created by github.com/influxdata/influxdb/tsdb.(*Store).loadShards
    /home/remi/tmp/influxdb/src/src/github.com/influxdata/influxdb/tsdb/store.go:170 +0x5ec
[run] 2016/10/19 22:56:58 InfluxDB starting, version 0.12.2, branch unknown, commit 383332daed5595926c235f250b11433f67229c35
[run] 2016/10/19 22:56:58 Go version go1.6.2, GOMAXPROCS set to 4
[run] 2016/10/19 22:56:58 Using configuration at: /etc/influxdb/influxdb.conf
[store] 2016/10/19 22:56:58 Using data dir: /var/lib/influxdb/data
[tsm1wal] 2016/10/19 22:56:58 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:58 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/102
[tsm1wal] 2016/10/19 22:56:58 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:58 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/105
[tsm1wal] 2016/10/19 22:56:58 tsm1 WAL starting with 10485760 segment size
[filestore] 2016/10/19 22:56:58 /var/lib/influxdb/data/_internal/monitor/102/000000003-000000002.tsm (#0) opened in 888.327µs
[cacheloader] 2016/10/19 22:56:58 reading file /var/lib/influxdb/wal/_internal/monitor/102/_05159.wal, size 0
[filestore] 2016/10/19 22:56:58 /var/lib/influxdb/data/_internal/monitor/105/000000003-000000002.tsm (#0) opened in 990.775µs
[tsm1wal] 2016/10/19 22:56:58 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:58 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/103
[cacheloader] 2016/10/19 22:56:58 reading file /var/lib/influxdb/wal/_internal/monitor/105/_05159.wal, size 0
[tsm1wal] 2016/10/19 22:56:58 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/104
[filestore] 2016/10/19 22:56:58 /var/lib/influxdb/data/_internal/monitor/103/000000003-000000002.tsm (#0) opened in 1.166868ms
[cacheloader] 2016/10/19 22:56:58 reading file /var/lib/influxdb/wal/_internal/monitor/103/_05159.wal, size 0
[filestore] 2016/10/19 22:56:58 /var/lib/influxdb/data/_internal/monitor/104/000000003-000000002.tsm (#0) opened in 1.177179ms
[cacheloader] 2016/10/19 22:56:58 reading file /var/lib/influxdb/wal/_internal/monitor/104/_05159.wal, size 0
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/102 database index loaded in 109.148101ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/102 opened in 115.081397ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/113
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/113/_00001.wal, size 13007
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/103 database index loaded in 117.19456ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/103 opened in 127.376214ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/107
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/105 database index loaded in 123.979464ms
[filestore] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/107/000000004-000000003.tsm (#0) opened in 942.754µs
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/107/_05161.wal, size 0
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/105 opened in 133.531955ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/108
[filestore] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/108/000000004-000000003.tsm (#0) opened in 1.665615ms
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/108/_05161.wal, size 0
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/113/_05149.wal, size 0
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/104 database index loaded in 136.562301ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/104 opened in 160.839384ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/111
[filestore] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/111/000000004-000000003.tsm (#0) opened in 9.29317ms
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/111/_05161.wal, size 0
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/107 database index loaded in 107.947068ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/107 opened in 115.427905ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/teleinfo/default/12
[filestore] 2016/10/19 22:56:59 /var/lib/influxdb/data/teleinfo/default/12/000000005-000000002.tsm (#0) opened in 215.259µs
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/teleinfo/default/12/_05173.wal, size 0
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/teleinfo/default/12 database index loaded in 293.696µs
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/teleinfo/default/12 opened in 13.023353ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/114
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/114/_00001.wal, size 13386
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/108 database index loaded in 114.420828ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/108 opened in 129.752344ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/112
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/113 database index loaded in 113.714687ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/113 opened in 154.289686ms
[filestore] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/112/000000002-000000002.tsm (#0) opened in 1.559469ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/_internal/monitor/116
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/112/_00009.wal, size 10492819
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/116/_00001.wal, size 1174993
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/111 database index loaded in 117.049926ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/111 opened in 137.671981ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/115
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/home_data/default/115/_00001.wal, size 109736
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/_internal/monitor/114/_05149.wal, size 0
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/114 database index loaded in 174.412057ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/_internal/monitor/114 opened in 221.792847ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/2
[filestore] 2016/10/19 22:56:59 /var/lib/influxdb/data/home_data/default/2/000000002-000000002.tsm (#0) opened in 348.383µs
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/home_data/default/115/_05148.wal, size 0
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/home_data/default/2/_05173.wal, size 0
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/home_data/default/2 database index loaded in 1.277961ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/home_data/default/2 opened in 36.105966ms
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/home_data/default/115 database index loaded in 1.674156ms
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/home_data/default/115 opened in 216.166842ms
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/teleinfo/default/106
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/100
[filestore] 2016/10/19 22:56:59 /var/lib/influxdb/data/teleinfo/default/106/000000004-000000003.tsm (#0) opened in 202.342µs
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/teleinfo/default/106/_05152.wal, size 0
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/teleinfo/default/106 database index loaded in 199.947µs
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/teleinfo/default/106 opened in 5.201425ms
[filestore] 2016/10/19 22:56:59 /var/lib/influxdb/data/home_data/default/100/000000001-000000001.tsm (#0) opened in 223.645µs
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL starting with 10485760 segment size
[tsm1wal] 2016/10/19 22:56:59 tsm1 WAL writing to /var/lib/influxdb/wal/home_data/default/109
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/home_data/default/100/_05155.wal, size 0
[cacheloader] 2016/10/19 22:56:59 reading file /var/lib/influxdb/wal/home_data/default/109/_00001.wal, size 10485804
[tsm1] 2016/10/19 22:56:59 /var/lib/influxdb/data/home_data/default/100 database index loaded in 983.327µs
[store] 2016/10/19 22:56:59 /var/lib/influxdb/data/home_data/default/100 opened in 12.639658ms
@Lahorde Lahorde changed the title 0.12.2 cannot restart influxdb after an out of space 0.12.2 cannot restart influxdb after a no space left on device Oct 19, 2016
@jwilder
Copy link
Contributor

jwilder commented Oct 19, 2016

Looks like either your /var/lib/influxdb/wal/home_data/default/109/_00006.wal or /var/lib/influxdb/wal/home_data/default/109/_00007.wal wal segments may have a corrupted block that is not being handled correctly. This was fixed in #6483. I'd suggest upgrading to the 1.0 releases or you can move those segments out of the way.

@jwilder jwilder closed this as completed Oct 19, 2016
@Lahorde
Copy link
Author

Lahorde commented Oct 20, 2016

OK, after deleting /var/lib/influxdb/wal/home_data/default/109/_00006.wal I can restart influxdb. In order to estimate consequences, does that mean that some failing writes (when disk was full) logs have been deleted, so these points are definitively lost?

@jwilder
Copy link
Contributor

jwilder commented Oct 20, 2016

If you deleted one of the wal segments as opposed to upgrading, then the points in that segment are lost. You can find the range of time by looking at show shards and see what time range is covered by shard 109 in you home_data, default db & rp.

Writes that came in while the disk was full would return an error to your client and would not have been stored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants