panic: runtime error: makeslice: len out of range

[ymettier]

Hello,

InfluxDB is crashed and fails to restart after a no space left on device.
I have a message panic: runtime error: makeslice: len out of range

• Are my data corrupted ?
• If yes, how do I recover ?
• How do I restart InfluxDB ?

@benbjohnson : is this related to issue #3261 you are working on ?

$ rpm -q influxdb
influxdb-0.9.2-1.x86_64
$ /opt/influxdb/influx --version
InfluxDB shell 0.9.2

Important configuration items (but this should not be related to the problem) :

[meta]
  ...
  peers = ["host1:8088", "host2:8088", "host3:8088"]
[retention]
  enabled = true
  check-interval = "10m"
  replication=3

Other configuration items were left unmodified (default values) except the listener.

Here is the log (pay attention to the first 2 lines).

[write] 2015/08/16 06:33:58 write failed for shard 1 on node 3: write /var/opt/influxdb/data/poc/default/1: no space left on device
[write] 2015/08/16 06:33:58 write failed for shard 1 on node 3: write /var/opt/panic: runtime error: makeslice: len out of range

goroutine 1 [running]:
github.com/influxdb/influxdb/services/hh.(*segment).current(0xc2080e22d0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/queue.go:499 +0x200
github.com/influxdb/influxdb/services/hh.(*queue).Open(0xc20802e6c0, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/queue.go:113 +0x246
github.com/influxdb/influxdb/services/hh.(*Processor).addQueue(0xc20802e660, 0x3, 0x0, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/processor.go:99 +0x37e
github.com/influxdb/influxdb/services/hh.(*Processor).loadQueues(0xc20802e660, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/processor.go:79 +0x1b5
github.com/influxdb/influxdb/services/hh.NewProcessor(0xc2080d0740, 0x14, 0x7f147ba22d10, 0xc2080dc580, 0x40000000, 0x0, 0x40f8ae, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/processor.go:49 +0x3e4
github.com/influxdb/influxdb/services/hh.NewService(0x1, 0xc2080d0740, 0x14, 0x40000000, 0x2260ff9290000, 0x0, 0x3b9aca00, 0x7f147ba22d10, 0xc2080dc580, 0x1)
        /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:46 +0x23f
github.com/influxdb/influxdb/cmd/influxd/run.NewServer(0xc2080a6b00, 0xc2f7ac, 0x5, 0xc2080a6b00, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:104 +0x6df
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run(0xc208011b90, 0xc20800a010, 0x4, 0x4, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:91 +0x602
main.(*Main).Run(0xc208097a80, 0xc20800a010, 0x4, 0x4, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:76 +0x3bc
main.main()
        /root/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:38 +0xdc

goroutine 6 [syscall]:
os/signal.loop()
        /root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
        /root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:27 +0x35

In my case, the root cause is obvious (1st line of log).

However, InfluxDB is now stopped and will not restart. Here is the log after a failed restart :

panic: runtime error: makeslice: len out of range

goroutine 1 [running]:
github.com/influxdb/influxdb/services/hh.(*segment).current(0xc2080e42d0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/queue.go:499 +0x200
github.com/influxdb/influxdb/services/hh.(*queue).Open(0xc2080466c0, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/queue.go:113 +0x246
github.com/influxdb/influxdb/services/hh.(*Processor).addQueue(0xc208046660, 0x3, 0x0, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/processor.go:99 +0x37e
github.com/influxdb/influxdb/services/hh.(*Processor).loadQueues(0xc208046660, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/processor.go:79 +0x1b5
github.com/influxdb/influxdb/services/hh.NewProcessor(0xc20806a740, 0x14, 0x7fd0dcac2c10, 0xc2080dc580, 0x40000000, 0x0, 0x40f8ae, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/services/hh/processor.go:49 +0x3e4
github.com/influxdb/influxdb/services/hh.NewService(0x1, 0xc20806a740, 0x14, 0x40000000, 0x2260ff9290000, 0x0, 0x3b9aca00, 0x7fd0dcac2c10, 0xc2080dc580, 0x1)
        /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:46 +0x23f
github.com/influxdb/influxdb/cmd/influxd/run.NewServer(0xc2080a0b00, 0xc2f7ac, 0x5, 0xc2080a0b00, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:104 +0x6df
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run(0xc208011c00, 0xc20800a010, 0x4, 0x4, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:91 +0x602
main.(*Main).Run(0xc208091a80, 0xc20800a010, 0x4, 0x4, 0x0, 0x0)
        /root/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:76 +0x3bc
main.main()
        /root/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:38 +0xdc

goroutine 6 [syscall]:
os/signal.loop()
        /root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
        /root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:27 +0x35

Because this was a POC with 3 days of data (on a very small FS with less than 5 Gb), I will restart from scratch. So I will not be able to give more information on this issue. I'm sorry about that.
Feel free to close if you think there is not enough information to help or if this is a duplicate of issue #3261.

Regards,
Yves