Snapshot - panic: not implemented yet

[trobers]

Hi,

i have a big problem with influxDB and Snapshots. All time influxDB tries to create a snapshot of database it crashes with the following message.

2015/06/17 09:25:12 [INFO] raft: Starting snapshot up to 8210
2015/06/17 09:25:12 [INFO] snapshot: Creating new snapshot at /var/opt/influxdb/meta/snapshots/2-8210-1434525912419.tmp
panic: not implemented yet

goroutine 12 [running]:
github.com/influxdb/influxdb/meta.(_storeFSMSnapshot).Persist(0xc2081146f0, 0x7f41d0793870, 0xc2080303c0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/meta/store.go:1592 +0x76
github.com/hashicorp/raft.(_Raft).takeSnapshot(0xc2080b61c0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:1674 +0x826
github.com/hashicorp/raft.(_Raft).runSnapshots(0xc2080b61c0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:1602 +0x105
github.com/hashicorp/raft._Raft.(github.com/hashicorp/raft.runSnapshots)·fm()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:243 +0x27
github.com/hashicorp/raft.func·011()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/state.go:153 +0xe3

I tried to deactivate the generating of snapshots by config.

[snapshot]
enabled = false # Enabled by default if not set.

But this has no effect. InfluxDB still tries generate snapshots. In the consequence I start InfluxDB and five minutes later it crashes. It crashes until I delete manually all databases from filesystem and restart InfluxDB

Is there anything I could try or it is a bug?

[maxcanada]

Same Issue here, ubuntu 14.04, influx 0.9.0

It seems to happen every now and then when I send a post request to the influx server (every 3 minutes or so). Here is a sample of what my post request looks like (each post request contains about 100 lines) :

arb,instrument=USD,scope=outer,exchange=ffff ask_spread=-15,bid_spread=23,timestamp=1434898978489999000 1434898978489999000
arb,instrument=EUR,scope=inner,exchange=ffff ask_spread=36,bid_spread=-88,timestamp=1434898978489999000 1434898978489999000
arb,instrument=EUR,scope=outer,exchange=ffff ask_spread=-55,bid_spread=69,timestamp=1434898978489999000 1434898978489999000
lag,target=open_orders,exchange=ffff time_to_receive=10805.418,time_to_send=4.306,timestamp=1434898980475000000 1434898980475000000
arb,instrument=USD,scope=inner,exchange=ffff ask_spread=-20,bid_spread=-20,timestamp=1434898983492999000 1434898983492999000
arb,instrument=USD,scope=outer,exchange=ffff ask_spread=0,bid_spread=0,timestamp=1434898983492999000 1434898983492999000
arb,instrument=USD,scope=inner,exchange=ffff ask_spread=-13,bid_spread=-13,timestamp=1434898983492999000 1434898983492999000

And here are the influx logs :

[http] 2015/06/21 16:02:23 127.0.0.1 - - [21/Jun/2015:16:02:22 +0100] POST /write?db=db HTTP/1.1 204 0 - python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-54-generic 85556529-1826-11e5-8013-000000000000 636.963818ms
[http] 2015/06/21 16:02:35 127.0.0.1 - - [21/Jun/2015:16:02:35 +0100] POST /write?db=db HTTP/1.1 204 0 - python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-54-generic 8c91ba18-1826-11e5-8014-000000000000 730.619837ms
[http] 2015/06/21 16:02:43 127.0.0.1 - - [21/Jun/2015:16:02:43 +0100] POST /write?db=db HTTP/1.1 204 0 - python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-54-generic 9170f34d-1826-11e5-8015-000000000000 612.34569ms
[http] 2015/06/21 16:02:53 127.0.0.1 - - [21/Jun/2015:16:02:53 +0100] POST /write?db=db HTTP/1.1 204 0 - python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-54-generic 9786a8ee-1826-11e5-8016-000000000000 525.915268ms
[http] 2015/06/21 16:03:09 127.0.0.1 - - [21/Jun/2015:16:03:08 +0100] POST /write?db=db HTTP/1.1 204 0 - python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-54-generic a09f9204-1826-11e5-8017-000000000000 629.729239ms
2015/06/21 16:03:17 [INFO] raft: Starting snapshot up to 8239
2015/06/21 16:03:17 [INFO] snapshot: Creating new snapshot at /var/opt/influxdb/meta/snapshots/18-8239-1434898997144.tmp
panic: not implemented yet

goroutine 12 [running]:
github.com/influxdb/influxdb/meta.(_storeFSMSnapshot).Persist(0xc208288250, 0x7f47a80fea48, 0xc208142140, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/meta/store.go:1592 +0x76
github.com/hashicorp/raft.(_Raft).takeSnapshot(0xc2080b81c0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:1674 +0x826
github.com/hashicorp/raft.(_Raft).runSnapshots(0xc2080b81c0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:1602 +0x105
github.com/hashicorp/raft._Raft.(github.com/hashicorp/raft.runSnapshots)·fm()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:243 +0x27
github.com/hashicorp/raft.func·011()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 1 [chan receive (nil chan), 3 minutes]:
main.(*Main).Run(0xc20801d760, 0xc20800a010, 0x4, 0x4, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/main.go:81 +0x450
main.main()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/main.go:39 +0x1d0

goroutine 6 [syscall, 3 minutes]:
os/signal.loop()
/root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:27 +0x35

goroutine 18 [chan receive, 3 minutes]:
github.com/influxdb/influxdb/tcp.(_listener).Accept(0xc20803a7d8, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/meta.(_Store).serveExecListener(0xc208074000)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/meta/store.go:436 +0x8f
created by github.com/influxdb/influxdb/meta.(*Store).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/meta/store.go:183 +0x1c2

goroutine 17 [syscall, 3 minutes, locked to thread]:
runtime.goexit()
/root/.gvm/gos/go1.4.2/src/runtime/asm_amd64.s:2232 +0x1

goroutine 8 [IO wait, 3 minutes]:
net.(_pollDesc).Wait(0xc2080103e0, 0x72, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(_pollDesc).WaitRead(0xc2080103e0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(_netFD).accept(0xc208010380, 0x0, 0x7f47abcfccc8, 0xc2080f8000)
/root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(_TCPListener).AcceptTCP(0xc20803a7c8, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(_TCPListener).Accept(0xc20803a7c8, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
github.com/influxdb/influxdb/tcp.(_Mux).Serve(0xc2080bb890, 0x7f47abcff310, 0xc20803a7c8, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/tcp/mux.go:45 +0x57
created by github.com/influxdb/influxdb/cmd/influxd/run.func·003
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:274 +0x762

goroutine 9 [chan receive, 3 minutes]:
github.com/influxdb/influxdb/tcp.(_listener).Accept(0xc20803a7d0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/meta.(_raftLayer).Accept(0xc2080bb8c0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/meta/store.go:1636 +0x64
github.com/hashicorp/raft.(*NetworkTransport).listen(0xc208010460)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/net_transport.go:346 +0x50
created by github.com/hashicorp/raft.NewNetworkTransport
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/net_transport.go:138 +0x2c4

goroutine 10 [select]:
github.com/hashicorp/raft.(_Raft).leaderLoop(0xc2080b81c0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:807 +0xc7c
github.com/hashicorp/raft.(_Raft).runLeader(0xc2080b81c0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:779 +0x74e
github.com/hashicorp/raft.(_Raft).run(0xc2080b81c0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:576 +0xc9
github.com/hashicorp/raft._Raft.(github.com/hashicorp/raft.run)·fm()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:241 +0x27
github.com/hashicorp/raft.func·011()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 11 [select]:
github.com/hashicorp/raft.(_Raft).runFSM(0xc2080b81c0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:490 +0xf74
github.com/hashicorp/raft._Raft.(github.com/hashicorp/raft.runFSM)·fm()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/raft.go:242 +0x27
github.com/hashicorp/raft.func·011()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 34 [select, 3 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0xc2080de000, 0xc2080463c0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:390 +0x14a
created by github.com/influxdb/influxdb/cmd/influxd/run.func·003
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:280 +0x8d3

goroutine 35 [select]:
github.com/influxdb/influxdb/services/hh.(_Service).retryWrites(0xc208073180)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/hh/service.go:98 +0x2e7
created by github.com/influxdb/influxdb/services/hh.(_Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/hh/service.go:62 +0xef

goroutine 36 [select, 3 minutes]:
github.com/influxdb/influxdb/services/hh.(_Service).expireWrites(0xc208073180)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/hh/service.go:116 +0x2a4
created by github.com/influxdb/influxdb/services/hh.(_Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/hh/service.go:63 +0x109

goroutine 37 [chan receive, 3 minutes]:
github.com/influxdb/influxdb/tcp.(_listener).Accept(0xc20803a7e0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/cluster.(_Service).serve(0xc208010230)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cluster/service.go:79 +0x102
created by github.com/influxdb/influxdb/cluster.(*Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cluster/service.go:56 +0x67

goroutine 38 [select, 3 minutes]:
github.com/influxdb/influxdb/services/precreator.(_Service).runPrecreation(0xc20800ad70)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/precreator/service.go:72 +0x407
created by github.com/influxdb/influxdb/services/precreator.(_Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/precreator/service.go:50 +0xcb

goroutine 39 [chan receive, 3 minutes]:
github.com/influxdb/influxdb/tcp.(_listener).Accept(0xc20803a7e8, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/services/snapshotter.(_Service).serve(0xc20800ae60)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/snapshotter/service.go:68 +0xad
created by github.com/influxdb/influxdb/services/snapshotter.(*Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/snapshotter/service.go:46 +0x62

goroutine 50 [IO wait, 3 minutes]:
net.(_pollDesc).Wait(0xc20813c7d0, 0x72, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(_pollDesc).WaitRead(0xc20813c7d0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(_netFD).accept(0xc20813c770, 0x0, 0x7f47abcfccc8, 0xc20802bc70)
/root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(_TCPListener).AcceptTCP(0xc208128028, 0xc208030000, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(_TCPListener).Accept(0xc208128028, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
net/http.(_Server).Serve(0xc20812e720, 0x7f47abcff310, 0xc208128028, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:1728 +0x92
net/http.Serve(0x7f47abcff310, 0xc208128028, 0x7f47abd08050, 0xc20802bc60, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:1606 +0xa1
github.com/influxdb/influxdb/services/admin.(_Service).serve(0xc2080bb6e0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/admin/service.go:71 +0x147
created by github.com/influxdb/influxdb/services/admin.(_Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/admin/service.go:39 +0xd9

goroutine 51 [select]:
github.com/influxdb/influxdb/services/continuous_querier.(_Service).backgroundLoop(0xc2080102a0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:151 +0x1fc
created by github.com/influxdb/influxdb/services/continuous_querier.(_Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:87 +0x1ed

goroutine 52 [IO wait, 3 minutes]:
net.(_pollDesc).Wait(0xc20813c610, 0x72, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(_pollDesc).WaitRead(0xc20813c610, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(_netFD).accept(0xc20813c5b0, 0x0, 0x7f47abcfccc8, 0xc20810e038)
/root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(_TCPListener).AcceptTCP(0xc208128030, 0xc2080ecdc8, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(_TCPListener).Accept(0xc208128030, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
net/http.(_Server).Serve(0xc20812e7e0, 0x7f47abcff310, 0xc208128030, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:1728 +0x92
net/http.Serve(0x7f47abcff310, 0xc208128030, 0x7f47abd08078, 0xc208010310, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:1606 +0xa1
github.com/influxdb/influxdb/services/httpd.(_Service).serve(0xc208093780)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/httpd/service.go:83 +0x79
created by github.com/influxdb/influxdb/services/httpd.(_Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/httpd/service.go:51 +0x283

goroutine 53 [select, 3 minutes]:
github.com/influxdb/influxdb/services/retention.(_Service).deleteShardGroups(0xc208046840)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/retention/service.go:67 +0x433
created by github.com/influxdb/influxdb/services/retention.(_Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/retention/service.go:44 +0x67

goroutine 54 [select, 3 minutes]:
github.com/influxdb/influxdb/services/retention.(_Service).deleteShards(0xc208046840)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/retention/service.go:101 +0x821
created by github.com/influxdb/influxdb/services/retention.(_Service).Open
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/services/retention/service.go:45 +0x81

goroutine 55 [chan receive, 3 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).startServerReporting(0xc2080de000)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:341 +0x5e
created by github.com/influxdb/influxdb/cmd/influxd/run.func·003
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:303 +0xd78

goroutine 56 [select, 3 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(_Command).monitorServerErrors(0xc208046360)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:118 +0x2da
created by github.com/influxdb/influxdb/cmd/influxd/run.(_Command).Run
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:102 +0xa1a

goroutine 20 [IO wait, 3 minutes]:
net.(_pollDesc).Wait(0xc2081240d0, 0x72, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(_pollDesc).WaitRead(0xc2081240d0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(_netFD).Read(0xc208124070, 0xc2081be000, 0x1000, 0x1000, 0x0, 0x7f47abcfccc8, 0xc2080d0238)
/root/.gvm/gos/go1.4.2/src/net/fd_unix.go:242 +0x40f
net.(_conn).Read(0xc20803a028, 0xc2081be000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/net.go:121 +0xdc
net/http.(_liveSwitchReader).Read(0xc208072048, 0xc2081be000, 0x1000, 0x1000, 0x1c00000000000000, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:214 +0xab
io.(_LimitedReader).Read(0xc2081a2020, 0xc2081be000, 0x1000, 0x1000, 0xc20831f940, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/io/io.go:408 +0xce
bufio.(_Reader).fill(0xc208122000)
/root/.gvm/gos/go1.4.2/src/bufio/bufio.go:97 +0x1ce
bufio.(_Reader).ReadSlice(0xc208122000, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/bufio/bufio.go:295 +0x257
bufio.(_Reader).ReadLine(0xc208122000, 0x0, 0x0, 0x0, 0xc207fedf00, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/bufio/bufio.go:324 +0x62
net/textproto.(_Reader).readLineSlice(0xc2081200c0, 0x0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/textproto/reader.go:55 +0x9e
net/textproto.(_Reader).ReadLine(0xc2081200c0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc208122000, 0xc2080352b0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/request.go:598 +0xcb
net/http.(_conn).readRequest(0xc208072000, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:586 +0x26f
net/http.(_conn).serve(0xc208072000)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:1162 +0x69e
created by net/http.(_Server).Serve
/root/.gvm/gos/go1.4.2/src/net/http/server.go:1751 +0x35e

goroutine 43 [IO wait]:
net.(_pollDesc).Wait(0xc2080107d0, 0x72, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(_pollDesc).WaitRead(0xc2080107d0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(_netFD).Read(0xc208010770, 0xc208158000, 0x1000, 0x1000, 0x0, 0x7f47abcfccc8, 0xc20825c0b8)
/root/.gvm/gos/go1.4.2/src/net/fd_unix.go:242 +0x40f
net.(_conn).Read(0xc20803a020, 0xc208158000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/net.go:121 +0xdc
net/http.(_liveSwitchReader).Read(0xc208142048, 0xc208158000, 0x1000, 0x1000, 0x1c00000000000002, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:214 +0xab
io.(_LimitedReader).Read(0xc20801e000, 0xc208158000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/io/io.go:408 +0xce
bufio.(_Reader).fill(0xc20812e780)
/root/.gvm/gos/go1.4.2/src/bufio/bufio.go:97 +0x1ce
bufio.(_Reader).ReadSlice(0xc20812e780, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/bufio/bufio.go:295 +0x257
bufio.(_Reader).ReadLine(0xc20812e780, 0x0, 0x0, 0x0, 0xc207fe8900, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/bufio/bufio.go:324 +0x62
net/textproto.(_Reader).readLineSlice(0xc2081760c0, 0x0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/textproto/reader.go:55 +0x9e
net/textproto.(_Reader).ReadLine(0xc2081760c0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc20812e780, 0xc20811c4e0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/request.go:598 +0xcb
net/http.(_conn).readRequest(0xc208142000, 0x0, 0x0, 0x0)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:586 +0x26f
net/http.(_conn).serve(0xc208142000)
/root/.gvm/gos/go1.4.2/src/net/http/server.go:1162 +0x69e
created by net/http.(_Server).Serve
/root/.gvm/gos/go1.4.2/src/net/http/server.go:1751 +0x35e

Is there a way to disable those snapshots ? What are they used for ?

[otoolep]

Snapshots are part of the Raft system, and allow the replicated log to be truncated. You can learn about them in the Raft paper: https://ramcloud.stanford.edu/raft.pdf

Until snapshots are implemented, you can increase the Raft snapshot threshold.

#3020 (comment)

but you must build from source to do this. In addition 0.9.1 includes fixes that ensures the snapshot threshold should not be reached as quickly. Are you able to build from source?

[Flaviogitt]

[maxcanada talking here]
I can try, but I reached the snapshot threshold in one day. I am not sure that increasing it would be a good solution.

any solution to avoid that the server would panic ? I use only one node if that can help.

[otoolep]

@dgnorton said he was looking at this.