Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InfluxDb(0.6.2) crashes after a number of 'multiple time series in a single POST' updates #538

Closed
Pavel---L opened this issue May 13, 2014 · 9 comments
Closed

InfluxDb(0.6.2) crashes after a number of 'multiple time series in a single POST' updates #538

Pavel---L opened this issue May 13, 2014 · 9 comments
Milestone
0.6.3

Comments

@Pavel---L
Copy link

After a number of 'multiple time series in a single POST' updates InfluxDb(0.6.2) crashes and can't be restarted without complete wipe out of /opt/influxdb/shared/data folder.
@jvshahid
Copy link
Contributor

@Pavel---L can you please provide more information. may be a script that can reproduce the problem or the log file for example.

@Pavel---L
Copy link
Author

Sure,
end of influxDb log.txt

[2014/05/13 21:31:42 FET] INFO local: Initializing write buffer with buffer size of 10000
[2014/05/13 21:31:42 FET] INFO Waiting for servers to recover
[2014/05/13 21:31:42 FET] INFO ProtobufServer listening on 0.0.0.0:8099
[2014/05/13 21:31:42 FET] INFO Recovering local server
[2014/05/13 21:31:42 FET] INFO No shards to recover for 1
[2014/05/13 21:31:42 FET] INFO Recovered local server
[2014/05/13 21:31:42 FET] INFO recovered
[2014/05/13 21:31:42 FET] INFO Connecting to other nodes in the cluster
[2014/05/13 21:31:42 FET] INFO Starting admin interface on port 8083
[2014/05/13 21:31:42 FET] INFO Starting Http Api server on port 8086
[2014/05/13 21:32:42 FET] INFO No matching shards for write at time 1400005962573618u, creating...
[2014/05/13 21:32:42 FET] INFO createShards: start: Sat May 10 03:00:00 +0300 FET 2014. end: Mon Jun 9 03:00:00 +0300 FET 2014
[2014/05/13 21:32:42 FET] INFO DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db/00001
[2014/05/13 21:32:42 FET] INFO Adding long term shard: 1 - start: Sat May 10 03:00:00 +0300 FET 2014 (1399680000). end: Mon Jun 9 03:00:00 +0300 FET 2014 (1402272000). isLocal: %!d(bool=true). servers: [%!s(uint32=1)]
[2014/05/13 21:32:42 FET] INFO Opening log file /opt/influxdb/shared/data/wal/log.1
[2014/05/13 21:32:42 FET] INFO Opening index file /opt/influxdb/shared/data/wal/index.1
panic: runtime error: index out of range

goroutine 19 [running]:
runtime.panic(0x888880, 0x10091f7)
/home/vagrant/bin/go/src/pkg/runtime/panic.c:266 +0xb6
datastore.(_LevelDbShard).Write(0xc210110800, 0xc210057086, 0x3, 0xc2100a4dc0, 0x0, ...)
/home/vagrant/influxdb/src/datastore/leveldb_shard.go:86 +0x86a
datastore.(_LevelDbShardDatastore).Write(0xc210072af0, 0xc210070c00, 0x0, 0x0)
/home/vagrant/influxdb/src/datastore/leveldb_shard_datastore.go:167 +0x116
cluster.(_WriteBuffer).write(0xc2100de1c0, 0xc210070c00)
/home/vagrant/influxdb/src/cluster/write_buffer.go:87 +0x86
cluster.(_WriteBuffer).handleWrites(0xc2100de1c0)
/home/vagrant/influxdb/src/cluster/write_buffer.go:78 +0xb7
created by cluster.NewWriteBuffer
/home/vagrant/influxdb/src/cluster/write_buffer.go:43 +0x24f

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f79b8d7e050, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc2100de450, 0x72, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc2100de450, 0xb, 0x7f79b8d7af88)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc2100de3f0, 0x9fa660, 0x0, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc210000118, 0x5cab7b, 0x7f79b8bebaa0, 0x5cab7b)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc210000118, 0x7f79b8d7eb18, 0xc210000a08, 0xc21008d900, 0x0)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(_Server).Serve(0xc2100ae0f0, 0x7f79b8d7d200, 0xc210000118, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1622 +0x91
api/http.(_HttpServer).serveListener(0xc21000f3c0, 0x7f79b8d7d200, 0xc210000118, 0xc210000120)
/home/vagrant/influxdb/src/api/http/api.go:185 +0x98
api/http.(_HttpServer).Serve(0xc21000f3c0, 0x7f79b8d7d200, 0xc210000118)
/home/vagrant/influxdb/src/api/http/api.go:155 +0xc7c
api/http.(_HttpServer).ListenAndServe(0xc21000f3c0)
/home/vagrant/influxdb/src/api/http/api.go:79 +0x16c
server.(_Server).ListenAndServe(0xc210072bd0, 0xc210072bd0, 0x0)
/home/vagrant/influxdb/src/server/server.go:153 +0x7a6
main.main()
/home/vagrant/influxdb/src/daemon/influxd.go:155 +0xb97

goroutine 3 [syscall]:
os/signal.loop()
/home/vagrant/bin/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
/home/vagrant/bin/go/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 4 [chan receive]:
code.google.com/p/log4go.ConsoleLogWriter.run(0xc210069000, 0x7f79b8d7b110, 0xc210000008)
/home/vagrant/influxdb/src/code.google.com/p/log4go/termlog.go:27 +0x60
created by code.google.com/p/log4go.NewConsoleLogWriter
/home/vagrant/influxdb/src/code.google.com/p/log4go/termlog.go:19 +0x67

goroutine 5 [select]:
code.google.com/p/log4go.func·002()
/home/vagrant/influxdb/src/code.google.com/p/log4go/filelog.go:84 +0x84c
created by code.google.com/p/log4go.NewFileLogWriter
/home/vagrant/influxdb/src/code.google.com/p/log4go/filelog.go:116 +0x2d1

goroutine 6 [syscall]:
runtime.goexit()
/home/vagrant/bin/go/src/pkg/runtime/proc.c:1394

goroutine 7 [chan receive]:
wal.(*WAL).processEntries(0xc210070d00)
/home/vagrant/influxdb/src/wal/wal.go:252 +0x3f
created by wal.NewWAL
/home/vagrant/influxdb/src/wal/wal.go:103 +0x9f3

goroutine 8 [sleep]:
time.Sleep(0x8bb2c97000)
/tmp/makerelease886106415/go/src/pkg/runtime/time.goc:31 +0x31
cluster.func·001()
/home/vagrant/influxdb/src/cluster/cluster_configuration.go:133 +0x35
created by cluster.(*ClusterConfiguration).CreateFutureShardsAutomaticallyBeforeTimeComes
/home/vagrant/influxdb/src/cluster/cluster_configuration.go:138 +0x63

goroutine 9 [finalizer wait]:
runtime.park(0x451590, 0x1020118, 0x100ab48)
/home/vagrant/bin/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
/home/vagrant/bin/go/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
/home/vagrant/bin/go/src/pkg/runtime/proc.c:1394

goroutine 11 [chan receive]:
main.waitForSignals(0x7f79b8d7d180, 0xc210072bd0)
/home/vagrant/influxdb/src/daemon/null_profiler.go:23 +0x126
created by main.startProfiler
/home/vagrant/influxdb/src/daemon/null_profiler.go:15 +0x38

goroutine 12 [IO wait]:
net.runtime_pollWait(0x7f79b8d7e1a0, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc210072ca0, 0x72, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc210072ca0, 0xb, 0x7f79b8d7af88)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc210072c40, 0x9fa660, 0x0, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc2100a8338, 0x18, 0xc210093010, 0x5ca953)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc2100a8338, 0x0, 0x0, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(_Server).Serve(0xc2100a4d70, 0x7f79b8d7d200, 0xc2100a8338, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1622 +0x91
coordinator.func·007()
/home/vagrant/influxdb/src/coordinator/raft_server.go:530 +0x3a
created by coordinator.(*RaftServer).Serve
/home/vagrant/influxdb/src/coordinator/raft_server.go:534 +0x4d9

goroutine 25 [IO wait]:
net.runtime_pollWait(0x7f79b8d7ddb0, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc210099c30, 0x72, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc210099c30, 0xb, 0x7f79b8d7af88)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).Read(0xc210099bd0, 0xc2100ff000, 0x1000, 0x1000, 0x0, ...)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(_conn).Read(0xc2100a8828, 0xc2100ff000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/net/net.go:122 +0xc5
net/http.(_liveSwitchReader).Read(0xc210070728, 0xc2100ff000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/net/http/server.go:204 +0xa5
io.(_LimitedReader).Read(0xc21009be60, 0xc2100ff000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/io/io.go:398 +0xbb
bufio.(_Reader).fill(0xc210045360)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(_Reader).ReadSlice(0xc210045360, 0xa, 0x0, 0x0, 0x0, ...)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:274 +0x204
bufio.(_Reader).ReadLine(0xc210045360, 0x0, 0x0, 0x0, 0x0, ...)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:305 +0x63
net/textproto.(_Reader).readLineSlice(0xc2100a9600, 0x7f79b8d72000, 0x7b9440, 0x7f79b549dce8, 0x466a22, ...)
/home/vagrant/bin/go/src/pkg/net/textproto/reader.go:55 +0x61
net/textproto.(_Reader).ReadLine(0xc2100a9600, 0xc2100381a0, 0x0, 0xc210100000, 0x7f79b549db6f)
/home/vagrant/bin/go/src/pkg/net/textproto/reader.go:36 +0x27
net/http.ReadRequest(0xc210045360, 0xc2100381a0, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/request.go:526 +0x88
net/http.(_conn).readRequest(0xc210070700, 0x0, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/server.go:575 +0x1bb
net/http.(_conn).serve(0xc210070700)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1123 +0x3b4
created by net/http.(_Server).Serve
/home/vagrant/bin/go/src/pkg/net/http/server.go:1644 +0x28b

goroutine 14 [select]:
github.com/goraft/raft.(_server).leaderLoop(0xc2100a77e0)
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:765 +0x5fe
github.com/goraft/raft.(_server).loop(0xc2100a77e0)
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:568 +0x33f
created by github.com/goraft/raft.(*server).Start
/home/vagrant/influxdb/src/github.com/goraft/raft/server.go:472 +0x7af

goroutine 15 [select]:
coordinator.(_RaftServer).CompactLog(0xc21006a580)
/home/vagrant/influxdb/src/coordinator/raft_server.go:320 +0x2ef
created by coordinator.(_RaftServer).startRaft
/home/vagrant/influxdb/src/coordinator/raft_server.go:374 +0x375

goroutine 16 [select]:
coordinator.(_RaftServer).raftLeaderLoop(0xc21006a580, 0xc2100a5e80)
/home/vagrant/influxdb/src/coordinator/raft_server.go:430 +0x29c
created by coordinator.(_RaftServer).raftEventHandler
/home/vagrant/influxdb/src/coordinator/raft_server.go:419 +0x1d0

goroutine 18 [IO wait]:
net.runtime_pollWait(0x7f79b8d7e0f8, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc210099450, 0x72, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc210099450, 0xb, 0x7f79b8d7af88)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc2100993f0, 0x9fa660, 0x0, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc2100a87d8, 0xc2100cdd20, 0x0, 0x7f79b8d7d1d0)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc2100a87d8, 0xc2100cdd20, 0x7f79b4c60f38, 0x1, 0x1)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:243 +0x27
coordinator.(_ProtobufServer).ListenAndServe(0xc2100a57c0)
/home/vagrant/influxdb/src/coordinator/protobuf_server.go:64 +0x1c7
created by server.(*Server).ListenAndServe
/home/vagrant/influxdb/src/server/server.go:116 +0x218

goroutine 21 [IO wait]:
net.runtime_pollWait(0x7f79b8d7dfa8, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc210099a00, 0x72, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc210099a00, 0xb, 0x7f79b8d7af88)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc2100999a0, 0x9fa660, 0x0, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc2100a8808, 0x5cab7b, 0x7f79b4c7ce68, 0x5cab7b)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc2100a8808, 0x7f79b8d7eb18, 0xc2100a8828, 0xc210070700, 0x0)
/home/vagrant/bin/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(_Server).Serve(0xc2100a4460, 0x7f79b8d7d200, 0xc2100a8808, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1622 +0x91
net/http.Serve(0x7f79b8d7d200, 0xc2100a8808, 0x7f79b8d7eaf0, 0xc2100cddd0, 0x1b, ...)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1561 +0x70
admin.(_HttpServer).ListenAndServe(0xc2100a5840)
/home/vagrant/influxdb/src/admin/http_server.go:35 +0x170
created by server.(_Server).ListenAndServe
/home/vagrant/influxdb/src/server/server.go:130 +0x460

goroutine 23 [IO wait]:
net.runtime_pollWait(0x7f79b8d7df00, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc2100de610, 0x72, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc2100de610, 0xb, 0x7f79b8d7af88)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).Read(0xc2100de5b0, 0xc2100fa000, 0x1000, 0x1000, 0x0, ...)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(_conn).Read(0xc210000030, 0xc2100fa000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/net/net.go:122 +0xc5
net/http.(_liveSwitchReader).Read(0xc21008d4a8, 0xc2100fa000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/net/http/server.go:204 +0xa5
io.(_LimitedReader).Read(0xc210095d40, 0xc2100fa000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/io/io.go:398 +0xbb
bufio.(_Reader).fill(0xc2100c4120)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(_Reader).ReadSlice(0xc2100c4120, 0xa, 0x0, 0x0, 0x0, ...)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:274 +0x204
bufio.(_Reader).ReadLine(0xc2100c4120, 0x0, 0x0, 0x0, 0x0, ...)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:305 +0x63
net/textproto.(_Reader).readLineSlice(0xc21007f960, 0x7f79b8d725e0, 0x7b9440, 0x7f79b4c7ace8, 0x466a22, ...)
/home/vagrant/bin/go/src/pkg/net/textproto/reader.go:55 +0x61
net/textproto.(_Reader).ReadLine(0xc21007f960, 0xc21010c340, 0x0, 0xc2100fb000, 0x7f79b4c7ab6f)
/home/vagrant/bin/go/src/pkg/net/textproto/reader.go:36 +0x27
net/http.ReadRequest(0xc2100c4120, 0xc21010c340, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/request.go:526 +0x88
net/http.(_conn).readRequest(0xc21008d480, 0x0, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/server.go:575 +0x1bb
net/http.(_conn).serve(0xc21008d480)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1123 +0x3b4
created by net/http.(_Server).Serve
/home/vagrant/bin/go/src/pkg/net/http/server.go:1644 +0x28b

goroutine 24 [IO wait]:
net.runtime_pollWait(0x7f79b8d7de58, 0x72, 0x0)
/tmp/makerelease886106415/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc2100de7d0, 0x72, 0x7f79b8d7af88, 0xb)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc2100de7d0, 0xb, 0x7f79b8d7af88)
/home/vagrant/bin/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).Read(0xc2100de770, 0xc2100fc000, 0x1000, 0x1000, 0x0, ...)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(_conn).Read(0xc2100003d0, 0xc2100fc000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/net/net.go:122 +0xc5
net/http.(_liveSwitchReader).Read(0xc21008d528, 0xc2100fc000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/net/http/server.go:204 +0xa5
io.(_LimitedReader).Read(0xc210095d80, 0xc2100fc000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/io/io.go:398 +0xbb
bufio.(_Reader).fill(0xc2100c4180)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(_Reader).ReadSlice(0xc2100c4180, 0xa, 0x0, 0x0, 0x0, ...)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:274 +0x204
bufio.(_Reader).ReadLine(0xc2100c4180, 0x0, 0x0, 0x0, 0x0, ...)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:305 +0x63
net/textproto.(_Reader).readLineSlice(0xc21007f9f0, 0x7f79b8d725e0, 0x7b9440, 0x7f79b4c78ce8, 0x466a22, ...)
/home/vagrant/bin/go/src/pkg/net/textproto/reader.go:55 +0x61
net/textproto.(_Reader).ReadLine(0xc21007f9f0, 0xc21010c1a0, 0x0, 0xc2100fd000, 0x7f79b4c78b6f)
/home/vagrant/bin/go/src/pkg/net/textproto/reader.go:36 +0x27
net/http.ReadRequest(0xc2100c4180, 0xc21010c1a0, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/request.go:526 +0x88
net/http.(_conn).readRequest(0xc21008d500, 0x0, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/server.go:575 +0x1bb
net/http.(_conn).serve(0xc21008d500)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1123 +0x3b4
created by net/http.(_Server).Serve
/home/vagrant/bin/go/src/pkg/net/http/server.go:1644 +0x28b

goroutine 35 [runnable]:
syscall.Syscall(0x0, 0xe, 0xc210101000, 0x1000, 0x3ae, ...)
/home/vagrant/bin/go/src/pkg/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0xe, 0xc210101000, 0x1000, 0x1000, 0x72, ...)
/home/vagrant/bin/go/src/pkg/syscall/zsyscall_linux_amd64.go:838 +0x72
syscall.Read(0xe, 0xc210101000, 0x1000, 0x1000, 0xffffffffffffff01, ...)
/home/vagrant/bin/go/src/pkg/syscall/syscall_unix.go:135 +0x5c
net.(_netFD).Read(0xc21010da80, 0xc210101000, 0x1000, 0x1000, 0x0, ...)
/home/vagrant/bin/go/src/pkg/net/fd_unix.go:200 +0x1eb
net.(_conn).Read(0xc210000a08, 0xc210101000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/net/net.go:122 +0xc5
net/http.(_liveSwitchReader).Read(0xc21008d928, 0xc210101000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/net/http/server.go:204 +0xa5
io.(_LimitedReader).Read(0xc210120040, 0xc210101000, 0x1000, 0x1000, 0x100cc64, ...)
/home/vagrant/bin/go/src/pkg/io/io.go:398 +0xbb
bufio.(_Reader).fill(0xc2100c41e0)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(_Reader).ReadSlice(0xc2100c41e0, 0x8c230a, 0x0, 0x0, 0x0, ...)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:274 +0x204
bufio.(_Reader).ReadLine(0xc2100c41e0, 0x0, 0x0, 0x0, 0x0, ...)
/home/vagrant/bin/go/src/pkg/bufio/bufio.go:305 +0x63
net/textproto.(_Reader).readLineSlice(0xc210114b40, 0x7f79b8d72000, 0x7b9440, 0x7f79b4c5ece8, 0x466a22, ...)
/home/vagrant/bin/go/src/pkg/net/textproto/reader.go:55 +0x61
net/textproto.(_Reader).ReadLine(0xc210114b40, 0xc21012e270, 0x0, 0x581d45, 0xc21010da80)
/home/vagrant/bin/go/src/pkg/net/textproto/reader.go:36 +0x27
net/http.ReadRequest(0xc2100c41e0, 0xc21012e270, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/request.go:526 +0x88
net/http.(_conn).readRequest(0xc21008d900, 0x0, 0x0, 0x0)
/home/vagrant/bin/go/src/pkg/net/http/server.go:575 +0x1bb
net/http.(_conn).serve(0xc21008d900)
/home/vagrant/bin/go/src/pkg/net/http/server.go:1123 +0x3b4
created by net/http.(_Server).Serve
/home/vagrant/bin/go/src/pkg/net/http/server.go:1644 +0x28b

@Pavel---L
Copy link
Author

Also complete ngrep log of all messages that lead to crash. All messages were applied to fresh db

interface: eth1 (192.168.40.0/255.255.255.0)
filter: (ip or ip6) and ( host 192.168.40.178 and port 8086 )

T 192.168.40.178:65268 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65269 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65270 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65268 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65269 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65270 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65268 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65269 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65270 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65290 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65290 -> 192.168.40.184:8086 [AP]
POST /cluster_admins?u=root&p=root HTTP/1.1.
User-Agent: curl/7.30.0.
Host: 192.168.40.184:8086.
Accept: /.
Content-Length: 50.
Content-Type: application/x-www-form-urlencoded.
.
{"name": "datadladmin", "password": "datadladmin"}

T 192.168.40.184:8086 -> 192.168.40.178:65290 [AP]
HTTP/1.1 200 OK.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: text/plain.
Date: Tue, 13 May 2014 18:32:41 GMT.
Content-Length: 0.
.

T 192.168.40.178:65290 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65290 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65291 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65291 -> 192.168.40.184:8086 [AP]
POST /db?u=datadladmin&p=datadladmin HTTP/1.1.
User-Agent: curl/7.30.0.
Host: 192.168.40.184:8086.
Accept: /.
Content-Length: 15.
Content-Type: application/x-www-form-urlencoded.
.
{"name": "EBS"}

T 192.168.40.184:8086 -> 192.168.40.178:65291 [AP]
HTTP/1.1 201 Created.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: text/plain.
Date: Tue, 13 May 2014 18:32:41 GMT.
Content-Length: 0.
.

T 192.168.40.178:65291 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65291 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65292 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65292 -> 192.168.40.184:8086 [AP]
POST /db/EBS/users?u=datadladmin&p=datadladmin HTTP/1.1.
User-Agent: curl/7.30.0.
Host: 192.168.40.184:8086.
Accept: /.
Content-Length: 48.
Content-Type: application/x-www-form-urlencoded.
.
{"name": "datadluser", "password": "datadluser"}

T 192.168.40.184:8086 -> 192.168.40.178:65292 [AP]
HTTP/1.1 200 OK.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: text/plain.
Date: Tue, 13 May 2014 18:32:41 GMT.
Content-Length: 0.
.

T 192.168.40.178:65292 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65292 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65293 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65293 -> 192.168.40.184:8086 [AP]
POST /db/EBS/users/datadluser?u=root&p=root HTTP/1.1.
User-Agent: curl/7.30.0.
Host: 192.168.40.184:8086.
Accept: /.
Content-Length: 15.
Content-Type: application/x-www-form-urlencoded.
.
{"admin": true}

T 192.168.40.184:8086 -> 192.168.40.178:65293 [AP]
HTTP/1.1 200 OK.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: text/plain.
Date: Tue, 13 May 2014 18:32:41 GMT.
Content-Length: 0.
.

T 192.168.40.178:65293 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65293 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65294 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65294 -> 192.168.40.184:8086 [AP]
GET /cluster_admins?u=datadladmin&p=datadladmin HTTP/1.1.
User-Agent: curl/7.30.0.
Host: 192.168.40.184:8086.
Accept: /.
.

T 192.168.40.184:8086 -> 192.168.40.178:65294 [AP]
HTTP/1.1 200 OK.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: application/json.
Date: Tue, 13 May 2014 18:32:41 GMT.
Content-Length: 48.
.
[{"username":"root"},{"username":"datadladmin"}]

T 192.168.40.178:65294 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65294 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65295 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65295 -> 192.168.40.184:8086 [AP]
GET /db/EBS/users?u=datadladmin&p=datadladmin HTTP/1.1.
User-Agent: curl/7.30.0.
Host: 192.168.40.184:8086.
Accept: /.
.

T 192.168.40.184:8086 -> 192.168.40.178:65295 [AP]
HTTP/1.1 200 OK.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: application/json.
Date: Tue, 13 May 2014 18:32:42 GMT.
Content-Length: 38.
.
[{"name":"datadluser","isAdmin":true}]

T 192.168.40.178:65295 -> 192.168.40.184:8086 [AF]
......

T 192.168.40.178:65295 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65297 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AP]
POST /db/EBS/series?u=root&p=root HTTP/1.1.
Accept: /.
Content-Length: 1092.
Content-Type: application/json.
Host: 192.168.40.184:8086.
Connection: Keep-Alive.
.

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AP]
[{"name":"GBPJPY_ASK_PRICE","columns":["price_0","price_1","price_2","price_3"],"points":[[172.17999,172.2,172.31,172.5]]},{"name":"SAUUSD_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[1291.90002,1292,1292.09998]]},{"name":"USDJPY_ASK_PRICE","columns":["price_0","price_1","price_2","price_3","price_4","price_5","price_6","price_7","price_8","price_9"],"points":[[102.565,102.57,102.575,102.58,102.585,102.59,102.595,102.6,102.605,102.61]]},{"name":"SAUUSD_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[1292,1292.09998,1292.19995]]},{"name":"GBPUSD_ASK_PRICE","columns":["price_0"],"points":[[1.6786]]},{"name":"EURGBP_ASK_PRICE","columns":["price_0","price_1"],"points":[[0.82315,0.8232]]},{"name":"GBPJPY_ASK_PRICE","columns":["price_0","price_1","price_2","price_3"],"points":[[172.17999,172.2,172.31,172.5]]},{"name":"LPTUSD_ASK_PRICE","columns":["price_0"],"points":[[1414]]},{"name":"XPTUSD_ASK_PRICE","columns":["price_0"],"points":[[1414]]},{"name":"EURGBP_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[0.82315,0.8232,0.82325]]}]

T 192.168.40.184:8086 -> 192.168.40.178:65297 [AP]
HTTP/1.1 200 OK.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: text/plain.
Date: Tue, 13 May 2014 18:32:42 GMT.
Content-Length: 0.
.

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AP]
POST /db/EBS/series?u=root&p=root HTTP/1.1.
Accept: /.
Content-Length: 1052.
Content-Type: application/json.
Host: 192.168.40.184:8086.
Connection: Keep-Alive.
.

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AP]
[{"name":"SAUUSD_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[1292,1292.09998,1292.19995]]},{"name":"USDJPY_ASK_PRICE","columns":["price_0","price_1","price_2","price_3","price_4","price_5","price_6","price_7","price_8","price_9"],"points":[[102.565,102.57,102.575,102.58,102.585,102.59,102.595,102.6,102.605,102.61]]},{"name":"EURGBP_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[0.82315,0.8232,0.82325]]},{"name":"GBPJPY_ASK_PRICE","columns":["price_0","price_1","price_2","price_3"],"points":[[172.17999,172.2,172.31,172.5]]},{"name":"LPTUSD_ASK_PRICE","columns":["price_0"],"points":[[1414]]},{"name":"SAGUSD_ASK_PRICE","columns":["price_0"],"points":[[19.58]]},{"name":"XAGUSD_ASK_PRICE","columns":["price_0","price_1"],"points":[[19.5775,19.58]]},{"name":"GBPUSD_ASK_PRICE","columns":["price_0"],"points":[[1.6786]]},{"name":"GBPJPY_ASK_PRICE","columns":["price_0","price_1","price_2","price_3"],"points":[[172.17999,172.2,172.31,172.5]]},{"name":"LPTUSD_ASK_PRICE","columns":["price_0"],"points":[[1414.5]]}]

T 192.168.40.184:8086 -> 192.168.40.178:65297 [AP]
HTTP/1.1 200 OK.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: text/plain.
Date: Tue, 13 May 2014 18:32:42 GMT.
Content-Length: 0.
.

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AP]
POST /db/EBS/series?u=root&p=root HTTP/1.1.
Accept: /.
Content-Length: 978.
Content-Type: application/json.
Host: 192.168.40.184:8086.
Connection: Keep-Alive.
.

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AP]
[{"name":"SAUUSD_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[1291.90002,1292,1292.09998]]},{"name":"XAGUSD_ASK_PRICE","columns":["price_0","price_1"],"points":[[19.58,19.5825]]},{"name":"XPTUSD_ASK_PRICE","columns":["price_0"],"points":[[1414.5]]},{"name":"SAGUSD_ASK_PRICE","columns":["price_0"],"points":[[19.585]]},{"name":"XAGUSD_ASK_PRICE","columns":["price_0","price_1"],"points":[[19.58,19.585]]},{"name":"XAGUSD_ASK_PRICE","columns":["price_0"],"points":[[19.585]]},{"name":"XPDUSD_ASK_PRICE","columns":["price_0","price_1"],"points":[[796.5,800.5]]},{"name":"USDJPY_ASK_PRICE","columns":["price_0","price_1","price_2","price_3","price_4","price_5","price_6","price_7","price_8","price_9"],"points":[[102.565,102.57,102.575,102.58,102.585,102.59,102.595,102.6,102.605,102.61]]},{"name":"GBPJPY_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[172.2,172.31,172.5]]},{"name":"SAGUSD_ASK_PRICE","columns":["price_0"],"points":[[19.585]]}]

T 192.168.40.184:8086 -> 192.168.40.178:65297 [AP]
HTTP/1.1 200 OK.
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE.
Access-Control-Allow-Origin: *.
Access-Control-Max-Age: 2592000.
Content-Type: text/plain.
Date: Tue, 13 May 2014 18:32:42 GMT.
Content-Length: 0.
.

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AP]
POST /db/EBS/series?u=root&p=root HTTP/1.1.
Accept: /.
Content-Length: 779.
Content-Type: application/json.
Host: 192.168.40.184:8086.
Connection: Keep-Alive.
.

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AP]
[{"name":"XAGUSD_ASK_PRICE","columns":["price_0","price_1"],"points":[[19.585,19.5875]]},{"name":"USDHKD_ASK_PRICE","columns":["price_0"],"points":[[7.7549]]},{"name":"XAGUSD_ASK_PRICE","columns":["price_0"],"points":[[19.5875]]},{"name":"LPTUSD_ASK_PRICE","columns":["price_0"],"points":[[1414]]},{"name":"SAUUSD_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[1292,1292.09998,1292.19995]]},{"name":"XPTUSD_ASK_PRICE","columns":["price_0"],"points":[[1414]]},{"name":"NZDUSD_ASK_PRICE","columns":["price_0"],"points":[[0.8597]]},{"name":"GBPUSD_ASK_PRICE","columns":[],"points":[[]]},{"name":"LPTUSD_ASK_PRICE","columns":["price_0"],"points":[[1414]]},{"name":"SAUUSD_ASK_PRICE","columns":["price_0","price_1","price_2"],"points":[[1292,1292.09998,1292.19995]]}]

T 192.168.40.178:65297 -> 192.168.40.184:8086 [A]
......

T 192.168.40.178:65297 -> 192.168.40.184:8086 [AF]
......

@Pavel---L
Copy link
Author

I'm a new to GitHub. Is there is a way to attach files instead of dumping the whole log in comment?

@jvshahid
Copy link
Contributor

The recommended way is to create a gist on https://gist.github.com/ and link to it in the comment. That said, I think we have the information that we need. I'll let you know if we need more info.

@Pavel---L
Copy link
Author

I can provide you with complete curl commands if it helps

@jvshahid
Copy link
Contributor

let me try what i have first.

@jvshahid
Copy link
Contributor

I can reproduce the bug, thanks for reporting it. We'll have a fix asap.

@Pavel---L
Copy link
Author

Thanks for fast answer. Looking forward for fix.

@jvshahid jvshahid added this to the 0.6.3 milestone May 13, 2014
@jvshahid jvshahid self-assigned this May 13, 2014
jvshahid added a commit that referenced this issue May 27, 2014
@jvshahid @pauldix
Fix #538. Don't panic when the same series is written with different …
441228e 
…columns
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.6.3
Development

No branches or pull requests
2 participants
@jvshahid @Pavel---L