Panic with unread show series iterators during drop database

[mark-rushakoff]

Bug report

System info:

OSX, built from 7e515cf

Steps to reproduce:

Run repro script here:
https://gist.github.com/desa/1d43052f99b44f8957b3194d69c0588f

Expected behavior:
Database is dropped correctly.

Actual behavior:

This panic:

[query] 2016/09/20 14:12:16 SELECT "key" FROM stress.autogen._series
[query] 2016/09/20 14:12:16 DROP DATABASE stress
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x24a699]

goroutine 491 [running]:
panic(0x7b07c0, 0xc82000a190)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:481 +0x3e6
github.com/influxdata/influxdb/tsdb.(*Measurement).AppendSeriesKeysByID(0xc8369db8c0, 0xc8375a6c30, 0x0, 0x1, 0xc8369fad70, 0x1, 0x1, 0x0, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/tsdb/meta.go:600 +0x239
github.com/influxdata/influxdb/tsdb.(*seriesIterator).nextKeys(0xc837bfc780, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1050 +0x1ed
github.com/influxdata/influxdb/tsdb.(*seriesIterator).Next(0xc837bfc780, 0xc83cc83290, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1007 +0x66
github.com/influxdata/influxdb/influxql.(*floatSortedMergeIterator).pop(0xc837a3a240, 0xd9f670c6d84be601, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:388 +0x3fd
github.com/influxdata/influxdb/influxql.(*floatSortedMergeIterator).Next(0xc837a3a240, 0x13b28, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:350 +0x2d
github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Next(0xc83edacf00, 0x663400, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:761 +0xb5
github.com/influxdata/influxdb/influxql.(*floatFastDedupeIterator).Next(0xc83edacf20, 0x1, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.go:1191 +0x5a
github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Next(0xc83edacf40, 0x1, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:96 +0x69
github.com/influxdata/influxdb/influxql.(*floatAuxIterator).stream(0xc83c8d6000)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:880 +0x2b
created by github.com/influxdata/influxdb/influxql.(*floatAuxIterator).Start
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:844 +0x35
exit status 2

Additional info:

The repro script opens an HTTP request for SHOW SERIES, but does not read or close the response body. Then a separate HTTP request runs DROP DATABASE. Eventually the iterator for show series advances, but it tries to read a shard that was removed when the database was dropped, and it panics there.

cc @jsternberg

[e-dard]

Has this been fixed @mark-rushakoff? The script doesn't seem to panic on master

[mark-rushakoff]

It still panics, although in my experience today, it either fails on the first couple runs from a fresh start of influxdb, or it succeeds forever.

I tried to write a shorter repro with only the standard library, but I had less luck reproducing the issue with that approach. Here's the stack trace on master at b385dec. I had to use the repro script in the gist, with influx-stress at commit 96012d6c5d72fdcc61bdafc029214d0b786e5e0b.

[query] 2016/10/18 09:03:47 DROP DATABASE stress
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x246e89]

goroutine 755 [running]:
panic(0x7c04a0, 0xc82000a190)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:481 +0x3e6
github.com/influxdata/influxdb/tsdb.(*Measurement).AppendSeriesKeysByID(0xc8215791a0, 0xc82331a410, 0x0, 0x1, 0xc8216fc668, 0x1, 0x1, 0x0, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/tsdb/meta.go:601 +0x239
github.com/influxdata/influxdb/tsdb.(*seriesIterator).nextKeys(0xc821d98f00, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1203 +0x1ed
github.com/influxdata/influxdb/tsdb.(*seriesIterator).Next(0xc821d98f00, 0xc82f21c650, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/tsdb/shard.go:1160 +0x66
github.com/influxdata/influxdb/influxql.(*floatSortedMergeIterator).pop(0xc82ef75680, 0xc82f1f6001, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:388 +0x3fd
github.com/influxdata/influxdb/influxql.(*floatSortedMergeIterator).Next(0xc82ef75680, 0x134b8, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:350 +0x2d
github.com/influxdata/influxdb/influxql.(*floatInterruptIterator).Next(0xc82dd68d80, 0x671120, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:777 +0xb5
github.com/influxdata/influxdb/influxql.(*floatFastDedupeIterator).Next(0xc82dd68dc0, 0x1, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.go:1191 +0x5a
github.com/influxdata/influxdb/influxql.(*bufFloatIterator).Next(0xc82dd68e00, 0x1, 0x0, 0x0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:96 +0x69
github.com/influxdata/influxdb/influxql.(*floatAuxIterator).stream(0xc82f10d9e0)
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:896 +0x2b
created by github.com/influxdata/influxdb/influxql.(*floatAuxIterator).Start
        /Users/mr/go/src/github.com/influxdata/influxdb/influxql/iterator.gen.go:860 +0x35
exit status 2