Deadlock with tsm1 engine

[PierreF]

During a stress testing of InfluxDB 0.11 cluster, it got stuck and all queries timed-out. During this test, I was sending writes, read (and also drop series).
I could see that number of active queries were high and didn't reduced even long time after stopping the stress tools.

After few search, I think this issue a a deadlock on a RWMutex in file_store.go:

• 1 thread get a RLock (in my case, it's FileStore.KeyCursor)
• another thread try to get a Lock (in my case it's FileStore.Replace) - this get blocked since first thread has a RLock
• the first thread try to acquire the RLock a second time (in FileStore.locations). This hang because the second thread is blocked on the Lock (https://golang.org/pkg/sync/#RWMutex.Lock : "a blocked Lock call excludes new readers from acquiring the lock")

To diagnose this issue, I've killed one a the InfluxDB node that got stuck with a SIGQUIT. Interesting part of the stacktrace is at https://gist.github.com/PierreF/7ed3ade69d56d04b48e8e22bfead43d5

Exact InfluxDB version : InfluxDB v0.11.1 (git: 0.11 57d625f)

This issue should still exists in 0.12, since FileStore.KeyCursor still get a RLock and call FileStore.locations (thought KeyCursor) which also get a RLock.

[e-dard]

Fixed via #6277