cassandraStore improper shutdown flow

[Dieterbe]

saw this when shutting down a prod MT:

(...)
2016/04/19 17:33:12 WRN    1 [metrics/tankb] there are 0 connections left alive
2016/04/19 17:33:12 INF    1 [metrics/tankb] stopping handlers
2016/04/19 17:33:12 INF    1 [metrics/tankb] rdyLoop exiting
2016/04/19 17:33:12 [I] closing store
2016/04/19 17:33:12 [store_cassandra.go:295 Search()] [E] cassandra query error. session has been closed
2016/04/19 17:33:12 [store_cassandra.go:295 Search()] [E] cassandra query error. session has been closed
2016/04/19 17:33:12 [store_cassandra.go:295 Search()] [E] cassandra query error. session has been closed
2016/04/19 17:33:12 [store_cassandra.go:295 Search()] [E] cassandra query error. session has been closed
2016/04/19 17:33:12 [I] terminating.

[Dieterbe]

the reason is that when we close metrictank, and we call store.Close() we only close the cassandra session, we should also:

1. stop the api server, or at least the render endpoint so that no more cass queries are being generated
2. wait for pending read queries to complete (maybe with a timeout)
3. stop accepting new write requests
4. wait for the write queue to drain (perhaps also with a tuneable timeout). making sure that after we stop sending writes to cassandra, we also cleanly submit the last metricpersist messages.
5. only then close the cassandra session

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.