Ingesters panic during shutdown causing data loss

[rsteneteg]

Describe the bug
During a rolling restart of the Cortex ingesters some of the ingesters paniced causing data loss as all data had not been flushed to WAL yet.

To Reproduce
I have been unable to repro this except in Unit tests

Expected behavior
Ingesters should not crash during shutdown

Environment:

• Infrastructure: [Kubernetes]
• Deployment tool: [helm]
• Version: 1.4.0

Storage Engine

• Blocks
• Chunks

Additional Context

Logs from crashed ingester

/Users/rsteneteg/work/cortex/pkg/chunk/cache/memcached.go:187 +0x98
created by github.com/cortexproject/cortex/pkg/chunk/cache.(*Memcached).fetchKeysBatched
/Users/rsteneteg/work/cortex/pkg/chunk/cache/memcached.go:190 +0x58
github.com/cortexproject/cortex/pkg/chunk/cache.(*Memcached).fetchKeysBatched.func1(0x42a07cdae0, 0x1, 0x1, 0x400, 0x4000942150, 0x37d9d80, 0x4085aaf0e0, 0x414fe3b680)
goroutine 10716258567 [running]:
panic: send on closed channel