master (future v0.13.0) Receive regression: Unexpected 2x memory usage for single tenant with multiTSDB change.

[bwplotka]

Single-tenant, similar ingestion just Thanos bump for Receiver for receive component.

master-2020-04-23-28ea33b5 -> master-2020-05-05-e5804d80

A slight increase in CPU as well.

Potential reasons:

• Multi TSDB PRs receive: Block WAL replay when starting receive component #2564, receive: Use read locks where possible to read tenants #2563, Multitsdb migration #2557, receive: Add support for TSDB per tenant #2012
• Go 1.14

[bwplotka]

It's actually an interesting pattern, potentially due to migration?

[brancz]

So at least for us this was an artifact of a messed up migration, we cleaned up the leftovers (which is why two identical TSDBs were running) and everything went back to normal.

[bwplotka]

Yes, this is mainly only in that case. Otherwise seems fine... except minor memory leak.

Can we for a start add leaktest fo all receive unit tests? cc @brancz