tsdb: Checkpoint closes mmaped chunk file despite open ChunkQuerier query; causing SIGSEGV

[bwplotka]

Prometheus version used: v1.8.2-0.20201029103703-63be30dceed9

Details: thanos-io/thanos#3497

Funny enough we hit this issue on ALL Thanos receivers every 16h ;p Exactly every 16h.

[bwplotka]

cc @codesome @pracucci

[roidelapluie]

Details: #8217

Inception

[bwplotka]

Sorry, a late hour. Edited (: -> thanos-io/thanos#3497

[codesome]

Not really sure how checkpointing would close a m-mapped chunk file, we would have faced this panic if that was the case. And I am not seeing the panic pointing to the TSDB codebase (was it truncated?).

[codesome]

Are you by any chance running the checkpointing in parallel?

[bwplotka]

TODO: Double check if simple iterator is affected by this truncation & chunkDiskMapper bug.

[bwplotka]

To potentially add: pending reader tracking as we have for blocks.

[roidelapluie]

To potentially add: pending reader tracking as we have for blocks.

Covered by #5877 I think

[codesome]

To add more info: once we close the m-map file, the byte slice that is m-mapped is no longer valid. Hence the panic when the query has already got the chunk and in the meanwhile when it is reading the m-map file was closed and truncated.

[fpetkovski]

We are still experiencing this issue in Thanos, exactly as reported here: thanos-io/thanos#3497. Prometheus version is 2.40.

I see that #8723 is merged, but it either does not fix the root cause, or there is another place where already released memory is accessed.