tsdb: Block Head GC till pending readers are done reading

[codesome]

Closes #5877 #8221 #9079

This PR tracks the in-flight readers on Head block and block Head truncation till those readers are done reading. And while the in-memory truncation is happening (and not the m-mapped stuff), getting querier on Head block is blocked.

I have copied the tests from #9077 and #9080 into this while modifying slightly to take care of blocking Compact()/Querier() calls.

I have opened this as draft since the current approach is very naive without considering the time ranges of the queriers. I will be looking at improving it further to only block (and get blocked on) the queriers that would be affected with the truncation.

Only blocks memory truncation on overlapping queries and if truncation is going on or has started, only block queries that are overlapping with the truncation.

The querier adjusts the time range based on the on-going truncation without taking a lock. The memory truncation waits for in-flight queries that overlap with the truncation to finish.

[roidelapluie]

Couldn't we reuse the isolation tracking here?

[codesome]

Couldn't we reuse the isolation tracking here?

I think we can, we will need to additionally store and mint and maxt of the reads. Let me try that.

[codesome]

I am now using the existing isolation tracking with mint and maxt in the isolation states. I have also added enhancements to only block memory truncation on overlapping queries and if truncation is going on or started, only block queries that are overlapping with the truncation.

The PR is ready for review now. I will try adding more tests in case any part of it is untested yet tests added.

[codesome]

I have added more tests cases now with some small improvements and fixes. Now it is finally ready for review.

[codesome]

I have updated the PR with a better lock-free approach for this, and much simpler to understand. I have added detailed comments in the code explaining what is it trying to do. PR is ready for review again.

[beorn7]

Hi @codesome , thanks for doing this. I have tried to fully understand this, but so far I have only gained a partial understanding. I have written down a few concerns and questions. Perhaps they are dumb, coming from my merely partial or even wrong understanding, but at least then I have the opportunity to improve my understanding.

[codesome]

@beorn7 those are all valid questions and I agree few things can get confusing. I have answered them in the comments.

[beorn7]

I cannot say I'm 100% confident I have understood everything, but I guess it's good enough to give approval from my side. Just a few nits about doc comments.

[codesome]

Thanks for the reviews! I made 1 additional fix to take care of exclusiveness of the truncation time in WaitForPendingReadersInTimeRange and added a test case for it. I will merge this on green.