Move check for active readers to message store GC action function

[hairyhum]

Proposed Changes

Message store GC postpones processing of file, which have readers.
When performing an action, it asserts that there are no readers.

Check for readers may race with readers update by a queue, crashing
the message store.

Make check and assert work with the same lookup to reduce failure rate.

In case of races the queue process should handle exception instead.

It's quite hard to reproduce the race without adding sleeps in the code, unfortunately.
I was not able to produce a test case.

Addresses #2000
[#165755203]

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue Message store GC race condition with readers. #2000)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask on the
mailing list. We're here to help! This is simply a reminder of what we are
going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• All tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in related repositories

[michaelklishin]

This looks reasonable. I'm in the process of adding a backing queue test that would trigger a compaction.

[michaelklishin]

I'm still working on a test that would specifically trigger compaction. Had to change my approach as communicated during the sync-up.

[michaelklishin]

Will backport after the pipeline passes.

[michaelklishin]

Backported to v3.7.x.