Scavenge: Handle events that have been deleted from (now empty) chunks but not from the index

[shaan1337]

Fixed: Handle events that have been deleted from (now empty) chunks but not from the index

Previously this would result in the scavenge stopping with an error like "Could not get TimeStamp range for chunk "

But this is a possible state if the old scavenger had scavenged the chunks but not the index. This fix correctly handles this scenario so that the scavenge process can continue

[shaan1337]

please hold on the review - i'm improving the test cases

[timothycoleman]

great, nice work 👍 squash the changes and i'll approve

[timothycoleman]

for posterity, shaan and i identified that the scenario where the if (allDiscardedSoFar) check returns false is quite narrow:

1. we have an event E in the index that is not in the chunk
2. the logical chunk that it would have been in, in fact, has no prepares left in it at all
3. there is an earlier event for the same stream, the stream has a maxage, and we MaybeDiscard it in the calculator
4. but we do, in fact, want to keep that previous event D based on the current metadata (which means that E was removed in error or because of an earlier metadata)

the net effect is that in this scenario, if the allDiscardedSoFar check was not present, D would be removed, perhaps to the surprise of the user.

note that a stream in this state is hard to read. in order to read D the read must not also try to include E

if it wasn't for this corner case, then the allDiscardedSoFar variable and the AlreadyDiscarded discard decision would be unnecessary, we could just return Discard (although it would unnecessarily add weight to that chunk)

[timothycoleman]

👍

[shaan1337]

great, nice work +1 squash the changes and i'll approve

thanks for the thorough review!

[timothycoleman]

great, nice work +1 squash the changes and i'll approve

thanks for the thorough review!

thanks for the careful code!

[github-actions]

@hayley-jean 👉 Created pull request targeting release/oss-v22.10: #3821