Fix TestPartitionReader_WaitReadConsistency flakyness

[pracucci]

What this PR does

@dimitarvdimitrov reported me that TestPartitionReader_WaitReadConsistency is flaky. We've seen it failing in this CI run because of:

--- FAIL: TestPartitionReader_WaitReadConsistency (0.00s)
    --- FAIL: TestPartitionReader_WaitReadConsistency/should_return_after_all_produced_records_up_have_been_consumed (1.02s)
        reader_test.go:128: produced 2 records
        reader_test.go:137: consumed records: [record-1]
        reader_test.go:148: started waiting for read consistency
        reader_test.go:153: finished waiting for read consistency
        reader_test.go:145: consumed records: [record-2]
        writer_test.go:394: 
            	Error Trace:	/__w/mimir/mimir/pkg/storage/ingest/writer_test.go:394
            	            				/__w/mimir/mimir/pkg/storage/ingest/reader_test.go:132
            	Error:      	Not equal: 
            	            	expected: []int{0, 1}
            	            	actual  : []int{1, 0}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,4 +1,4 @@
            	            	 ([]int) (len=2) {
            	            	- (int) 0,
            	            	- (int) 1
            	            	+ (int) 1,
            	            	+ (int) 0
            	            	 }
            	Test:       	TestPartitionReader_WaitReadConsistency/should_return_after_all_produced_records_up_have_been_consumed
FAIL

The problem is that there's a race between the 2 async functions, where the function calling consumer.waitRecords() may end after the one calling reader.WaitReadConsistency(ctx) even if reader.WaitReadConsistency(ctx) effectively waited for all records to be consumed.

In this PR I'm reworking the test. Instead of using runAsyncAndAssertCompletionOrder(), I define a custom consume function which adds a delay on the 2nd (and last) record consumption and then increases a counter of processed records. Such counter is increased after the record has been processed by consumption (processing time is simulated with the artificial delay) but before the consume function returns. Then we can just assert that the expected number of records have been consumed once reader.WaitReadConsistency(ctx) returns.

Which issue(s) this PR fixes or relates to

N/A

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[dimitarvdimitrov]

LGTM, thanks for fixing this