Writes to a persistent map store may be lost when coalescing is turned on.

[palisie]

It is a continuation of issue #15060.
Correction of this issue used sequence number of DelayedEntry to detect a situation when at the time of removing already persisted DelayedEntries there is already a newer DelayedEntry.
But there is still a small window when sequence number is not yet initialized but the DelayedEntry has already been added to the queue. See WriteBehindStore::add() @ version 4.1.1

public void add(DelayedEntry<Data, Object> delayedEntry) {
        writeBehindQueue.addLast(delayedEntry, false);

        stagingArea.put(delayedEntry.getKey(), delayedEntry);

        delayedEntry.setSequence(sequence.incrementAndGet());
    }

We are still observing occasional lost updates and we think this is the reason. Under normal circumstances the issue reproduces very rarely. To increase chances of reproduction it is needed to increase frequency of StoreWorker runs.

Attached a diagram of the problematic mechanism

Attached a test case. Contains a modified StoreWorker to increase chance of reproduction.
CoalescedSequenceTest.zip

[nfrankel]

Just barging in to to thank you for all the diagnostics data @palisie. Great job!

[Holmistr]

Hi @palisie , thanks a lot for the report and a nice description. @ahmetmircik will take a look at this issue as he's the expert on the MapLoader stuff but he's on vacation this week, so please bear with us :)

[palisie]

I'm adding a corrected version of the test case. Added a wait time after changing the items in cache to allow writes to finish and eliminate any false positives.
CoalescedSequenceTest-2.zip