Sync client should wait for MARK message before marking subscriptions complete

[sync-by-unito]

Due to a recent change on the server, we may not receive all changes when bootstrapping a flx realm, so a subscription may be prematurely marked as complete. We should probably send a MARK when sending a QUERY message to make sure we wait for all objects matching the subscription before marking it complete.

This has several slack threads associated with it:

https://mongodb.slack.com/archives/G0121N9LJ14/p1658977516466089
https://mongodb.slack.com/archives/C01N0DHTW9X/p1658150145232439
https://mongodb.slack.com/archives/G0121N9LJ14/p1659533132162879

[sync-by-unito]

➤ James Stone commented:

[~jonathan.reams@mongodb.com] can you assign this to someone on your team or if it is not a high priority, place it on the backlog?

[nirinchev]

I believe this is causing issues with automatic client reset for flexible sync. I have not reviewed the actual reset code, but if it's relying on subscription synchronization without waiting for download, this would explain some timing issues we're seeing in our tests.

@ironage can you check if that may be an issue for the reset logic, particularly when recovering changes.

[ironage]

Downloading a fresh FLX Realm during client reset just waits for a completed subscription set so yes, it is a problem there as well.

[jbreams]

Hi @nirinchev , I will drop everything to work on this right away so your tests pass! Thanks!

[jbreams]

The long-term fix for this requires new testing infrastructure on the server side. However, I've merged a stop-gap fix for how FLX sync client resets are handled so that client resets with recovery mode always wait to be fully synchronized with the server before doing any of their recovery work. Hopefully this will unblock @nirinchev's client reset work while we build out the infrastructure to test the real fix for this problem.