Tombstone local pending blobs to be resilient to reupload

[dannimad]

When a blob attach op is reuploaded (resubmit/ expired or stashed scenarios), we could hit asserts checking for its existence in the pending blob list if such op was already processed. Adding a tombstone list to make sure we recognize such blobs

[ChumpChief]

I think I see what you're doing with the tombstoning, but it also feels like maybe if we need that then there was some earlier invariant that was violated. A couple particular questions:

• Is it valid for the same localId to appear in an opsInFlight list twice? Should we error if we try to insert twice? Should opsInFlight be a Map<string, Set<string>>?
• I'm not sure I understand how we get to the point that there are two outstanding blobAttach ops - it looks like the opsent flag should prevent us from ever calling sendBlobAttachOp more than once for a single localId. Could you please elaborate on the sequence of events that causes this error?

[dannimad]

opSent does not prevent us to resubmit the op (look at reSubmit method). The scenario is:
-blob is uploaded
-op is sent
-disconnect
-reconnect after TTL expires which triggers sendBlobAttachOp logic for expired blob which is simply reuploading the blob.
-once the blob uploads, another blob attach op will be sent.
Checking the failing test should help to understand the scenario.

I never quite understood opsInFlight logic so it may be that a Set is more appropriate. I can't think of a scenario in which one response id has multiple local Ids. I'll test it and see if something fails.

[microsoft-github-policy-service]

This PR has been automatically marked as stale because it has had no activity for 60 days. It will be closed if no further activity occurs within 8 days of this comment. Thank you for your contributions to Fluid Framework!