Skip to content

Support applyingStashedOps while detached#19802

Merged
anthony-murphy merged 16 commits into
microsoft:mainfrom
anthony-murphy:stash-while-detached
Feb 28, 2024
Merged

Support applyingStashedOps while detached#19802
anthony-murphy merged 16 commits into
microsoft:mainfrom
anthony-murphy:stash-while-detached

Conversation

@anthony-murphy
Copy link
Copy Markdown
Contributor

@anthony-murphy anthony-murphy commented Feb 26, 2024

Background

This change is in preparation for allowing consumers of fluid to call serialize on a container while a container is in an attaching state, which will include a failed attach where the failure didn't cause the container to close. This will aid in the case of offline, network, or server outage by allowing the application to capture the un-attached state of the container, preserve it locally, and later create a new file from it.

AB#5502

Overview

The majority of these changes are test changes which simulate having DDS enter the attaching state, capturing their state, and then rehydrating to a new instance of the DDS.

This was made possible with #19518. As that change made DDS apply stashed ops via the DDSes existing functions to apply local changes, and since those existing functions already supported the detached state, there were no additional per DDS modifications necessary to apply stashed ops on a rehydrated detached container.

Beyond the test changes this change makes the attachment state visible to the apply stashed op pipeline, so it can know to not expect new ops to be generated when detached, and not populate the pending state manager.

The next PR (AB#7077) of this work will enable capturing the pending state while detached. That change will bring end to end test coverage that can't be done currently. However, there is substantial coverage for the internals in this PR via the integration with the stress harness.

AB#7078

@github-actions github-actions Bot added area: dds Issues related to distributed data structures area: runtime Runtime related issues base: main PRs targeted against main branch labels Feb 26, 2024
@anthony-murphy anthony-murphy force-pushed the stash-while-detached branch 2 times, most recently from 80e44ee to 4620137 Compare February 26, 2024 21:21
@github-actions github-actions Bot added the public api change Changes to a public API label Feb 26, 2024
@github-actions github-actions Bot added the public api change Changes to a public API label Feb 27, 2024
Comment thread packages/dds/task-manager/src/test/taskManager.fuzz.spec.ts Outdated
@anthony-murphy anthony-murphy marked this pull request as ready for review February 27, 2024 00:21
@anthony-murphy anthony-murphy requested review from a team as code owners February 27, 2024 00:21
Comment thread packages/dds/test-dds-utils/src/clientLoading.ts Outdated
* Creates the load data from the client. The load data include everything needed to load a new client. It includes the summaries and the minimumSequenceNumber.
* @internal
*/
export function createLoadData(client: Client<IChannelFactory>): ClientLoadData {
Copy link
Copy Markdown
Contributor Author

@anthony-murphy anthony-murphy Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is new, the rest is copy pasted from dds test harness. this function unifies getting a load data out of a client, which was replicated all over the place

const localOpMetadata = await this.stateHandler.applyStashedOp(nextMessage.content);
nextMessage.localOpMetadata = localOpMetadata;
if (!this.stateHandler.isAttached()) {
if (localOpMetadata !== undefined) {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we are not attached, then there shouldn't be any local op metadata, and we don't track pending messages, as there are not messages when not attached.

public applyStashedOp(content: any): unknown {
try {
this.stashedOpMd = createStashedOpMetadata();
this.stashedOpMd = this.isAttachedAndVisible() ? createStashedOpMetadata() : undefined;
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only create and return stashed op metadata if we are attached and visible, otherwise we'll just pass undefined up, and it will be dropped by pending state manager

@msfluid-bot
Copy link
Copy Markdown
Collaborator

msfluid-bot commented Feb 27, 2024

@fluid-example/bundle-size-tests: +809 Bytes
Metric NameBaseline SizeCompare SizeSize Diff
aqueduct.js 513.7 KB 514.2 KB +517 Bytes
connectionState.js 680 Bytes 680 Bytes No change
containerRuntime.js 247.81 KB 248.01 KB +204 Bytes
loader.js 171.3 KB 171.32 KB +22 Bytes
map.js 46.69 KB 46.7 KB +11 Bytes
matrix.js 148.74 KB 148.74 KB No change
odspDriver.js 97.31 KB 97.33 KB +22 Bytes
odspPrefetchSnapshot.js 42.28 KB 42.29 KB +11 Bytes
sharedString.js 167.32 KB 167.33 KB +11 Bytes
sharedTree.js 334.63 KB 334.63 KB No change
Total Size 1.87 MB 1.87 MB +809 Bytes

Baseline commit: dca5a73

Generated by 🚫 dangerJS against 1803321

Comment thread packages/dds/tree/src/test/shared-tree/fuzz/composeVsIndividual.fuzz.spec.ts Outdated
…l.fuzz.spec.ts

Co-authored-by: alex-pardes <105307164+alex-pardes@users.noreply.github.com>
Comment thread packages/dds/test-dds-utils/src/ddsFuzzHarness.ts Outdated
Co-authored-by: Connor Skees <39542938+connorskees@users.noreply.github.com>
@anthony-murphy anthony-murphy enabled auto-merge (squash) February 28, 2024 19:02
@anthony-murphy anthony-murphy merged commit 11e0d79 into microsoft:main Feb 28, 2024
@anthony-murphy anthony-murphy deleted the stash-while-detached branch February 29, 2024 02:40
anthony-murphy added a commit that referenced this pull request Mar 1, 2024
…19893)

## Background
This change is the final change for allowing consumers of fluid to call
serialize on a container while a container is in an attaching state,
which will include a failed attach where the failure didn't cause the
container to close. This will aid in the case of offline, network, or
server outage by allowing the application to capture the un-attached
state of the container, preserve it locally, and later create a new file
from it.


[AB#5502](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/5502)

## Overview
This change enables calling container.serialize while a container is in
the attaching state. It is the culmination of many smaller changes that
have gone in to enable this change.
- #18829
- #19400
 - #19246
 - #19517
 - #19518
 - #19590
 - #19634
 - #19802

---------

Co-authored-by: Tony Murphy <anthonm@microsoft.com>
Co-authored-by: Daniel Madrid <105010181+dannimad@users.noreply.github.com>
Co-authored-by: jzaffiro <110866475+jzaffiro@users.noreply.github.com>
Co-authored-by: Matt Rakow <ChumpChief@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: dds: sharedstring area: dds: tree area: dds Issues related to distributed data structures area: runtime Runtime related issues base: main PRs targeted against main branch public api change Changes to a public API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants