Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Put channels in .channels subtrees in summary #5280

Merged
merged 47 commits into from
Mar 23, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
01745d5
Update lock-files
Feb 19, 2021
e94a8e2
Expose summarize function which returns tree
Feb 23, 2021
420f6a0
Add simple summaries end-to-end test
Feb 25, 2021
a7fe1e0
Merge branch 'main' into sumamrize-for-tests
Feb 25, 2021
d0f932a
Increment summary format version and put data stores/dds under .chann…
Feb 26, 2021
5dfb0cf
Fix some tests
Feb 26, 2021
15aaa0a
Merge branch 'sumamrize-for-tests' into summary-name-collisions-write
Feb 26, 2021
205a66b
Update basic summary test for subtrees
Feb 26, 2021
4390009
Write in new format in all places
Mar 2, 2021
791c9ab
Add runtime option to disable isolated channels
Mar 3, 2021
8873cda
Merge branch 'main' into summary-name-collisions-write
Mar 3, 2021
a389097
Fix handling initial summary from attach op
Mar 4, 2021
7d50749
Merge branch 'main' into summary-name-collisions-write
Mar 4, 2021
8969b11
Strip out new format version features from snapshots in comparison
Mar 4, 2021
09eab2c
Merge branch 'main' into summary-name-collisions-write
Mar 4, 2021
aeef805
Fix all broken tests and places where summaryFormatVersion matters
Mar 8, 2021
81d3065
Merge branch 'main' into summary-name-collisions-write
Mar 8, 2021
2f58830
Final fixes
Mar 8, 2021
b393dc7
Update tests
Mar 9, 2021
dab4912
Merge branch 'main' into summary-name-collisions-write
Mar 9, 2021
c7b50ba
Add back missing skip
Mar 9, 2021
be7216b
Disable by default
Mar 9, 2021
23504c0
Merge branch 'main' into summary-name-collisions-write
Mar 9, 2021
8bc2977
UseContainerRuntime for checking if disabled on write; use helper fu…
Mar 10, 2021
2a7a344
Fix tests
Mar 10, 2021
d0daa5a
Fix summaries test
Mar 11, 2021
2f15f97
Merge branch 'main' into summary-name-collisions-write
Mar 11, 2021
a35dc84
Snapshot should always include metadata blob
Mar 11, 2021
50adf47
Random lerna fix
Mar 11, 2021
789273a
Do not modify snapshots for comparison
Mar 11, 2021
857b4cd
Merge branch 'main' into summary-name-collisions-write
Mar 11, 2021
6dc045d
Remove extra constructor arg
Mar 11, 2021
a9f96cc
only expose disableIsolatedChannels instead of entire ContainerRuntime
Mar 11, 2021
b1a8eea
Point to updated generated snapshot test data
Mar 12, 2021
db328c6
Inline formMetadata calls
Mar 12, 2021
b5317e8
Switch datastore attributes to summaryFormatVersion
Mar 12, 2021
cd6dd02
Use undefined for metadata
Mar 12, 2021
64cd716
Merge branch 'main' into summary-name-collisions-write
Mar 16, 2021
111c6b0
Small PR changes
Mar 17, 2021
eb0fbb8
Add a few more tests
Mar 17, 2021
74d00f8
Do not write in new format version unless isolated channels are enabled
Mar 19, 2021
16d9cf2
Revert snapshot test generation
Mar 19, 2021
bda90e8
Fix summaries test
Mar 19, 2021
a5f06cc
Run all 8 combinations for intiialization
Mar 19, 2021
1032cbe
PR comments
Mar 22, 2021
fe5eb8d
Merge branch 'main' into summary-name-collisions-write
Mar 22, 2021
be22647
Fix build
Mar 22, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 22 additions & 5 deletions packages/runtime/container-runtime/src/containerRuntime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ import { CreateContainerError } from "@fluidframework/container-utils";
import { runGarbageCollection } from "@fluidframework/garbage-collector";
import {
BlobTreeEntry,
TreeTreeEntry,
} from "@fluidframework/protocol-base";
import {
IClientDetails,
Expand Down Expand Up @@ -94,6 +95,7 @@ import {
IChannelSummarizeResult,
CreateChildSummarizerNodeParam,
SummarizeInternalFn,
channelsTreeName,
} from "@fluidframework/runtime-definitions";
import {
addBlobToSummary,
Expand Down Expand Up @@ -124,7 +126,8 @@ import {
chunksBlobName,
IContainerRuntimeMetadata,
metadataBlobName,
} from "./snapshot";
wrapSummaryInChannelsTree,
} from "./summaryFormat";

export enum ContainerMessageType {
// An op to be delivered to store
Expand Down Expand Up @@ -943,7 +946,15 @@ export class ContainerRuntime extends TypedEventEmitter<IContainerRuntimeEvents>
* @deprecated - Use summarize to get summary of the container runtime.
*/
public async snapshot(): Promise<ITree> {
const root: ITree = { entries: await this.dataStores.snapshot() };
const root: ITree = { entries: [] };

root.entries.push(new TreeTreeEntry(
channelsTreeName,
{ entries: await this.dataStores.snapshot() },
));

const metadata: IContainerRuntimeMetadata = { snapshotFormatVersion: 1 };
root.entries.push(new BlobTreeEntry(metadataBlobName, JSON.stringify(metadata)));
arinwt marked this conversation as resolved.
Show resolved Hide resolved

if (this.chunkMap.size > 0) {
root.entries.push(new BlobTreeEntry(chunksBlobName, JSON.stringify([...this.chunkMap])));
Expand All @@ -953,6 +964,8 @@ export class ContainerRuntime extends TypedEventEmitter<IContainerRuntimeEvents>
}

private addContainerBlobsToSummary(summaryTree: ISummaryTreeWithStats) {
const metadata: IContainerRuntimeMetadata = { snapshotFormatVersion: 1 };
addBlobToSummary(summaryTree, metadataBlobName, JSON.stringify(metadata));
agarwal-navin marked this conversation as resolved.
Show resolved Hide resolved
if (this.chunkMap.size > 0) {
const content = JSON.stringify([...this.chunkMap]);
addBlobToSummary(summaryTree, chunksBlobName, content);
Expand Down Expand Up @@ -1346,6 +1359,8 @@ export class ContainerRuntime extends TypedEventEmitter<IContainerRuntimeEvents>

private async summarizeInternal(fullTree: boolean, trackState: boolean): Promise<ISummarizeInternalResult> {
const summarizeResult = await this.dataStores.summarize(fullTree, trackState);
// Wrap data store summaries in .channels subtree.
wrapSummaryInChannelsTree(summarizeResult);
this.addContainerBlobsToSummary(summarizeResult);
return {
...summarizeResult,
Expand All @@ -1364,9 +1379,11 @@ export class ContainerRuntime extends TypedEventEmitter<IContainerRuntimeEvents>
}

public createSummary(): ISummaryTree {
const summaryTree = this.dataStores.createSummary();
this.addContainerBlobsToSummary(summaryTree);
return summaryTree.summary;
const summarizeResult = this.dataStores.createSummary();
// Wrap data store summaries in .channels subtree.
wrapSummaryInChannelsTree(summarizeResult);
this.addContainerBlobsToSummary(summarizeResult);
return summarizeResult.summary;
}

public async getAbsoluteUrl(relativeUrl: string): Promise<string | undefined> {
Expand Down
12 changes: 8 additions & 4 deletions packages/runtime/container-runtime/src/dataStoreContext.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,14 +64,15 @@ import { addBlobToSummary, convertSummaryTreeToITree } from "@fluidframework/run
import { ContainerRuntime } from "./containerRuntime";
import {
dataStoreAttributesBlobName,
DataStoreSnapshotFormatVersion,
} from "./snapshot";
DataStoreSummaryFormatVersion,
wrapSummaryInChannelsTree,
} from "./summaryFormat";

function createAttributes(pkg: readonly string[], isRootDataStore: boolean): IFluidDataStoreAttributes {
const stringifiedPkg = JSON.stringify(pkg);
return {
pkg: stringifiedPkg,
snapshotFormatVersion: "0.1",
snapshotFormatVersion: 2,
isRootDataStore,
};
}
Expand All @@ -87,7 +88,7 @@ export function createAttributesBlob(pkg: readonly string[], isRootDataStore: bo
*/
export interface IFluidDataStoreAttributes {
pkg: string;
readonly snapshotFormatVersion: DataStoreSnapshotFormatVersion;
readonly snapshotFormatVersion: DataStoreSummaryFormatVersion;
/**
* This tells whether a data store is root. Root data stores are never collected.
* Non-root data stores may be collected if they are not used. If this is not present, default it to
Expand Down Expand Up @@ -382,6 +383,9 @@ export abstract class FluidDataStoreContext extends TypedEventEmitter<IFluidData
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const summarizeResult = await this.channel!.summarize(fullTree, trackState);

// Wrap dds summaries in .channels subtree.
wrapSummaryInChannelsTree(summarizeResult);

// Add data store's attributes to the summary.
const { pkg, isRootDataStore } = await this.getInitialSnapshotDetails();
const attributes: IFluidDataStoreAttributes = createAttributes(pkg, isRootDataStore);
Expand Down
6 changes: 3 additions & 3 deletions packages/runtime/container-runtime/src/dataStores.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,10 @@ import {
LocalDetachedFluidDataStoreContext,
} from "./dataStoreContext";
import {
ContainerRuntimeSnapshotFormatVersion,
ContainerRuntimeSummaryFormatVersion,
dataStoreAttributesBlobName,
nonDataStorePaths,
} from "./snapshot";
} from "./summaryFormat";

/**
* This class encapsulates data store handling. Currently it is only used by the container runtime,
Expand Down Expand Up @@ -501,7 +501,7 @@ export class DataStores implements IDisposable {

export function getSnapshotForDataStores(
snapshot: ISnapshotTree | undefined,
snapshotFormatVersion: ContainerRuntimeSnapshotFormatVersion,
snapshotFormatVersion: ContainerRuntimeSummaryFormatVersion,
): ISnapshotTree | undefined {
if (!snapshot) {
return undefined;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
* Licensed under the MIT License.
*/

import { ISnapshotTree } from "@fluidframework/protocol-definitions";
import { channelsTreeName } from "@fluidframework/runtime-definitions";
import { SummaryType } from "@fluidframework/protocol-definitions";
import { channelsTreeName, ISummaryTreeWithStats } from "@fluidframework/runtime-definitions";

export type ContainerRuntimeSnapshotFormatVersion =
export type ContainerRuntimeSummaryFormatVersion =
/**
* Version 0: format version is missing from snapshot.
* Version 0: format version is missing from summary.
* This indicates it is an older version.
*/
| undefined
Expand All @@ -18,9 +18,9 @@ export type ContainerRuntimeSnapshotFormatVersion =
*/
| 1;

export type DataStoreSnapshotFormatVersion =
export type DataStoreSummaryFormatVersion =
/**
* Version 0: format version is missing from snapshot.
* Version 0: format version is missing from summary.
* This indicates it is an older version.
*/
| undefined
Expand All @@ -35,8 +35,8 @@ export type DataStoreSnapshotFormatVersion =
*/
| 2;

export function snapshotFormatVersionToNumber(
version: ContainerRuntimeSnapshotFormatVersion | DataStoreSnapshotFormatVersion,
export function summaryFormatVersionToNumber(
version: ContainerRuntimeSummaryFormatVersion | DataStoreSummaryFormatVersion,
): number {
if (version === undefined) {
return 0;
Expand All @@ -52,30 +52,43 @@ export const chunksBlobName = ".chunks";
export const blobsTreeName = ".blobs";

export interface IContainerRuntimeMetadata {
snapshotFormatVersion: ContainerRuntimeSnapshotFormatVersion;
snapshotFormatVersion: ContainerRuntimeSummaryFormatVersion;
}

export const protocolTreeName = ".protocol";

/**
* List of tree IDs at the container level which are reserved.
* This is for older versions of snapshots that do not yet have an
* This is for older versions of summaries that do not yet have an
* isolated data stores namespace. Without the namespace, this must
* be used to prevent name collisions with data store IDs.
*/
export const nonDataStorePaths = [protocolTreeName, ".logTail", ".serviceProtocol", blobsTreeName];

export const dataStoreAttributesBlobName = ".component";

export interface IRuntimeSnapshot {
id: string | null;
blobs: {
[chunksBlobName]: string;
[metadataBlobName]: string;
/**
* Modifies summary tree and stats to put tree under .channels tree.
* Converts from: {
* type: SummaryType.Tree,
* tree: { a: {...}, b: {...}, c: {...} },
* }
* to: {
* type: SummaryType.Tree,
* tree: {
* ".channels": {
* type: SummaryType.Tree,
* tree: { a: {...}, b: {...}, c: {...} }
* },
* },
* }
* And adds +1 to treeNodeCount in stats.
* @param summarizeResult - summary tree and stats to modify
*/
export function wrapSummaryInChannelsTree(summarizeResult: ISummaryTreeWithStats): void {
summarizeResult.summary = {
type: SummaryType.Tree,
tree: { [channelsTreeName]: summarizeResult.summary },
};
trees: {
[protocolTreeName]: ISnapshotTree;
[blobsTreeName]: ISnapshotTree;
[channelsTreeName]: ISnapshotTree;
},
summarizeResult.stats.treeNodeCount++;
}
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ import {
RemotedFluidDataStoreContext,
} from "../dataStoreContext";
import { ContainerRuntime } from "../containerRuntime";
import { dataStoreAttributesBlobName } from "../snapshot";
import { dataStoreAttributesBlobName } from "../summaryFormat";

describe("Data Store Context Tests", () => {
const dataStoreId = "Test1";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { strict as assert } from "assert";
import { ISnapshotTree } from "@fluidframework/protocol-definitions";
import { channelsTreeName } from "@fluidframework/runtime-definitions";
import { getSnapshotForDataStores } from "../dataStores";
import { nonDataStorePaths } from "../snapshot";
import { nonDataStorePaths } from "../summaryFormat";

describe("Runtime", () => {
describe("Container Runtime", () => {
Expand Down