Shared Tree: Schema FormatV2 and codecs #24812

TommyBrosman · 2025-06-10T18:57:44Z

Adds support for persisting metadata along with the schema.

Changes

Add schema FormatV2
- Node schema persisted metadata
- Field schema persisted metadata
Schema codec for FormatV2
- Add encodeV2 method to TreeNodeStoredSchema hierarchy
- Persist node schema metadata in TreeNodeStoredSchema
- Add support for encoding v2 fields
- Update the storedSchemaDecodeDispatcher to dispatch on the inner kind union in FormatV2 and pass node schema metadata
- Create and register schema change codec handling FormatV2
- Create and register schema codec handling FormatV2
- Generate new snapshots for schema tests
Add a new Shared Tree format version
- Update enums
- Generate new snapshots for existing Shared Tree format tests
simple-tree changes
- Persist node metadata in arraySchema, mapSchema, objectSchema
- Pass metadata to NodeStoredSchema types in toStoredSchema.ts
Other tests
- Update codec.spec to cover FormatV2 as well as FormatV1
- FormatV2-specific tests
API documentation
- Changeset covering the new APIs
Feature flag for enabling Shared Tree Format v5

Notes for reviewers

Todo

Merge with main once Shared Tree: In-memory format and API changes for persisted schema metadata #24759 completes
Write changeset

- Added in-memory schema format v2. - Implemented the schema format v2 codec. - Updated test utils that previously assumed there was only one format. - Added snapshots for many of the v2 tests.

…ion as a parameter. - Fixed naming of schema index formats. - Changed schemaChangeFormat to be schema format-agnostic.

…formats. - Moved SchemaCodecVersion.

Co-authored-by: Craig Macomber (Microsoft) <42876482+CraigMacomber@users.noreply.github.com>

… we want something more intuitive (e.g., minimum client version). - Split the encode method for TreeNodeStoredSchema.

…luidFramework into metadata-schema-3

…cVersion. - Changed schema-stored/FormatV2 to only define the new elements. - Updated all instantiations of TreeFieldStoredSchema so that persistedMetadata is specified. - HACK: importCompressed needs to handle both schema formats. A few of the roundtrip JSON tests are failing.

Co-authored-by: Alex Villarreal <716334+alexvy86@users.noreply.github.com>

- Made importCompressed always use a v1 schema when encoding.

- Added better test data to one of the codec tests for the v2 schema format.

…added in another PR. - Updated a comment.

- Added tests calling schemaFactoryAlpha.

.changeset/spotty-bottles-rest.md

… clearer now.

.changeset/spotty-bottles-rest.md

packages/dds/tree/src/feature-libraries/schema-index/codec.ts

packages/dds/tree/src/core/schema-stored/schema.ts

packages/dds/tree/src/test/shared-tree/sharedTree.spec.ts

Co-authored-by: Abram Sanderson <Abram.sanderson@gmail.com>

packages/dds/tree/src/core/schema-stored/schema.ts

.changeset/whole-days-argue.md

alexvy86 · 2025-06-16T23:52:11Z

/azp run Build - client packages

azure-pipelines · 2025-06-16T23:52:29Z

Azure Pipelines successfully started running 1 pipeline(s).

alexvy86

Approving for docs. Left a few comments on the changeset but can be addressed in a follow-up PR.

.changeset/spotty-bottles-rest.md

Co-authored-by: Alex Villarreal <716334+alexvy86@users.noreply.github.com>

…work into formatv2

github-actions · 2025-06-17T00:46:47Z

🔗 No broken links found! ✅

Your attention to detail is admirable.

linkcheck output


> fluid-framework-docs-site@0.0.0 ci:check-links /home/runner/work/FluidFramework/FluidFramework/docs
> start-server-and-test "npm run serve -- --no-open" 3000 check-links

1: starting server using command "npm run serve -- --no-open"
and when url "[ 'http://127.0.0.1:3000' ]" is responding with HTTP status code 200
running tests using command "npm run check-links"


> fluid-framework-docs-site@0.0.0 serve
> docusaurus serve --no-open

[SUCCESS] Serving "build" directory at: http://localhost:3000/

> fluid-framework-docs-site@0.0.0 check-links
> linkcheck http://localhost:3000 --skip-file skipped-urls.txt

Crawling...

Stats:
  225443 links
    1710 destination URLs
    1941 URLs ignored
       0 warnings
       0 errors

rohandubal

Reviewed the delta since previous approval; merge from main and a doc update.

TommyBrosman and others added 30 commits May 12, 2025 14:59

Initial change.

bd8406a

- Added in-memory schema format v2. - Implemented the schema format v2 codec. - Updated test utils that previously assumed there was only one format. - Added snapshots for many of the v2 tests.

persistedMetadata is now backed by an object instead of a string.

3165fe3

Updated codec.spec.ts to cover FormatV2.

9709ede

Removed extraneous whitespace change.

ba058e1

- Changed TreeNodeStoredSchema implementations to take the write vers…

b819742

…ion as a parameter. - Fixed naming of schema index formats. - Changed schemaChangeFormat to be schema format-agnostic.

- Changed the stored schema implementation to handle multiple schema …

0b5c842

…formats. - Moved SchemaCodecVersion.

Merge branch 'main' into metadata-schema-3

ecdca7a

Apply suggestions from code review

1117c2c

Co-authored-by: Craig Macomber (Microsoft) <42876482+CraigMacomber@users.noreply.github.com>

- Switched to using the SchemaCodecVersion in public APIs. Eventually…

94a5c39

… we want something more intuitive (e.g., minimum client version). - Split the encode method for TreeNodeStoredSchema.

Merge branch 'metadata-schema-3' of https://github.com/TommyBrosman/F…

302cb21

…luidFramework into metadata-schema-3

Minor: reverted changes to FluidClientVersion utils.

3275ee0

Switched back to using min client version for importCompressed.

7c48b9d

Minor: reverted accidental change.

a2f4195

Apply suggestions from code review

4786dcc

Co-authored-by: Alex Villarreal <716334+alexvy86@users.noreply.github.com>

Made persisted metadata schema field naming consistent.

6fa8836

- Created a separate storedSchemaDecodeDispatcher for v2 schemas.

e4064c7

- Made importCompressed always use a v1 schema when encoding.

- Refactored the node kind dispatch.

ab1b882

- Added better test data to one of the codec tests for the v2 schema format.

Updated snapshots.

1027daa

- Removed minimum client version changes. Equivalent changes will be …

e527eba

…added in another PR. - Updated a comment.

Removed more minimumClientVersion glue.

3eae1f6

Updated API files.

98e1b76

Rename: SchemaCodecVersion -> SchemaVersion.

ecc77c9

Merge branch 'main' into metadata-schema-3

37e900f

Removed top-level persisted metadata.

737633b

Fixed comments.

4bb136c

Exposed persistedMetadata as a JsonCompatibleReadOnlyObject.

0bcb90f

Changeset.

a675338

Wired up toStoredSchema. Still needs tests.

1d1ef5f

- Updated an old snapshot that didn't include the metadata field.

b9898e8

- Added tests calling schemaFactoryAlpha.

TommyBrosman commented Jun 13, 2025

View reviewed changes

.changeset/spotty-bottles-rest.md Outdated Show resolved Hide resolved

TommyBrosman added 4 commits June 13, 2025 16:18

Updated docs. The explanation of how to turn on persisted metadata is…

03bce3f

… clearer now.

Fixed indentation.

51a6c12

Deleted old changeset.

9d42ebb

Merge branch 'main' into formatv2

52bbe1a

Abe27342 reviewed Jun 13, 2025

View reviewed changes

TommyBrosman commented Jun 16, 2025

View reviewed changes

packages/dds/tree/src/test/shared-tree/sharedTree.spec.ts Show resolved Hide resolved

TommyBrosman and others added 5 commits June 16, 2025 10:57

Refactored fieldsObject encoding so the code is no longer duplicated.

5abf135

Fixed a test.

59db6ac

Apply suggestions from code review

8454d80

Co-authored-by: Abram Sanderson <Abram.sanderson@gmail.com>

Refactored node schema encoding to reduce duplication.

ef2fb5e

Merge branch 'main' into formatv2

633c551

Abe27342 reviewed Jun 16, 2025

View reviewed changes

packages/dds/tree/src/core/schema-stored/schema.ts Outdated Show resolved Hide resolved

Abe27342 reviewed Jun 16, 2025

View reviewed changes

packages/dds/tree/src/core/schema-stored/schema.ts Outdated Show resolved Hide resolved

Abe27342 reviewed Jun 16, 2025

View reviewed changes

.changeset/whole-days-argue.md Show resolved Hide resolved

Small refactor.

b145898

TommyBrosman enabled auto-merge (squash) June 16, 2025 21:05

Abe27342 approved these changes Jun 16, 2025

View reviewed changes

alexvy86 approved these changes Jun 17, 2025

View reviewed changes

.changeset/spotty-bottles-rest.md Outdated Show resolved Hide resolved

.changeset/spotty-bottles-rest.md Outdated Show resolved Hide resolved

TommyBrosman and others added 4 commits June 16, 2025 17:30

Update .changeset/spotty-bottles-rest.md

8a64c0a

Co-authored-by: Alex Villarreal <716334+alexvy86@users.noreply.github.com>

Tweaked documentation.

2b2dc5f

Merge branch 'formatv2' of https://github.com/TommyBrosman/FluidFrame…

05f00bd

…work into formatv2

Merge branch 'main' into formatv2

cd49ae1

rohandubal approved these changes Jun 17, 2025

View reviewed changes

TommyBrosman merged commit 3f81ab5 into microsoft:main Jun 17, 2025
36 checks passed

This was referenced Jun 17, 2025

Upcoming Release: FluidFramework v2.43.0 #22101

Open

Upcoming Release: FluidFramework v2.43.0 tylerbutler/FluidFramework#26

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Shared Tree: Schema FormatV2 and codecs #24812

Shared Tree: Schema FormatV2 and codecs #24812

Uh oh!

TommyBrosman commented Jun 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexvy86 commented Jun 16, 2025

Uh oh!

azure-pipelines bot commented Jun 16, 2025

Uh oh!

alexvy86 left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 17, 2025

Uh oh!

rohandubal left a comment

Uh oh!

Uh oh!

Uh oh!

Shared Tree: Schema FormatV2 and codecs #24812

Shared Tree: Schema FormatV2 and codecs #24812

Uh oh!

Conversation

TommyBrosman commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Notes for reviewers

Todo

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexvy86 commented Jun 16, 2025

Uh oh!

azure-pipelines bot commented Jun 16, 2025

Uh oh!

alexvy86 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 17, 2025

linkcheck output

Uh oh!

rohandubal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

TommyBrosman commented Jun 10, 2025 •

edited

Loading