Add support for JSON, composite, include and exclude Cosmos indexes by AndriySvyryd · Pull Request #38360 · dotnet/efcore

AndriySvyryd · 2026-06-04T06:51:20Z

Add support for chaining into complex collections in indexes
Add support for keys spanning non-collection complex properties

Fixes #38067
Fixes #17303

Copilot

Pull request overview

This PR extends the EF Core Cosmos provider to support richer indexing scenarios (including indexes through complex types/complex collections, composite indexes, and container-level automatic indexing include/exclude behavior), and updates validation and tests accordingly.

Changes:

Add Cosmos container automatic-indexing configuration (HasAutomaticIndexing(...) + Except(...)) and flow these settings into container creation.
Generate Cosmos indexing-policy paths for scalar properties nested in complex types and complex collections (including escaping/quoting as needed), and emit composite/vector/full-text index policy entries.
Update Cosmos model validation and expand functional/unit test coverage (including compiled-model baselines) for the new indexing behavior.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/EFCore.Cosmos.Tests/Storage/Internal/CosmosClientWrapperTest.cs	Adds unit coverage for JSON path generation/escaping for scalar, complex, and complex-collection segments.
test/EFCore.Cosmos.Tests/Metadata/CosmosEntityTypeExtensionsTest.cs	Adds unit coverage for the new automatic-indexing builder behavior and convention APIs.
test/EFCore.Cosmos.Tests/Infrastructure/CosmosModelValidatorTest.cs	Updates/expands validator tests for composite, vector, full-text, include/exclude, and container consistency scenarios.
test/EFCore.Cosmos.FunctionalTests/TestUtilities/CosmosTestStore.cs	Plumbs automatic-indexing enabled/exceptions into test container creation.
test/EFCore.Cosmos.FunctionalTests/Scaffolding/CompiledModelCosmosTest.cs	Adds compiled-model test for index types and ensures design-time-only annotations are stripped.
test/EFCore.Cosmos.FunctionalTests/Scaffolding/Baselines/Cosmos_model_with_index_types/IndexedDataUnsafeAccessors.cs	New compiled-model baseline output for the added indexing test model.
test/EFCore.Cosmos.FunctionalTests/Scaffolding/Baselines/Cosmos_model_with_index_types/IndexedDataEntityType.cs	New compiled-model baseline output representing entity/index metadata.
test/EFCore.Cosmos.FunctionalTests/Scaffolding/Baselines/Cosmos_model_with_index_types/DbContextModelBuilder.cs	New compiled-model baseline output for runtime model builder.
test/EFCore.Cosmos.FunctionalTests/Scaffolding/Baselines/Cosmos_model_with_index_types/DbContextModel.cs	New compiled-model baseline output for runtime model initialization.
test/EFCore.Cosmos.FunctionalTests/Scaffolding/Baselines/Cosmos_model_with_index_types/DbContextAssemblyAttributes.cs	New compiled-model baseline assembly attribute for Cosmos runtime model.
test/EFCore.Cosmos.FunctionalTests/Query/Translations/VectorSearchTranslationsCosmosTest.cs	Adds a roundtrip test for vector index configuration through a complex collection.
test/EFCore.Cosmos.FunctionalTests/Query/Translations/FullTextSearchTranslationsCosmosTest.cs	Adds a roundtrip test for full-text index configuration through a complex collection.
test/EFCore.Cosmos.FunctionalTests/Query/Associations/ComplexProperties/ComplexPropertiesProjectionCosmosTest.cs	Adds TODO markers for known “select entire document” behavior (tracking an existing issue).
test/EFCore.Cosmos.FunctionalTests/Query/AdHocComplexTypeQueryCosmosTest.cs	Updates complex-type tests to reflect newly-supported index scenarios and adds discriminator-in-id coverage.
test/EFCore.Cosmos.FunctionalTests/ModelBuilding/CosmosModelBuilderGenericTest.cs	Updates expectations for owned-type index traversal (now rejected with a more specific message).
test/EFCore.Cosmos.FunctionalTests/CosmosSessionTokensTest.cs	Stabilizes session token assertions by separating create vs delete phases.
test/EFCore.Cosmos.FunctionalTests/CosmosApiConsistencyTest.cs	Registers the new fluent API builder type for API consistency checks.
src/EFCore.Cosmos/Storage/Internal/CosmosDatabaseCreator.cs	Flows automatic indexing enabled/exceptions from model into container-creation parameters.
src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs	Implements indexing-policy emission (included/excluded paths, composite indexes) and robust JSON path building from root (incl. complex/collection segments).
src/EFCore.Cosmos/Storage/Internal/ContainerProperties.cs	Extends container-creation parameter record with automatic-indexing enabled/exceptions.
src/EFCore.Cosmos/Query/Internal/CosmosShapedQueryCompilingExpressionVisitor.CosmosProjectionBindingRemovingExpressionVisitorBase.cs	Ensures property reads navigate into complex JSON objects when a scalar property is declared on a complex type.
src/EFCore.Cosmos/Properties/CosmosStrings.resx	Adds/updates Cosmos provider error messages for automatic-indexing consistency and owned-type index traversal.
src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs	Updates generated string accessors/summaries to match new/changed resources.
src/EFCore.Cosmos/Metadata/Internal/CosmosAnnotationNames.cs	Adds new annotation names for automatic-indexing enabled/exceptions.
src/EFCore.Cosmos/Metadata/Conventions/CosmosRuntimeModelConvention.cs	Strips automatic-indexing annotations from runtime model (design-time-only).
src/EFCore.Cosmos/Metadata/Builders/CosmosAutomaticIndexingBuilder.cs	Introduces fluent builder enabling `Except(...)` chaining for indexing-policy excluded paths.
src/EFCore.Cosmos/Infrastructure/Internal/CosmosModelValidator.cs	Adds container-level validation for automatic-indexing settings and rejects regular indexes declared on owned types.
src/EFCore.Cosmos/Extensions/CosmosIndexExtensions.cs	Fixes convention annotation setter return type for full-text index flag.
src/EFCore.Cosmos/Extensions/CosmosEntityTypeExtensions.cs	Adds extensions for getting/setting automatic-indexing enabled/exceptions (incl. convention APIs).
src/EFCore.Cosmos/Extensions/CosmosEntityTypeBuilderExtensions.cs	Adds `HasAutomaticIndexing(...)` fluent API and convention builder helpers.
src/EFCore.Cosmos/Extensions/CosmosComplexCollectionTypePropertyBuilderExtensions.cs	Enables full-text/vector configuration APIs for properties inside complex collections.
src/EFCore.Cosmos/EFCore.Cosmos.baseline.json	Updates API baseline for new public surface (automatic-indexing builder + new extension methods).
src/EFCore.Cosmos/Design/Internal/CosmosCSharpRuntimeAnnotationCodeGenerator.cs	Ensures automatic-indexing annotations aren’t emitted into the runtime model during design-time code generation.

Files not reviewed (1)

src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported

Copilot

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated 6 comments.

Files not reviewed (1)

src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported

Copilot

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated 3 comments.

Files not reviewed (1)

src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported

Add support for Vector and FullText indexes with properties in complex properties or collections Add support for keys spanning non-collection complex properties Fixes #38067 Fixes #17303

AndriySvyryd · 2026-06-05T19:32:17Z

+                    }
+                }
+            }
+


@roji @JoasE This adds support for keys defined across non-collection complex types. This is a targeted change which will probably need to be rewritten in the future to allow more general features like #34067 or #34567

@AndriySvyryd
IIUC; The generated structural type materializer expression contains a ValueBufferTryReadValue call for a complex type scalar property which does not have a ComplexPropertyBindingExpression value buffer. Before, any value retrieval of these were generated in CosmosShapedQueryCompilingExpressionVisitor using a ComplexPropertyBindingExpression.

I guess this is generated because complex type scalar properties can be part of indexes now, and the generated materializer will perform some sort of operation on these because of this.

Is that correct? If so this change does make sense to me, but knowing this it feels like the ComplexPropertyBindingExpression I chose wasn't the best solution direction, but since it will be rewritten in #34567 this LGTM.

If not I might have to debug this to get a better understanding.

BTW, Wouldn't the generated retrieval always be visited after the initialization of the MaterializationContext and thus _projectionBindings contain a ComplexPropertyBindingExpression pointing to a variable containing the jObject subdocument we are trying to find here using pathSegments? In that case,

jTokenExpression = _projectionBindings.First(x => x.Key is ComplexPropertyBindingExpression cpb && cpb.ComplexProperty == complexType.ComplexProperty).Value

Should work aswell

I guess this is generated because complex type scalar properties can be part of indexes now, and the generated materializer will perform some sort of operation on these because of this.

This is to enable the materializer to access the key values directly.

jTokenExpression = _projectionBindings.First(x => x.Key is ComplexPropertyBindingExpression cpb && cpb.ComplexProperty == complexType.ComplexProperty).Value

That doesn't work in this case, the binding looked up - _projectionBindings[ComplexPropertyBindingExpression], points to complexPropertyBindingExpression.JObjectParameter, which is the jObjectVariable created in CreateComplexPropertyAssignmentBlock. That variable is block-local. But the case this code handles is a property read directly on the complex type for identity resolution - that read is emitted in the outer entity shaper block, before the instance is even constructed. Referencing the inner block's jObjectVariable from there would produce an out-of-scope variable reference.

AndriySvyryd requested a review from Copilot June 4, 2026 06:51

Copilot started reviewing on behalf of AndriySvyryd June 4, 2026 06:51 View session

Copilot AI reviewed Jun 4, 2026

View reviewed changes

AndriySvyryd force-pushed the Issue38067 branch from 3226f61 to 3eaac3a Compare June 4, 2026 19:15

AndriySvyryd marked this pull request as ready for review June 4, 2026 19:16

AndriySvyryd requested a review from a team as a code owner June 4, 2026 19:16

Copilot AI review requested due to automatic review settings June 4, 2026 19:16

Copilot started reviewing on behalf of AndriySvyryd June 4, 2026 19:16 View session

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 4, 2026 19:33

Copilot started reviewing on behalf of AndriySvyryd June 4, 2026 19:33 View session

AndriySvyryd requested a review from cincuranet June 4, 2026 19:34

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs Outdated

Comment thread src/EFCore.Cosmos/Storage/Internal/CosmosClientWrapper.cs

Comment thread test/EFCore.Cosmos.Tests/Metadata/CosmosEntityTypeExtensionsTest.cs

AndriySvyryd mentioned this pull request Jun 4, 2026

Add Cosmos support for complex types #31253

Closed

3 tasks

Add support for JSON, composite, include and exclude Cosmos indexes

c8568f7

Add support for Vector and FullText indexes with properties in complex properties or collections Add support for keys spanning non-collection complex properties Fixes #38067 Fixes #17303

AndriySvyryd force-pushed the Issue38067 branch from 7a68ad7 to c8568f7 Compare June 5, 2026 05:11

AndriySvyryd commented Jun 5, 2026

View reviewed changes

Conversation

AndriySvyryd commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndriySvyryd Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

JoasE Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndriySvyryd Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AndriySvyryd commented Jun 4, 2026 •

edited

Loading

JoasE Jun 5, 2026 •

edited

Loading