UniverseQuery 3.5.0-preview.2

What's Changed

• security: Harden query builder, exception handling, and storage permissions by @norarrsgd in #49
• security: Harden query builder limits and provisioning by @norarrsgd in #50
• nuget: upgrade dependencies by @norarrsgd in #51
• fix: Potential fix for code scanning alert no. 4: Workflow does not contain permissions by @norarrsgd in #52
• feature: add optional document cache by @norarrsgd in #53
• feature: add enhanced bulk and atomic operations by @norarrsgd in #54
• release: harden UniverseQuery 3.5.0-preview.2 by @norarrsgd in #55

Full Changelog: universe-3.4.0...universe-3.5.0-preview.2

Universe v3.4.0

What's Changed

• feature: Native query builder naming policy from serializer by @norarrsgd in #47
• feature: type-based projection Select() for Orbit by @norarrsgd in #48

Full Changelog: universe-3.3.0...universe-3.4.0

Universe v3.3.0

What's Changed

• feature: Add fluent query builder (Orbit) API by @norarrsgd in #46

Full Changelog: universe-3.2.0...universe-3.3.0

Universe v3.2.0

What's Changed

• perf: Partition QueryTuner by QueryType and compile regex by @kuromukira in #44
• fix: Allow custom storage paths and add Azure-aware defaults for SQLite persistence by @kuromukira in #45

Full Changelog: universe-3.1.1...universe-3.2.0

Universe v3.1.1

• Changed from c.<column name> to c["<column name>"]
• Added protections for SQL Injections

Universe v3.1.0

(New) Generate CosmosDb-Query without execution

This is particularly helpful when trying to test your query logic first before fully adding to your applications.

IGalaxy<MyObject> myGalaxy = new MyGalaxy(cosmosClient, "MyDatabase", "People", ["/partitionKey"]);

IReadOnlyList<Cluster> complexClusters = [
    new Cluster("Age", ClusterOperator.GreaterThanOrEqual, 18),
    new Cluster("Status", ClusterOperator.Equals, "Active"),
    new Cluster("Department", ClusterOperator.Eq, "Engineering")
];

ColumnOptions selectedColumns = new ColumnOptions(["Id", "Name", "Department", "Salary"]);
IReadOnlyList<Sorting.Option> complexSorting = [
    new Sorting.Option("Department", Sorting.Direction.Ascending),
    new Sorting.Option("Salary", Sorting.Direction.Descending)
];

Gravity complexQuery = myGalaxy.GenerateQuery(
    clusters: complexClusters,
    columnOptions: selectedColumns,
    sorting: complexSorting,
    group: null
);

Output would be (Gravity object):

// Expected Output: "SELECT c.Id, c.Name, c.Department, c.Salary FROM c WHERE c.Age >= @param0 AND c.Status = @param1 AND c.Department IN (@param2, @param3, @param4) ORDER BY c.Department ASC, c.Salary DESC"
// Parameters: { "@param0": 18, "@param1": "Active", "@param2": "Engineering" }

Universe v3.0.5

Breaking Changes

public record MyObject : CosmicEntity
{
    [PartitionKey(2)]
    public string Code { get; set; }

    public string Name { get; set; }

    [PartitionKey(1)]
    public string Category { get; set; }
}

_ = services.AddScoped<IGalaxy<MyModel>, MyRepository>(service => new MyRepository(
    client: service.GetRequiredService<CosmosClient>(),
    database: "database-name",
    container: "container-name",
    partitionKey: typeof(MyObject).BuildPartitionKey()
));

Partition Keys are now generated as lowerCamelCase by default (.BuildPartitionKey()), regardless of Serializer Option configured in the client.

You can bypass this by adding a custom KeyName parameter to the attribute.

[PartitionKey(1, "KeyName")]

The UniverseSerializer now accepts a Naming Policy as parameter.

CosmosClient cosmosClient = new(
    CosmosDbUri,
    CosmosDbPrimaryKey,
    clientOptions: new()
    {
        Serializer = new UniverseSerializer(JsonNamingPolicy.CamelCase),
        AllowBulkExecution = true
    }
);

(Preview) Added support for Full-Text Search

Please note that this feature is unstable at best now and requires careful integration. You might encounter performance issues when using this feature.

Setup

Initialize your Cosmos DB client and repository:

MyRepo galaxy = new(
    client: cosmosClient,
    database: "test-database",
    container: "documents-container",
    partitionKey: typeof(MyObject).BuildPartitionKey()
);

Basic Full-Text Contains Search

// Input
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("title", "machine learning", Operator: Q.Operator.FTContains)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "description"], Top: 10)
);

Generated SQL:

SELECT TOP 10 c.id, c.title, c.description 
FROM c 
WHERE (FullTextContains(c.title, @title))

Full-Text Search with Relevance Scoring

// Input
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("content", new[] { "artificial", "intelligence" }, Operator: Q.Operator.FTScore)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "content"], Top: 5)
);

Generated SQL:

SELECT TOP 5 c.id, c.title, c.content 
FROM c 
ORDER BY RANK FullTextScore(c.content, @content)

Multi-Field Full-Text Search with RRF

// Input
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("title", new[] { "machine", "learning" }, Operator: Q.Operator.FTScore),
            new("description", new[] { "neural", "networks" }, Operator: Q.Operator.FTScore)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "description"], Top: 10)
);

Generated SQL:

SELECT TOP 10 c.id, c.title, c.description 
FROM c 
ORDER BY RANK RRF(FullTextScore(c.title, @title), FullTextScore(c.description, @description))

Weighted Multi-Field Full-Text Search

// Input
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("title", new[] { "machine", "learning" }, Operator: Q.Operator.FTScore),
            new("description", new[] { "deep", "learning" }, Operator: Q.Operator.FTScore)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "description"], Top: 10),
    sorting: [
        new(Column: "[0.8, 0.2]", Direction: Sorting.Direction.WEIGHTED)
    ]
);

Generated SQL:

SELECT TOP 10 c.id, c.title, c.description 
FROM c 
ORDER BY RANK RRF(FullTextScore(c.title, @title), FullTextScore(c.description, @description), [0.8, 0.2])

Full-Text Contains All Terms

// Input - All specified terms must be present
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("content", new[] { "machine", "learning", "algorithms" }, Operator: Q.Operator.FTContainsAll)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "content"], Top: 10)
);

Generated SQL:

SELECT TOP 10 c.id, c.title, c.content 
FROM c 
WHERE (FullTextContainsAll(c.content, @content))

Full-Text Contains Any Terms

// Input - Any of the specified terms can be present
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("tags", new[] { "AI", "ML", "DL", "NLP" }, Operator: Q.Operator.FTContainsAny)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "tags"], Top: 15)
);

Generated SQL:

SELECT TOP 15 c.id, c.title, c.tags 
FROM c 
WHERE (FullTextContainsAny(c.tags, @tags))

Negated Full-Text Search

// Input - Exclude documents containing specific terms
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("content", "deprecated", Operator: Q.Operator.NotFTContains),
            new("title", new[] { "obsolete", "legacy" }, Operator: Q.Operator.NotFTContainsAny)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "content"], Top: 10)
);

Generated SQL:

SELECT TOP 10 c.id, c.title, c.content 
FROM c 
WHERE (NOT FullTextContains(c.content, @content) AND NOT FullTextContainsAny(c.title, @title))

Hybrid Search: Full-Text + Traditional Filters

// Input - Combine full-text search with regular filters
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        // Full-text search cluster
        new(Catalysts: [
            new("content", new[] { "machine", "learning" }, Operator: Q.Operator.FTScore)
        ]),
        // Traditional filter cluster
        new(Where: Q.Where.And, Catalysts: [
            new("category", "Technology", Operator: Q.Operator.Eq),
            new("publishDate", DateTime.Now.AddDays(-30), Operator: Q.Operator.Gte),
            new("status", "Published", Operator: Q.Operator.Eq)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "category", "publishDate"], Top: 10)
);

Generated SQL:

SELECT TOP 10 c.id, c.title, c.category, c.publishDate 
FROM c 
WHERE (c.category = @category AND c.publishDate >= @publishDate AND c.status = @status)
ORDER BY RANK FullTextScore(c.content, @content)

Complex Multi-Cluster Full-Text Search

// Input - Multiple search clusters with different criteria
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        // Primary content search
        new(Catalysts: [
            new("title", new[] { "artificial", "intelligence" }, Operator: Q.Operator.FTScore)
        ]),
        // Secondary content search
        new(Where: Q.Where.OR, Catalysts: [
            new("description", new[] { "machine", "learning" }, Operator: Q.Operator.FTContainsAny),
            new("tags", new[] { "AI", "ML" }, Operator: Q.Operator.FTContainsAny)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "description", "tags"], Top: 20)
);

Generated SQL:

SELECT TOP 20 c.id, c.title, c.description, c.tags 
FROM c 
WHERE (FullTextContainsAny(c.description, @description) AND FullTextContainsAny(c.tags, @tags))
ORDER BY RANK FullTextScore(c.title, @title)

Full-Text Search with Aggregation

// Input - Group results by category and count
(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("content", new[] { "technology", "innovation" }, Operator: Q.Operator.FTContainsAny)
        ])
    ],
    columnOptions: new(
        Names: ["category"],
        Aggregates: [
            new(Aggregate: Q.Aggregate.Count)
        ]
    )
);

Generated SQL:

SELECT c.category, COUNT(1) AS CountAggregate 
FROM c 
WHERE (FullTextContainsAny(c.content, @content))
GROUP BY c.category

Hybrid Search: Full-Text + Vector Search

// Input - Combine full-text scoring with vector similarity
float[] queryVector = [0.1f, 0.8f, 0.3f, 0.9f, 0.2f];

(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("contentEmbedding", queryVector, Operator: Q.Operator.VectorDistance),
            new("title", new[] { "machine", "learning" }, Operator: Q.Operator.FTScore),
            new("description", new[] { "artificial", "intelligence" }, Operator: Q.Operator.FTScore)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "description"], Top: 10)
);

Generated SQL:

SELECT TOP 10 c.id, c.title, c.description, VectorDistance(c.contentEmbedding, @contentEmbedding) AS contentEmbeddingScore
FROM c 
ORDER BY RANK RRF(VectorDistance(c.contentEmbedding, @contentEmbedding), FullTextScore(c.title, @title), FullTextScore(c.description, @description))

Weighted Hybrid Search: Full-Text + Vector Search

// Input - Combine full-text and vector search with custom weights
float[] queryVector = [0.1f, 0.8f, 0.3f, 0.9f, 0.2f];

(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new("contentEmbedding", queryVector, Operator: Q.Operator.VectorDistance),
            new("title", new[] { "machine", "learning" }, Operator: Q.Operator.FTScore)
        ])
    ],
    columnOptions: new(Names: ["id", "title", "content"], Top: 10),
    sorting: [
        new(Column: "[0.6, 0.4]", Direction: Sorting.Direction.WEIGHTED) // 60% vector, 40% text
    ]
);

Generated SQL:

SELECT TOP 10 c.id, c.title, c.content, VectorDistance(c.contentEmbedding, @contentEmbedding) AS con...

Universe v3.0.4

Breaking Changes

Partition Key Changes

The Old approach

public override string PartitionKey => Code;

Is now deprecated and can no longer be used. This is to encourage the adoption of the PartitionKeyAttribute.

---

(New) Added support for VectorDistance() of the CosmosDB NoSQL API

Setup

Make sure to initialize your Cosmos DB client and create a vector repository:

MyRepoVector vectorGalaxy = new(
    client: cosmosClient,
    database: "test-database",
    container: "vector-container",
    partitionKey: typeof(MyObjectVector).BuildPartitionKey()
);

Single Vector Search

// Input
float[] queryVector = [0.1f, 0.8f, 0.3f, 0.9f, 0.2f];

(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new(nameof(MyObjectWithVector.DescriptionEmbedding), queryVector, Operator: Q.Operator.VectorDistance)
        ])
    ],
    columnOptions: new(Names: ["id", "name"], Top: 5)
);

Generated SQL:

SELECT TOP 5 c.id, c.name, VectorDistance(c.DescriptionEmbedding, @DescriptionEmbedding) AS DescriptionEmbeddingScore 
FROM c 
ORDER BY VectorDistance(c.DescriptionEmbedding, @DescriptionEmbedding)

Multi-Vector Search with RRF (Reciprocal Rank Fusion)

// Input
float[] titleVector = [0.2f, 0.9f, 0.1f, 0.8f];
float[] descVector = [0.3f, 0.7f, 0.2f, 0.9f];

(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new(nameof(MyObjectWithVector.TitleEmbedding), titleVector, Operator: Q.Operator.VectorDistance),
            new(nameof(MyObjectWithVector.DescriptionEmbedding), descVector, Operator: Q.Operator.VectorDistance)
        ])
    ],
    columnOptions: new(Names: ["id", "name"], Top: 3)
);

Generated SQL:

SELECT TOP 3 c.id, c.name, 
       VectorDistance(c.TitleEmbedding, @TitleEmbedding) AS TitleEmbeddingScore,
       VectorDistance(c.DescriptionEmbedding, @DescriptionEmbedding) AS DescriptionEmbeddingScore
FROM c 
ORDER BY RANK RRF(VectorDistance(c.TitleEmbedding, @TitleEmbedding), VectorDistance(c.DescriptionEmbedding, @DescriptionEmbedding))

Multi-Vector Search with Weighted RRF

// Input
float[] titleVector = [0.2f, 0.9f, 0.1f, 0.8f];
float[] descVector = [0.3f, 0.7f, 0.2f, 0.9f];

(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        new(Catalysts: [
            new(nameof(MyObjectWithVector.TitleEmbedding), titleVector, Operator: Q.Operator.VectorDistance),
            new(nameof(MyObjectWithVector.DescriptionEmbedding), descVector, Operator: Q.Operator.VectorDistance)
        ])
    ],
    columnOptions: new(Names: ["id", "name"], Top: 3),
    sorting: [
        new(Column: "[1, 2]", Direction: Sorting.Direction.WEIGHTED)
    ]
);

Generated SQL:

SELECT TOP 3 c.id, c.name, 
       VectorDistance(c.TitleEmbedding, @TitleEmbedding) AS TitleEmbeddingScore,
       VectorDistance(c.DescriptionEmbedding, @DescriptionEmbedding) AS DescriptionEmbeddingScore
FROM c 
ORDER BY RANK RRF(VectorDistance(c.TitleEmbedding, @TitleEmbedding), VectorDistance(c.DescriptionEmbedding, @DescriptionEmbedding), [1, 2])

Hybrid Search (Vector + Filters)

// Input  
float[] queryVector = [0.4f, 0.6f, 0.8f, 0.2f];

(Gravity, IList<T>) results = await galaxy.List(
    clusters: [
        // Vector similarity
        new(Catalysts: [
            new(nameof(MyObjectWithVector.DescriptionEmbedding), queryVector, Operator: Q.Operator.VectorDistance)
        ]),
        // Traditional filters
        new(Where: Q.Where.And, Catalysts: [
            new(nameof(MyObjectWithVector.Category), "Electronics", Operator: Q.Operator.Eq),
            new(nameof(MyObjectWithVector.Price), 1000.0, Operator: Q.Operator.Lt)
        ])
    ],
    columnOptions: new(Names: ["id", "name", "price"], Top: 5)
);

Generated SQL:

SELECT TOP 5 c.id, c.name, c.price, VectorDistance(c.DescriptionEmbedding, @DescriptionEmbedding) AS DescriptionEmbeddingScore
FROM c 
WHERE (c.Category = @Category AND c.Price < @Price)
ORDER BY VectorDistance(c.DescriptionEmbedding, @DescriptionEmbedding)

Universe v3.0.3

Breaking Changes

Partition Key Changes

The most significant breaking change is in how partition keys are handled:

Old approach

public override string PartitionKey => Code;

New approach (required)

// If planning to use single partitions
[PartitionKey]
public string Category { get; set; }

public string Code { get; set; }

// If planning to use multi-level partitions
[PartitionKey(1)]
public string Category { get; set; }

[PartitionKey(2)]
public string Code { get; set; }

Constructor Parameter Changes

For a "Repository" class, the constructor signature has been changed:

Old signature

Galaxy(CosmosClient client, string database, string container, string partitionKey, bool recordQueries = false)

New signature (required)

Galaxy(CosmosClient client, string database, string container, IReadOnlyList<string> partitionKey, bool recordQueries = false)

Note that you might need you adjust dependency injections for "Repository" classes:

MyRepo galaxy = new(
    client: cosmosClient,
    database: "test-database",
    container: "my-container",
    partitionKey: typeof(MyObject).BuildPartitionKey() // Using extension method
);

Universe v3.0.2

#28