Cosmos: API to materiailze entities from raw JSON data

[JoasE]

What problem are you trying to solve?

When working with Azure Cosmos DB, you might interact with cosmos db directly (outside of EF) but still want to deserialize the document to an entity.
For example, when using the change feed processor:

var client = context.Database.GetCosmosClient();
var container = client.GetContainer(
    context.Database.GetCosmosDatabaseId(),
    context.Model.GetEntityTypes().Single().GetContainer());

var processor = container.GetChangeFeedProcessorBuilder("processor", async (IReadOnlyCollection<JObject> changes, CancellationToken ctx) =>
{
    foreach (var change in changes)
    {
        // Deserialize to entity?
    }
}).WithInMemoryLeaseContainer().Build();

Describe the solution you'd like

Some way to deserialize documents to entity types

[roji]

@JoasE I'm not too sure what to think of this... For change feed, exposing an actual API for that (#17319) seems like the right thing to do, rather then exposing an intermediate low-level API to accept arbitrary JSON and deserialize. What other scenarios do you see this being useful for?

[JoasE]

@roji I understand your reluctance!
Implementing a change feed support will definitely make this api less useful, but I don't think it will be completely obsolete:

• Besides a ChangeFeedProcessor from the SDK, you could use an Azure Functions Cosmos DB trigger. When receiving documents from the trigger (or input binding), you wouldn't be able to leverage EF's API to interact with the change feed. You would, however, be able to retrieve a JObject, JsonObject (and I think also string or Stream) from the trigger. You would need to be able to leverage EF's internal materializer to deserialize your entity from there on.
• Applications might implement a 2 tier storage solution, where older or less used data is stored outside of cosmos db. For example in a Storage Account (Blob or Table). You might not be able to use EF for interacting with your 2nd storage solution, but can retrieve the document's content and attempt to deserialize it with this api.

I have personally done both before. Besides that I don't think it would have many use cases, and I understand if you wouldn't find these reasons enough to justify this api in EF. The app could implement some form of deserialization logic itself by using the DbContext's Model, but it would be basically rewriting the materialization logic from EF.

[roji]

You would, however, be able to retrieve a JObject, JsonObject (and I think also string or Stream) from the trigger. You would need to be able to leverage EF's internal materializer to deserialize your entity from there on.

FYI we plan to at some point move away from deserializing from JObject/JsonObject or similar - having EF first deseriailze to a JObject and then materialize the user's .NET objects is quite inefficient; we've already done this change in relational, where EF directly deserializes via Utf8JsonReader. So I definitely wouldn't introduce an EF API which deserializes from a JObject (though if it accepts a string/Stream, that might make more sense).

Otherwise, the scenarios you describe make some sense, but seem quite edge-casey. I'd want to first see users requesting such an API (that hasn't ever happened so far, AFAIK), and implement it as a response to such requests.

So I'll go ahead and put this on the backlog for now, until we see some traction on it. On the other hand, the change feed API makes more sense to me as a 1st-class feature.

[JoasE]

@roji Just to be clear: Does that mean you currently won't consider a community pr for this? (no worries if so)

[roji]

@JoasE yeah, it does. I'm sorry, but this is a new feature proposal, and both designing it and reviewing it take time (not to mention that we'd take ownership of it if there's any trouble later), and I just don't see the justification for it at the moment. Hopefully that's understandable - but there's certainly a lot of other stuff to do on the Cosmos provider (I really will try to be more reactive to your PRs).

[JoasE]

@roji No worries at all! No need to apologize, I understand your reasoning.

I will leave this quick and dirty workaround here for anyone who runs into the same issue. It does use 1 additional round trip and RU per document and uses reflection at runtime (which can be refactored away), but it works.

using System.Linq.Expressions;
using System.Text.Json.Nodes;
using Microsoft.Azure.Functions.Worker;
using Microsoft.EntityFrameworkCore;

// @TODO: Refactor away reflection
public class Function(Context context)
{
    [Function(nameof(Function))]
    public async Task Run([CosmosDBTrigger(
        databaseName: "Db",
        containerName: "Context",
        Connection = "CosmosConnectionString",
        LeaseContainerName = "leases",
        CreateLeaseContainerIfNotExists = true)] IReadOnlyList<JsonObject> input)
    {
        if (input == null)
        {
            return;
        }

        var documentRoots = context.Model.GetEntityTypes().Where(x => !x.IsOwned()).ToList();
        foreach (var inputItem in input)
        {
            foreach (var documentRoot in documentRoots)
            {
                var discriminatorProperty = documentRoot.FindDiscriminatorProperty();
                if (discriminatorProperty == null)
                {
                    throw new InvalidOperationException("Entity must have a discriminator property.");
                }

                var discriminatorValue = inputItem[discriminatorProperty.GetJsonPropertyName()]?.GetValue<string>();
                if (discriminatorValue == documentRoot.GetDiscriminatorValue()?.ToString())
                {
                    var key = documentRoot.FindPrimaryKey();
                    var partitionKeys = documentRoot.GetPartitionKeyProperties();
                    if (key == null || !partitionKeys.Any())
                    {
                        throw new InvalidOperationException("Entity must have a primary key and partition key defined.");
                    }

                    var keyValues = key.Properties
                        .ToDictionary(p => p, p => typeof(JsonNode).GetMethod(nameof(JsonNode.GetValue))!.MakeGenericMethod(p.ClrType).Invoke(inputItem[p.GetJsonPropertyName()], null));
                    var partitionKeyValues = partitionKeys
                        .ToDictionary(p => p, p => typeof(JsonNode).GetMethod(nameof(JsonNode.GetValue))!.MakeGenericMethod(p.ClrType).Invoke(inputItem[p.GetJsonPropertyName()], null));

                    // Entity entity => 
                    var parameter = Expression.Parameter(typeof(object), "entity");
                    var castedParameter = Expression.Convert(parameter, documentRoot.ClrType);

                    var filter = keyValues.Concat(partitionKeyValues)
                        .Select(kv =>
                        {
                            // EF.Property<PropertyType>(entity, "propertyName") == (PropertyType)kv.Value
                            var propertyAccess = Expression.Call(
                                typeof(EF),
                                nameof(EF.Property),
                                [kv.Key.ClrType],
                                castedParameter,
                                Expression.Constant(kv.Key.Name));
                            var constant = Expression.Constant(kv.Value);
                            var cast = Expression.Convert(constant, kv.Key.ClrType);
                            var equality = Expression.Equal(propertyAccess, cast);
                            return equality;
                        })
                        .Aggregate(Expression.AndAlso); // &&
                    var whereExpression = Expression.Lambda<Func<object, bool>>(filter, parameter);

                    var set = (IQueryable<object>)typeof(DbContext).GetMethod("Set", [])!.MakeGenericMethod(documentRoot.ClrType).Invoke(context, null)!;
                    var entity = await set.Where(whereExpression).FirstOrDefaultAsync();
                }
            }
        }
    }
}