Query: Adds support for System.Text LINQ Custom Serializer

[Maya-Painter]

Description

This change allows users to use custom serializers for the translation of LINQ queries.

To use their own custom serializer for LINQ translations users must implement CosmosLinqSerializer and set as a custom serializer on the CosmosClient. (Sample in CosmosLinqSerializer.cs)

This allows for the following scenarios:

Property name customization via JsonPropertyName

Given an object:

public record MyItem
{
      [JsonPropertyName("id")]
      public string Id { get; set; }

      [JsonPropertyName("project_id")]
      public string ProjectId { get; set; }
}

With the following query:
query = testContainer.GetItemLinqQueryable<MyItem>().Where(i => i == new MyItem() { ProjectId = "test", Id = "id1"});

This will be translated into the following SQL query, taking into account the property attributes on the MyItem class
{"query":"SELECT VALUE root FROM root WHERE (root = {\"id\": \"id1\", \"project_id\": \"test\"})"}

Enum conversion via StringEnumConverter

This change also means that the System.Text.Json.Serialization.StringEnumConverter() will be honored when translating enums in LINQ queries. If JsonStringEnumConverter() is specified as a converter on the custom serializer, enums do not need to be decorated with [JsonConverter(typeof(JsonStringEnumConverter))].

Type of change

• Bug fix (non-breaking change which fixes an issue)
• This change requires a documentation update

Closing issues

closes #1936,
closes #3207,
closes #2685,
closes #3250,
closes #3697,
closes #1253,
closes #2386,
closes #3280,
closes #3491,
closes #484

[adityasa]

[onionhammer]

When is this getting released in a non-preview version?

[HSven1611]

Is this included in microsoft.azure.cosmos\3.39.0-preview.1?

It seems I'm missing something as I'm not able to make it use the [JsonPropertyName] Attribute for creating a linq query. It seems to respect only the Newtonsoft Attribute.

I made it use the STJ Attributes in storing and loading data by providing my own custom CosmosSerializer. I cannot find the respective attribute in the version mentioned above though.

I've tried to follow this code found in commit 0bd4a26 but I'm not able to find the property LinqSerializerType in CosmosLinqSerializerOptions. It only had PropertyNamingPolicy exposed.

My guess is, it is simply not published ¯\(ツ)/¯

[Maya-Painter]

@onionhammer It will be in the next public release (see #4323). We do not have an ETA for this release yet.

[Maya-Painter]

@HSven1611 It is included in 3.39.0-preview.1.

If you're only using CosmosSerializer it will not apply to LINQ translations. You will need to use CosmosLinqSerializer. There is an example STJ serializer here. Documentation will be updated to make this clearer once the feature is public.

[machie27]

In version 3.39.0-preview.1

I do see 2 properties of the Internal linqSerializerOptions:

• CosmosLinqSerializerOptions and CustomCosmosLinqSerializer. However the PropertyNamingPolicy is not respected.

A notable distinction exists between utilizing the default serializer option JsonNamingPolicy.CamelCase and CosmosPropertyNamingPolicy.CamelCase.

It would be beneficial for these to be unified to behave equally or at least recognized as equivalent.

This is also the case for the JsonNamingPolicy in the JsonStringEnumConverter, as of now this is just disregarded by the CosmosLinqSerializer

Is this possible? Or what is the difference, this would help us make the options more generic. Because this way it still requires a very specific JsonSerializerOptions just for the CosmosLinqSerializer

The options I would like to reuse for my CosmosLinqSerializer are:

public static JsonSerializerOptions DefaultOptions => new()
{
    PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
    DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
    Converters =
    {
        new JsonStringEnumConverter(JsonNamingPolicy.CamelCase, allowIntegerValues: false)
    }
};

Workaround for now could be something like:

public override string SerializeMemberName(MemberInfo memberInfo)
{
    var jsonPropertyNameAttribute = memberInfo.GetCustomAttribute<JsonPropertyNameAttribute>(true);

    string memberName = !string.IsNullOrEmpty(jsonPropertyNameAttribute?.Name)
        ? jsonPropertyNameAttribute.Name
        : memberInfo.Name;

    return JsonNamingPolicy.CamelCase.ConvertName(memberName);
}

Nonetheless, addressing the PropertyNamingPolicy directly would present a cleaner, more straightforward solution.

But I understand if this falls under a different request

[onionhammer]

@onionhammer It will be in the next public release (see #4323). We do not have an ETA for this release yet.

Any update on this? Ballpark? This summer? This year?

[Maya-Painter]

@onionhammer we are planning on releasing this within a week.

[sreejith-ms]

@Maya-Painter Could you please provide a basic sample for system text json serializer for linq and default serializer?

I tried with 3.39.1 version.

public class FooEntity
{
    [JsonPropertyName("id")]
    public string Id { get; set; }

    [JsonPropertyName("props")]
    public List<string> Props { get; set; }
}


public class Program
{

    var cosmosClientOptions = new CosmosClientOptions()
    {
        Serializer = new SystemTextJsonLinqSerializer(new JsonSerializerOptions()),    
    };

    var client = new CosmosClient("connection string", cosmosClientOptions);
    var container = cosmosClient.GetContainer("DatabaseId", "FooEntity");

    var query = container.GetItemLinqQueryable<FooEntity>()
        .Where(x => x.Id == "test")
        .Select(x => x.Props)
        //.Select(x => new { x.Props })
        .Take(1);

    var iterator = query.ToFeedIterator();
    var resp = await iterator.ReadNextAsync();
    var props = resp.FirstOrDefault();

    Console.WriteLine(props);
}

I'm using this LinqSerializer:

azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.EmulatorTests/Linq/LinqTestsCommon.cs

Line 817 in dee9aba

class SystemTextJsonLinqSerializer : CosmosLinqSerializer

I'm not able to figure out how to configure both LinqSerializer and JsonSerializer.
When I executed the above snippet, the generated linq query respects JsonPropertyName. However the Json deserialize FromStream<T> fails with exception:
System.Text.Json.JsonException: 'The JSON value could not be converted to System.Collections.Generic.List`1[System.String]. Path: $[0] | LineNumber: 0 | BytePositionInLine: 70.'

[onionhammer]

@sreejith-ms the linq serializer inherits from the normal JSON serializer, so it does both functions

[sreejith-ms]

@onionhammer Thanks for the clarification.
Do you know why linq Select operator is breaking with deserialize error?

[sreejith-ms]

The Select operator issue is resolved, it's a data issue. Now working fine.