LINQ custom serializer: query translated incorrectly when passing variable to the nested "where" clause

[Rasmus715]

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

The problem persists with the actual & preview versions available at the current moment

Describe the bug
When a custom LINQ serializer configured, database request with the nested .Where() clause with the variable passed inside is translated incorrectly, resulting the data loss.

To Reproduce

1. Configure the custom CosmosDb LINQ Serializer
2. Populate the database with a object, containing an array
3. Query the object and the underlying one using a defined variable in the application and .Where() clause

Expected behavior
The data is returned successfully

Actual behavior
The root object is returned successfully, however the nested entities are lost

Environment summary
Affected SDK versions: 3.39.0, 3.39.1, 3.40.0.preview.0, 3.40.0.preview.1
OS Version: Windows 11 23H2

Additional context
Minimal repository with reproduction: https://drive.google.com/file/d/1B09-t8XJ8Bl7-u66y8IfgVEV1mTWT4nU/view?usp=sharing

Generated query using custom LINQ serializer (inherited from CosmosLinqSerializer):
{{"query":"SELECT TOP 1 VALUE {\"id\": root[\"id\"], \"nestedEntities\": v0} FROM root JOIN (SELECT VALUE ARRAY(SELECT VALUE c0 FROM root JOIN c0 IN root[\"nestedEntities\"] WHERE (c0[\"fieldToSearchBy\"] = {}[\"fieldToSearchByVariable\"]))) AS v0"}}

Generated query using custom CosmosDbSerializer (inherited from CosmosSerializer)
{{"query":"SELECT TOP 1 VALUE {\"Id\": root[\"Id\"], \"NestedEntities\": v0} FROM root JOIN (SELECT VALUE ARRAY(SELECT VALUE c0 FROM root JOIN c0 IN root[\"NestedEntities\"] WHERE (c0[\"FieldToSearchBy\"] = {\"fieldToSearchByVariable\": 8}[\"fieldToSearchByVariable\"]))) AS v0"}}

The Linq serializer is copied from the official sample

[Trimatix]

I've also just run into this issue, it was tricky to track down!

For now we have rolled back to the previous package version, but being able to use custom serializer options was a game changer for us in this version. Currently we work around all of our custom serializer options by hand - for example casting enums to objects to allow for enum name comparison instead of value

[technight]

👍 Would also really like to see this issue fixed ASAP

[Maya-Painter]

@Rasmus715 As a mitigation could you try with FieldToSearchByVariable as a const int?

This issue seems to be stemming from how Azure.Core.Serialization.JsonObjectSerializer translates the object. When serializing the object with the fieldToSearchByVariable property in the ToStream() function of the custom serializer, it serializes it as an empty object. The following function returns a payload of {}.

Unfortunately, we have no control over the inner workings of the JsonObjectSerializer. I understand it's not ideal, but labeling the variable to filter by as a const seems to circumvent this scenario.

Another possibility is to do some custom handling of these cases in your ToStream() custom serializer implementation.

[Rasmus715]

@Rasmus715 As a mitigation could you try with FieldToSearchByVariable as a const int?

This issue seems to be stemming from how Azure.Core.Serialization.JsonObjectSerializer translates the object. When serializing the object with the fieldToSearchByVariable property in the ToStream() function of the custom serializer, it serializes it as an empty object. The following function returns a payload of {}.

Unfortunately, we have no control over the inner workings of the JsonObjectSerializer. I understand it's not ideal, but labeling the variable to filter by as a const seems to circumvent this scenario.

Another possibility is to do some custom handling of these cases in your ToStream() custom serializer implementation.

Moving from declaring a variable to a constant solved the issue. However, that won't work in my case since these values are coming to query from a method parameters. Also, I've tested the same scenario passing not a primitive type but a complex object and then trying to access a field of it but ended up with the same result

Could you please provide the implementation of ToStream() method, if it's not possible to handle this case internally in the library?