Code first model for System.Text.Json

[casperOne]

I am currently publishing libraries which utilize the System.Text.Json libraries for parsing JSON content from HTTP endpoints.

Assuming I have a library Ofl.Twitch, I like to publish three libraries:

Ofl.Twitch.Abstractions ➡ Contains interface definitions, as well as data models that the interface definitions rely on.
Ofl.Twitch ➡ Contains the implementation of the interfaces in Ofl.Twitch.Abstractions; contains a NuGet package reference to Ofl.Twitch.Abstractions.
Ofl.Twitch.Extensions ➡ Contains helper functions that are not tied to direct implementations in Ofl.Twitch; contains a NuGet package reference to Ofl.Twitch.Abstractions.

I find that the issue I'm facing with System.Text.Json is the same as what I faced with JSON.Net; there is not an easy way to customize the serialization/deserialization of the data models outside of the assembly that they are housed in.

This means that we have to establish and distribute a reference to System.Text.Json along with our abstractions, which is not ideal.

This could be circumvented somewhat by separating the attributes into another assembly, but I don't think this is the right approach.

Ideally, the abstractions package would not have a reference to anything that contributes to implementation. Additionally, we would have an API that we could use in the implementation assembly which would be able to take a code-first approach to defining how properties on a model would behave, a la Entity Framework's code-first approach.

Presumably, much of this is already implemented internally in the JsonSerializer (and other contributing) class(es), it's just a matter of exposing it.

Granted, we could achieve this with a JsonConverter much in the same way that is recommended to handle polymorphism:

https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-converters-how-to#converter-samples-for-common-scenarios

IMO, this is a bit unwieldy, and a little too close to the metal, when in reality, we may need to just register a bunch of properties, hand tweaking how a few of them are serialized.

I suspect a mechanism, if exposed, would be exposed through the JsonSerializerOptions class, allowing us to set where the builder for a type is.

[scalablecory]

Can you provide a more specific example of what you're having trouble doing, and also your proposed solution? Are you thinking some way to annotate a class with a converter, or something else?

[casperOne]

@scalablecory Thanks for the response.

Can you provide a more specific example of what you're having trouble doing

I want a cleaner separation between my abstractions, which are in one assembly, and the implementation, which is in another.

Take for example, this definition of a Video object, specifically the Duration property, which requires me to define a JsonConverter:

https://github.com/OneFrameLink/Ofl.Twitch.Abstractions/blob/bdb092c0c0f5343079a0485d949eb1460ef8853e/src/Ofl.Twitch.Abstractions/Video.cs#L12

This is an assembly that has only contract definitions, no implementation.

I'd like to do something like the following in another assembly when deserializing the content:

// Create a model builder.
var mb = new ModelBuilder<Video>();

// Add properties on the Video object.
mb
    .AddProperty(m => m.Id)
    .AddProperty(m => m.Language)
    // Add other properties.
    // ....
    // There is an overload that allows use to specify the converter.
    .AddProperty(m => m.Duration, new DurationJsonConverter());

// Add the model builder to the serializer options.
var options = new JsonSerializerOptions();
options.ModelBuilders.Add(mb);

// The content.
string json = ...;

// Deserialize content.
Video video = JsonSerializer.Deserialize<Video>(json, options);

(Note: I'm not married to the names, or this exact model)

In the example above, the JsonSerializer would use the ModelBuilder to determine which properties to serialize and how to assign them.

Additionally, it would allow me to specify the attributes that would help control serialization on a specific property as we see fit closer to the call site of the serialization, and away from the model (for those where this is a concern).

The general idea is to be able to more finely control serialization/deserialization away from the models, and not to distribute the implementation details related to serialization with abstractions.

I'm aware I can currently get away with this using a JsonConverter<Video> instance, but the overhead of managing that is onerous, at best; JsonSerializer must be doing all of this work already.

The ask here is to expose a mechanism that allows us to specify the details of how to control serialization without adding attributes to our models, and in a way that doesn't always involve dropping down to a low-level construct such as JsonConverter<T>.

[CreepyGnome]

@casperOne While you should continue to always consider dependencies you are adding to any library, especially on a library you are choosing to call Abstractions why do you have a problem with your Abstractions library having a NuGet package dependency on System.Text.Json?

There is no hard-fast rule that required all Abstractions library to have zero dependencies. While it's nice when any library has no dependency you shouldn't force yourself to get to zero here.

The rule I like to follow for dependencies on any library is that one should attempt to ensure any dependencies you add would not be a burden on the intended users/clients of the library.

Then if it would be a burden then I would consider that I have code that doesn't belong in that library and should be refactored out to an existing one or even a new one that will be clear what it does and would clearly not be a burden on the users/clients of that new library.

For example Ofl.Twitch.Json or Ofl.Twitch.Models etc.

[casperOne]

@CreepyGnome All of your points are subject to individual projects, and I agree there is merit in that discussion, but that discussion is orthogonal to the original ask.

The big issue here is that it relies on annotation of a CLR type to provide fine grained-control of the serialization process.

If I do not have control of the CLR type that I want to serialize/deserialize and it already has annotations to determine serialization that are different from my requirements, then I have no choice but to generate a new CLR type, annotate that, and then in the process, allocate a throwaway instance of the type I really want to use and copy the values over.

Many frameworks that rely on type annotations have this issue (which I see as a failing), which is why I brought up the code-first model in Entity Framework; while you can solely use attributes for most things Entity Framework, that is a layer on top of the code-first model.

The serializer already does this internally (it must, and the other frameworks that rely on type annotations which I mentioned as failings must as well), what I'm asking for is to expose that so that I don't have to work around it in the manner I mentioned above.

[CreepyGnome]

@casperOne I didn't think it was orthogonal as I thought I was addressing your statement that appeared to me to be the core reason you wrote the issue. I apologize if that is not the case.

This means that we have to establish and distribute a reference to System.Text.Json along with our abstractions, which is not ideal.

I propose the questions basically of:

• Why is it not ideal?
• What is the burden you are placing on those libraries that depend on your Abstractions?
• Is the burden unreasonable?

Then possible solutions that don't require making changes System.Text.Json:

If it is not then everything is good. If it is not you could consider refactoring out the code that has this dependency into a library that already has a dependency on this, or into a new one.

I didn't see any mention of the CLR type in question? For clarity, a CLR type to me has always been a built-in type that only exists in the CLR natively itself, however, based on your usage I assume you mean any type that may be defined in the BCL or other supporting libraries. If I am wrong please let me know.

As you mentioned you solved this by making your similar type. That seems valid as in this situation it doesn't appear to be a violation of DRY. As the type clearly has a different purpose than you need it to be as it wasn't designed to work how you want at least with serialization. Therefore being WET here should be acceptable. I don't know the specific type in question, I assume it outside of your OFL set of libraries. I would assume you could just stop using it once you created your version that works how you need it to, so it wouldn't be a throw-away.

[casperOne]

@CreepyGnome I appreciate your attempts to help here, but I'm not currently facing a blocking issue; I'm aware of all the ways that this can be solved in order to achieve what I'm looking for. The point of the issue is that many of them are onerous, and can be circumvented (for many, I believe) by exposing what is already being done, internally.

To answer your questions:

• Why is it not ideal?

It is not ideal for two reasons:

• I am shipping implementation details along with this assembly, which I would like to avoid for an abstractions-only assembly
• Because attributes on the type are used to drive serialization, if this is going to be used outside of this assembly (and that's very possible), I am imposing an undue burden of having to do the copy to a separate type solution which in many cases may be an unnecessary piece of overengineering, as well as eliminate potential performance gains when using a library that explicitly tries to reduce allocations in order to improve performance

• What is the burden you are placing on those libraries that depend on your Abstractions?

Answered above in the second point

• Is the burden unreasonable?

In my opinion, yes, it is, hence the filing of the issue.

I agree there are many scenarios where people inappropriately double-dip types. However, there are many valid scenarios where people double-dip types as well (perhaps they want to maintain fidelity against the original source, or other reasons).

[layomia]

This looks like a duplicate of #30694. This sort of extension point to the serializer is not on our roadmap for now. It may be possible to build this on top of our existing converter model in another library. If you want to prototype this in https://github.com/dotnet/corefxlab, we could collaborate with you.