Reopen 7439 - ModelState json serialization should be camel cased

[antonioortizpola]

Describe the bug

The issue 7439 was not completely addressed, the bot blocked the issue from commenting, but I think this main problem should be addressed:

As is, Net Core responds almost everything as camel case, except for model validations, which I think is odd:

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "|706e0133-46db8fce16cb675d.",
  "errors": {
    "Email": [
      "The Email field is not a valid e-mail address."
    ],
    "Password": [
      "The field Password must be a string or array type with a minimum length of '8'."
    ]
  }
}

As you see, all the properties are camel case but the field names, a workaround is to use CustomProblemDetailsFactory from the comment in the same issue.

To Reproduce

Create a project and a controller with a model with validations.

Make sure the controller is annotated with [ApiController], so validation is handled automatically.

Example:

    [Authorize]
    [ApiController]
    [Route("Api/Address/[controller]")]
    public class ColonyController : Controller
    {
        
        [HttpPost]
        [Route("[action]")]
        public Task<int> Create([FromBody]ColonyToAddDto colonyToAddDto)
        {
            return Task.FromResult(0);
        }
    }

With the model

    public class ColonyToAddDto
    {
        [Required]
        [StringLength(300, MinimumLength = 3)]
        public string Name { get; set; }
        [Range(1, 99999)]
        public int PostalCode { get; set; }
        [Range(1, double.PositiveInfinity)]
        public int MunicipalityId { get; set; }
    }

Further technical details

• ASP.NET Core version 3.1

[pranavkm]

Thanks for the issue report. We'll consider this issue during our next planning. In the meanwhile, writing a custom factory as you have discovered would be the best workaround.

[mjenzen]

A different, perhaps more direct to the issue workaround (albeit, requiring more code copy/pasting) is to override the implementation of ValidationProblemDetailsJsonConverter.

The issue here seems to be that the Errors dictionary is manually being serialized within this converter and the property names are written directly to the writer without going though the JsonSerializer. This manual dictionary serialization happens in other JsonConverters within the Mvc Project as well. (ex. ProblemDetailsJsonConverter)

the core runtime still appears to be having issues with dictionary serialization casing captured here (dotnet/runtime#31849 and dotnet/runtime#33508), which prevents directly serializing the dictionary in this case, so my workaround is adapted from a comment in one of the above-mentioned issues.

Here is the relevant code chunk within the ValidationProblemDetailsJsonConverter.Write function:

            writer.WriteStartObject(Errors);
            foreach (var kvp in value.Errors)
            {
                writer.WritePropertyName(kvp.Key);
                JsonSerializer.Serialize(writer, kvp.Value, kvp.Value?.GetType() ?? typeof(object), options);
            }
            writer.WriteEndObject();

my change:

            writer.WriteStartObject(Errors);
            foreach (var kvp in value.Errors)
            {
                // Updated Line below
                writer.WritePropertyName(options?.DictionaryKeyPolicy?.ConvertName(kvp.Key) ?? kvp.Key);
                JsonSerializer.Serialize(writer, kvp.Value, kvp.Value?.GetType() ?? typeof(object), options);
            }
            writer.WriteEndObject();

As these JsonConverter classes are internal, I had to copy/paste them into my project to fix. Here are the relevant additions.

ProblemDetailsJsonConverter.cs

using Microsoft.AspNetCore.Mvc;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading.Tasks;

namespace Sample.Web.Mvc.Infrastructure
{
    public class ProblemDetailsJsonConverter : JsonConverter<ProblemDetails>
    {
        private static readonly JsonEncodedText Type = JsonEncodedText.Encode("type");
        private static readonly JsonEncodedText Title = JsonEncodedText.Encode("title");
        private static readonly JsonEncodedText Status = JsonEncodedText.Encode("status");
        private static readonly JsonEncodedText Detail = JsonEncodedText.Encode("detail");
        private static readonly JsonEncodedText Instance = JsonEncodedText.Encode("instance");

        public override ProblemDetails Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            var problemDetails = new ProblemDetails();

            if (!reader.Read())
            {
                throw new JsonException();
            }

            while (reader.Read() && reader.TokenType != JsonTokenType.EndObject)
            {
                ReadValue(ref reader, problemDetails, options);
            }

            if (reader.TokenType != JsonTokenType.EndObject)
            {
                throw new JsonException();
            }

            return problemDetails;
        }

        public override void Write(Utf8JsonWriter writer, ProblemDetails value, JsonSerializerOptions options)
        {
            writer.WriteStartObject();
            WriteProblemDetails(writer, value, options);
            writer.WriteEndObject();
        }

        internal static void ReadValue(ref Utf8JsonReader reader, ProblemDetails value, JsonSerializerOptions options)
        {
            if (TryReadStringProperty(ref reader, Type, out var propertyValue))
            {
                value.Type = propertyValue;
            }
            else if (TryReadStringProperty(ref reader, Title, out propertyValue))
            {
                value.Title = propertyValue;
            }
            else if (TryReadStringProperty(ref reader, Detail, out propertyValue))
            {
                value.Detail = propertyValue;
            }
            else if (TryReadStringProperty(ref reader, Instance, out propertyValue))
            {
                value.Instance = propertyValue;
            }
            else if (reader.ValueTextEquals(Status.EncodedUtf8Bytes))
            {
                reader.Read();
                if (reader.TokenType == JsonTokenType.Null)
                {
                    // Nothing to do here.
                }
                else
                {
                    value.Status = reader.GetInt32();
                }
            }
            else
            {
                var key = reader.GetString();
                reader.Read();
                value.Extensions[key] = JsonSerializer.Deserialize(ref reader, typeof(object), options);
            }
        }

        internal static bool TryReadStringProperty(ref Utf8JsonReader reader, JsonEncodedText propertyName, out string value)
        {
            if (!reader.ValueTextEquals(propertyName.EncodedUtf8Bytes))
            {
                value = default;
                return false;
            }

            reader.Read();
            value = reader.GetString();
            return true;
        }

        internal static void WriteProblemDetails(Utf8JsonWriter writer, ProblemDetails value, JsonSerializerOptions options)
        {
            if (value.Type != null)
            {
                writer.WriteString(Type, value.Type);
            }

            if (value.Title != null)
            {
                writer.WriteString(Title, value.Title);
            }

            if (value.Status != null)
            {
                writer.WriteNumber(Status, value.Status.Value);
            }

            if (value.Detail != null)
            {
                writer.WriteString(Detail, value.Detail);
            }

            if (value.Instance != null)
            {
                writer.WriteString(Instance, value.Instance);
            }

            foreach (var kvp in value.Extensions)
            {
                writer.WritePropertyName(options?.DictionaryKeyPolicy?.ConvertName(kvp.Key) ?? kvp.Key);
                JsonSerializer.Serialize(writer, kvp.Value, kvp.Value?.GetType() ?? typeof(object), options);
            }
        }
    }
}

ValidationProblemDetailsJsonConverter.cs

using Microsoft.AspNetCore.Mvc;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading.Tasks;
using static Ostar.Web.Mvc.Infrastructure.ProblemDetailsJsonConverter;

namespace Sample.Web.Mvc.Infrastructure
{
    public class ValidationProblemDetailsJsonConverter : JsonConverter<ValidationProblemDetails>
    {
        private static readonly JsonEncodedText Errors = JsonEncodedText.Encode("errors");

        public override ValidationProblemDetails Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            var problemDetails = new ValidationProblemDetails();

            if (!reader.Read())
            {
                throw new JsonException();
            }

            while (reader.Read() && reader.TokenType != JsonTokenType.EndObject)
            {
                if (reader.ValueTextEquals(Errors.EncodedUtf8Bytes))
                {
                    var errors = JsonSerializer.Deserialize<Dictionary<string, string[]>>(ref reader, options);
                    foreach (var item in errors)
                    {
                        problemDetails.Errors[item.Key] = item.Value;
                    }
                }
                else
                {
                    ReadValue(ref reader, problemDetails, options);
                }
            }

            if (reader.TokenType != JsonTokenType.EndObject)
            {
                throw new JsonException();
            }

            return problemDetails;
        }

        public override void Write(Utf8JsonWriter writer, ValidationProblemDetails value, JsonSerializerOptions options)
        {
            writer.WriteStartObject();
            WriteProblemDetails(writer, value, options);

            writer.WriteStartObject(Errors);
            foreach (var kvp in value.Errors)
            {
                writer.WritePropertyName(options?.DictionaryKeyPolicy?.ConvertName(kvp.Key) ?? kvp.Key);
                JsonSerializer.Serialize(writer, kvp.Value, kvp.Value?.GetType() ?? typeof(object), options);
            }
            writer.WriteEndObject();

            writer.WriteEndObject();
        }
    }
}

Finally register the converters within AddJsonOptions in Startup.cs

            services.AddControllersWithViews(config => config.Filters.Add<ContextManagerActionFilter>()).AddJsonOptions(opts =>
            {
                opts.JsonSerializerOptions.DictionaryKeyPolicy = JsonNamingPolicy.CamelCase;
                opts.JsonSerializerOptions.Converters.Add(new ValidationProblemDetailsJsonConverter());
            });

EDIT: Bugfix to fix property name having literal quotes around it :(

[antonioortizpola]

That is great @mjenzen, I am hitting some cases where the deserialization is wrong, this could be a step in the right direction. For my part I have already returned to newtonsoft, system.text does not seem ready for production environments yet

[bcronje]

@javiercn @pranavkm can I implore that you reconsider the affected-few and severity-nice-to-have labels and re-apply it as a bug? This issue is still present in .NET 5 and clearly does not honor the DictionaryKeyPolicy = JsonNamingPolicy.CamelCase policy. I think it is safe to say the majority of users consuming an API from javascript expects camelcase properties, so not sure why you would think it only affects a few?

See also the following two issues which has been closed by bot:

#7439
dotnet/runtime#31849

How can the below "Email" and "Password" keys not be considered a bug?

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "|706e0133-46db8fce16cb675d.",
  "errors": {
    "Email": [
      "The Email field is not a valid e-mail address."
    ],
    "Password": [
      "The field Password must be a string or array type with a minimum length of '8'."
    ]
  }
}

[Snowirbix]

Yes please I just lost a few hours because of this bug.
The documentation states that DictionaryKeyPolicy is used to convert a IDictionary key's name to another format, such as camel-casing (serialization only)
Yet this ValidationProblemDetails does not respect this behaviour.

[pranavkm]

Thanks. Are you open to sending us a PR to change this? #17999 (comment) would be pretty close to what you might do in the in-box converters.

[mjenzen]

Thanks. Are you open to sending us a PR to change this? #17999 (comment) would be pretty close to what you might do in the in-box converters.

I could do that

[p3rh5g]

@mjenzen For nested validation errors like Foo.Bar this would convert to foo.Bar. When using camelcase you probably have sent something like foo.bar in your request. I would like to suggest this change to ValidationProblemDetailsJsonConverter.Write:

public override void Write(Utf8JsonWriter writer, ValidationProblemDetails value, JsonSerializerOptions options)
{
    writer.WriteStartObject();
    WriteProblemDetails(writer, value, options);

    writer.WriteStartObject(Errors);
    foreach (var kvp in value.Errors)
    {
        writer.WritePropertyName(ConvertName(kvp.Key, options));
        JsonSerializer.Serialize(writer, kvp.Value, kvp.Value?.GetType() ?? typeof(object), options);
    }

    writer.WriteEndObject();

    writer.WriteEndObject();
}

private static string ConvertName(string name, JsonSerializerOptions options) =>
    options?.DictionaryKeyPolicy != null
        ? string.Join(".", name.Split(".").Select(options.DictionaryKeyPolicy.ConvertName))
        : name;