JsonElement.GetInt32() fails when working with JsonElement directly

[Skyppid]

Description

We have a model class that represents our JSON data model. For some fields we have dynamic data types. Value can be string, integer, double, boolean, etc.

So for this we use a model class that defines the value as JsonElement so we can access the JsonValueKind and parse it depending on our needs and what it actually represents.

public sealed record DocumentValue(JsonElement Value, bool ValueFound, double Confidence, RelationalOperator? Operator, string Unit);

We parse our incoming JSON like this JsonSerializer.Deserialize<TMessage>(data, KafkaAdapterFactory.SerializationOptions) where in that case TMessage is our model which contains a list of DocumentValue at certain points in the structure.

Here's an example of the JSON (just a fragment) what we parse:

"fields": [
{
	"name": "Count",
	"value": {
		"value": 5,
		"valueFound": true,
		"confidence": 0,
		"operator": null,
		"unit": null
	}
}, // ...
]

As you see, it's clearly no floating point number (and will never be in that case). When I now set a breakpoint on this line:
var actualCount = count.Value.Value.GetInt32();

I get an FormatException. On the breakpoint I can verify that value kind is JsonValueKind.Number. So it should certainly be able to use GetInt32() without issues.

The exception looks like this:

System.FormatException: One of the identified items was in an invalid format.
at void System.Text.Json.ThrowHelper.ThrowFormatException()
at int System.Text.Json.JsonElement.GetInt32()

Checking the JsonElement with the debugger it says: Number: 5.0. When using GetDouble() and converting this into an int it works fine.

PS: To clarify, the setting used for deserialization are these:

    internal static JsonSerializerOptions SerializationOptions = new()
    {
        Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) },
        PropertyNameCaseInsensitive = true,
        WriteIndented = false
    };

EDIT: More detailed information as I've stepped into it:

            if (Utf8Parser.TryParse(segment, out int tmp, out int consumed) &&
                consumed == segment.Length)

Here segment is a slice of 3 bytes, the call returns false, tmp: 5, consumed: 1, comsumed (1) == segment.Length (3). These are the values of it. So seems like while it parses it correctly, the check still fails due to the segment length mismatching the consumed parameter.

Reproduction Steps

Use a model with JsonElement as property type. Parse a JSON document where this value is represented as non-floating number i.e. "5". Deserialize using JsonSerializer.Deserialize<TMessage>() and evaluate the actual value. It should be of kind Number and GetInt32() should work on this element.

Expected behavior

JsonElement.GetInt32() should return a valid value. Even if it's internally represented as floating-point number, it should be able to convert it.

Actual behavior

A FormatException is being thrown.

Regression?

Can't tell, newly implemented feature that wasn't there before we upgraded to .NET 8.

Known Workarounds

Retrieving the value using JsonElement.GetDouble() and converting it works but is an unncessary extra step that shouldn't be necessary.

Configuration

.NET 8 (latest)
Windows 11, Linux (not sure which one exactly)
x64
Likely unrelated to configuration

Other information

No response

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

We have a model class that represents our JSON data model. For some fields we have dynamic data types. Value can be string, integer, double, boolean, etc.

So for this we use a model class that defines the value as JsonElement so we can access the JsonValueKind and parse it depending on our needs and what it actually represents.

public sealed record DocumentValue(JsonElement Value, bool ValueFound, double Confidence, RelationalOperator? Operator, string Unit);

We parse our incoming JSON like this JsonSerializer.Deserialize<TMessage>(data, KafkaAdapterFactory.SerializationOptions) where in that case TMessage is our model which contains a list of DocumentValue at certain points in the structure.

Here's an example of the JSON (just a fragment) what we parse:

"fields": [
{
	"name": "Count",
	"value": {
		"value": 5,
		"valueFound": true,
		"confidence": 0,
		"operator": null,
		"unit": null
	}
}, // ...
]

As you see, it's clearly no floating point number (and will never be in that case). When I now set a breakpoint on this line:
var actualCount = count.Value.Value.GetInt32();

I get an FormatException. On the breakpoint I can verify that value kind is JsonValueKind.Number. So it should certainly be able to use GetInt32() without issues.

The exception looks like this:

System.FormatException: One of the identified items was in an invalid format.
at void System.Text.Json.ThrowHelper.ThrowFormatException()
at int System.Text.Json.JsonElement.GetInt32()

Checking the JsonElement with the debugger it says: Number: 5.0. When using GetDouble() and converting this into an int it works fine.

PS: To clarify, the setting used for deserialization are these:

    internal static JsonSerializerOptions SerializationOptions = new()
    {
        Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) },
        PropertyNameCaseInsensitive = true,
        WriteIndented = false
    };

Reproduction Steps

Use a model with JsonElement as property type. Parse a JSON document where this value is represented as non-floating number i.e. "5". Deserialize using JsonSerializer.Deserialize<TMessage>() and evaluate the actual value. It should be of kind Number and GetInt32() should work on this element.

Expected behavior

JsonElement.GetInt32() should return a valid value. Even if it's internally represented as floating-point number, it should be able to convert it.

Actual behavior

A FormatException is being thrown.

Regression?

Can't tell, newly implemented feature that wasn't there before we upgraded to .NET 8.

Known Workarounds

Retrieving the value using JsonElement.GetDouble() and converting it works but is an unncessary extra step that shouldn't be necessary.

Configuration

.NET 8 (latest)
Windows 11, Linux (not sure which one exactly)
x64
Likely unrelated to configuration

Other information

No response

Author:	Skyppid
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[elgonzo]

Not reproducible (https://dotnetfiddle.net/fKbaC4)

[eiriktsarpalis]

Not reproducible (https://dotnetfiddle.net/fKbaC4)

I presume the issue occurs when trying to read a 5.0 value as an integer? This is by design, generally speaking STJ will not attempt to auto-convert decimal numbers to integers.

EDIT I see now that the OP specifically refers to this being an issue when reading non-decimal numbers, which as @elgonzo points out is not reproducible.

This issue has been marked needs-author-action and may be missing some important information.

[Skyppid]

Okay I gotta correct myself. I just decoded the data I get from Kafka before deserializing it. Turns out the value actually is 5.0 in that case. Kafka UI shows it as 5 though.

So @eiriktsarpalis you say it's by design? Is there a reason to it? I feel it's kinda bad since there's no differentiation between floating point and integer numbers by looking at the value kind. How should I be able to tell if I must use GetInt32() or GetDouble() then? Please don't tell me it's expected to try-catch it or do some kind of TryGet*() cascade?

I feel like either it should be provided whether it's a floating point number or integer OR it should convert it. After all it's something a developer should be aware of that if you use GetInt32() on a floating point it will end up rounding it. That's basic knowledge for any developer.

Obviously in this case it might be an issue of another service sending it in the wrong format, but I feel like it could use some improvement here. Feels odd to use and I'm probably not the only one who gets confused by that exception. At least the docs don't tell you that it has to exactly match.

[gregsdennis]

@Skyppid I've got .GetNumber() and .GetInteger() extensions in my Json.More.Net library (see my json-everything repo if you want to copy out the code) that may help you. They check for all the applicable number types. Could at least give you a starting point to meet your need.

[eiriktsarpalis]

@Skyppid Out of the box the library is strict when it comes to enforcing JSON contracts. The presence of a decimal point in the deserialized JSON suggests that there is a mismatch between the ingested schema and the contract of the deserialized type, so the serializer will fail early, even if in that particular case the underlying value can be coerced to an integer without loss of precision.

There are two possible workarounds that can be followed here:

1. Assuming that the contract admits decimals, deserialize to a type that admits decimal representations such as double or decimal or
2. Write a custom converter for int that converts decimal values to integers using conversion logic that is appropriate for your application.

Hope this helps.

[Skyppid]

Thanks @gregsdennis, but my point still stands. I see room for improvement here. To me this looks like a design-flaw in the .NET API. At least I'd like to hear the reasoning why this decision was made. Or maybe this can be improved in .NET 9?

I'll need to work around it anyway. Right now we fixed it by changing the data type the other service sends to an actual integer. Count makes no sense to be a floating point. Still I'd expect GetInt32() to return an integer if I use it on a number field.

@eiriktsarpalis Thanks for the explanation. I see your point, but still I don't see why it should be impossible to use GetInt32() on a floating point number. As developer it's obvious that this will apply rounding when converting. Either I have a contract in place by enforcing the schema or I need to handle the possible cases on my own. I feel like this really limits the developer in case you don't use JSON schemas.

For me the minimum would be either add that information to the docs as nowhere is mentioned that it will fail. A double can be represented as integer, with loss of precision obviously. But it's not really clear that it will fail if it isn't an actual integer.
Otherwise expose that information in the JsonElement if it's an integer or double or let it convert. But right now I would say the API is confusing and not convenient to use at all.

At least I would have avoided switching back from Newtonsoft had I known what amount of issues this change incurs.

[eiriktsarpalis]

For me the minimum would be either add that information to the docs as nowhere is mentioned that it will fail.

Did you check the API documentation for the method?

To me this looks like a design-flaw in the .NET API. At least I'd like to hear the reasoning why this decision was made. Or maybe this can be improved in .NET 9?

The library has been designed with standards compliance, performance and security in mind. As such, it is much less flexible compared to what Json.NET provides and that is very much intentional. Out of the box, it won't attempt to perform any implicit conversions (e.g. number to bool, decimal to int or, say, accepting date values not conforming to ISO8601) or allow non-standard JSON such as comments, trailing commas or properties without quotes.

A good way to think of this is that STJ forces a strongly typed deserialization model. Similar to how

int value = 5.0;

is a type error in C#, it is also a type error in STJ. At the same time, we know that the web can be much more fluid when it comes to schema enforcement, which is why the library comes with extensibility that let you configure the desired behaviour whenever necessary.