System.Text.Json.JsonReader is not reading numbers correctly

[Gonkers]

Description

System.Text.Json cannot read JSON that just consists of just a number. According to the specifications, this should be valid.

System.Text.Json.JsonReaderException
  HResult=0x80131500
  Message='0x00' is an invalid end of a number. Expected 'E' or 'e'. LineNumber: 0 | BytePositionInLine: 5.
  Source=System.Text.Json

Here are some unit tests to reproduce the issue.

        public class NumberValueTests
        {
            public static JsonTokenType GetTokenType(string jsonValueOnly, byte[] buffer)
            {
                var bytesEncoded = Encoding.UTF8.GetBytes(jsonValueOnly, buffer);
                var jsonReaderOptions = new JsonReaderOptions { AllowTrailingCommas = true, CommentHandling = JsonCommentHandling.Skip };
                var readerState = new JsonReaderState(jsonReaderOptions);
                var reader = new Utf8JsonReader(buffer, false, readerState);
                reader.Read();

                return reader.TokenType;
            }


            [Fact]
            public void  NumberValues()
            {
                Assert.Equal(JsonTokenType.String, GetTokenType("\"4444\"", new byte[10]));
                Assert.Equal(JsonTokenType.False, GetTokenType("false", new byte[10]));
                Assert.Equal(JsonTokenType.True, GetTokenType("true", new byte[10]));
                Assert.Equal(JsonTokenType.Null, GetTokenType("null", new byte[10]));

                // !!!  THESE SHOULD BE NUMBER  !!!
                Assert.Equal(JsonTokenType.Number, GetTokenType("44.44", new byte[5]));
                Assert.Equal(JsonTokenType.Number, GetTokenType("4444", new byte[4]));

                // !!!  THIS SHOULD NOT THOW AN INTERNAL EXCEPTION  !!!
                Assert.Throws<System.Text.Json.JsonReaderException>(() => GetTokenType("4444", new byte[10]));
            }
        }

Configuration

• Which version of .NET is the code running on? 5.0.300
• What OS and version, and what distro if applicable? Windows 10
• What is the architecture (x64, x86, ARM, ARM64)? x64
• Do you know whether it is specific to that configuration? I don't know.

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

System.Text.Json cannot read JSON that just consists of just a number. According to the specifications, this should be valid.

System.Text.Json.JsonReaderException
  HResult=0x80131500
  Message='0x00' is an invalid end of a number. Expected 'E' or 'e'. LineNumber: 0 | BytePositionInLine: 5.
  Source=System.Text.Json

Here are some unit tests to reproduce the issue.

    public class UnitTest
    {
        public class NumberValueTests
        {
            public static JsonTokenType GetTokenType(string jsonValueOnly, byte[] buffer)
            {
                var bytesEncoded = Encoding.UTF8.GetBytes(jsonValueOnly, buffer);
                var jsonReaderOptions = new JsonReaderOptions { AllowTrailingCommas = true, CommentHandling = JsonCommentHandling.Skip };
                var readerState = new JsonReaderState(jsonReaderOptions);
                var reader = new Utf8JsonReader(buffer, false, readerState);
                reader.Read();

                return reader.TokenType;
            }


            [Fact]
            public void  NumberValues()
            {
                Assert.Equal(JsonTokenType.String, GetTokenType("\"4444\"", new byte[10]));
                Assert.Equal(JsonTokenType.False, GetTokenType("false", new byte[10]));
                Assert.Equal(JsonTokenType.True, GetTokenType("true", new byte[10]));
                Assert.Equal(JsonTokenType.Null, GetTokenType("null", new byte[10]));

                // !!!  THESE SHOULD BE NUMBER  !!!
                Assert.Equal(JsonTokenType.Number, GetTokenType("44.44", new byte[5]));
                Assert.Equal(JsonTokenType.Number, GetTokenType("4444", new byte[4]));

                // !!!  THIS SHOULD NOT THOW AN INTERNAL EXCEPTION  !!!
                Assert.Throws<System.Text.Json.JsonReaderException>(() => GetTokenType("4444", new byte[10]));
            }
        }
    }

Configuration

• Which version of .NET is the code running on? 5.0.300
• What OS and version, and what distro if applicable? Windows 10
• What is the architecture (x64, x86, ARM, ARM64)? x64
• Do you know whether it is specific to that configuration? I don't know.

Author:	Gonkers
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[Tornhoof]

You're ignoring the length of GetBytes, so you're passing the Utf8JsonReader the full byte array and not the sliced one. You need to slice your buffer properly.
var reader = new Utf8JsonReader(buffer.AsSpan(0, bytesEncoded), false, readerState);
That it happens to work for the other types is because string has a delimiter symbol and null, false/true a specific fixed length tokens.
See #29947 for someone with the same problem.

As for specification reference, see https://datatracker.ietf.org/doc/html/rfc8259#section-2, whitespace is defined as:

ws = *(
              %x20 /              ; Space
              %x09 /              ; Horizontal tab
              %x0A /              ; Line feed or New line
              %x0D )              ; Carriage return

and this excludes 0x00 bytes.

[Gonkers]

@Tornhoof, you seem to have misunderstood my tests. Let me explain further.

1. Assert.Equal(JsonTokenType.Number, GetTokenType("44.44", new byte[5]));
   Assert.Equal(JsonTokenType.Number, GetTokenType("4444", new byte[4]));

   These lines produce JsonTokenType.None not JsonTokenType.Number like they should. These tests fail. The byte[] are the exact length length they need to be. Which you claim that I am not doing.

2. Assert.Throws<System.Text.Json.JsonReaderException>(() => GetTokenType("4444", new byte[10]));
   

   This line does not compile, it generates an error. The exception thrown here by the call GetTokenType("4444", new byte[10]) is from an internal class named JsonReaderException. I cannot use a try...catch block to catch the exception specifically because JsonReaderException does not exist to me. This is a bug.

   The type or namespace name 'JsonReaderException' does not exist in the namespace 'System.Text.Json' (are you missing an assembly reference?)
   

   In summary, this unit test should not pass

   try
   {
       GetTokenType("4444", new byte[10]);
   }
   catch (System.Exception ex)
   {
       Assert.Equal("System.Text.Json.JsonReaderException", ex.GetType().FullName);
   }

I provided unit tests to demonstrate the bugs. Please in the future run them to give us users the benefit of the doubt.

Thanks.

[Gonkers]

So to catch this internal exception I guess I'll need to do this...
catch (JsonException ex) when (ex.GetType().FullName.Equals("System.Text.Json.JsonReaderException"))

And I'll need to add spaces to all of my JSON so JsonReader doesn't throw exceptions...

Assert.Equal(JsonTokenType.Number, GetTokenType("44.44" + " ", new byte[6]));
Assert.Equal(JsonTokenType.Number, GetTokenType("4444" + " ", new byte[5]));
Assert.Equal(JsonTokenType.Number, GetTokenType("4444" + " ", new byte[10]));

[davidfowl]

var bytesEncoded = Encoding.UTF8.GetBytes(jsonValueOnly, buffer);

This token isn't used in GetTokenType, that's what @Tornhoof was getting at.

[Tornhoof]

I provided unit tests to demonstrate the bugs. Please in the future run them to give us users the benefit of the doubt.

Actually I did, that's how I saw your problem with the slice. The other problem your code has, is that you set finalblock to false new Utf8JsonReader(buffer, false, readerState);. According to the comment for that parameter it specifies that no other data is following it, I admit I forgot that in my comment, as I just added the slice and not the true.
If you pass true and fix the buffer slice the tests pass.

runtime/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs

Lines 197 to 199 in 4aa29f4

	/// <param name="isFinalBlock">True when the input span contains the entire data to process.
	/// Set to false only if it is known that the input span contains partial data with more data to follow.</param>
	/// <param name="state">If this is the first call to the ctor, pass in a default state. Otherwise,

As for the exception, JsonReaderException is derived from JsonException that's the one you want to catch.

[Gonkers]

Okay please humor me. I do really appreciate you all responding. Can you please tell me if any of these json files are invalid? https://github.com/Gonkers/HowDoesJsonWork

I can read all of these files. I CANNOT read the number.json file without failing the test.

[Fact]
public void ReadingFiles()
{
    var dir = new DirectoryInfo(@"C:\Users\Gonke\Dev\HowDoesJsonWork");
    var jsonFiles = dir.GetFiles("*.json");
    foreach (var file in jsonFiles)
    {
        using var jsonStream = file.Open(FileMode.Open, FileAccess.Read, FileShare.Read);
        var buffer = new byte[4096];
        var bytesRead = jsonStream.Read(buffer);
        var jsonReaderOptions = new JsonReaderOptions { AllowTrailingCommas = true, CommentHandling = JsonCommentHandling.Skip };
        var readerState = new JsonReaderState(jsonReaderOptions);
        var reader = new Utf8JsonReader(buffer.AsSpan(0, bytesRead), false, readerState);;
        var readSucessful = reader.Read();
        Assert.True(readSucessful);
    }
}

[Tornhoof]

Your files are all valid, your code is just not valid, as described before your code is ignoring the length of the file (in your examples) and you're passing false into the Utf8JsonReader.
Try

[Fact]
public void ReadingFiles()
{
	var dir = new DirectoryInfo(@"D:\Repos\HowDoesJsonWork");
	var jsonFiles = dir.GetFiles("*.json");
	foreach (var file in jsonFiles)
	{
		using var jsonStream = file.Open(FileMode.Open, FileAccess.Read, FileShare.Read);
		var buffer = new byte[4096];
		var bytesRead = jsonStream.Read(buffer);
		var jsonReaderOptions = new JsonReaderOptions { AllowTrailingCommas = true, CommentHandling = JsonCommentHandling.Skip };
		var readerState = new JsonReaderState(jsonReaderOptions);
		var reader = new Utf8JsonReader(buffer.AsSpan(0, bytesRead), true, readerState);
		var readSucessful = reader.Read();
		Assert.True(readSucessful);
	}
}

As for why your bug happens only for number, as I said, your other examples are either fixed length tokens (null, false,true) or have boundary characters (string)

[Gonkers]

I provided unit tests to demonstrate the bugs. Please in the future run them to give us users the benefit of the doubt.

Actually I did, that's how I saw your problem with the slice. The other problem your code has, is that you set finalblock to false new Utf8JsonReader(buffer, false, readerState);. According to the comment for that parameter it specifies that no other data is following it, I admit I forgot that in my comment, as I just added the slice and not the true.
If you pass true and fix the buffer slice the tests pass.

runtime/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs

Lines 197 to 199 in 4aa29f4

	/// <param name="isFinalBlock">True when the input span contains the entire data to process.
	/// Set to false only if it is known that the input span contains partial data with more data to follow.</param>
	/// <param name="state">If this is the first call to the ctor, pass in a default state. Otherwise,

As for the exception, JsonReaderException is derived from JsonException that's the one you want to catch.

OMG Okay I get it now. While I was coding up that last example I didn't even see your reply come in. I get it now. Sorry for the confusion. Thanks.

[Gonkers]

This is a note for the next person to come along. This is a more "correct" example. When you're reading your stream you need to detect if you're at the end of if it or if there is possibly more to come. Make note of the following line in the example.

var isFinalBlock = bytesRead < buffer.Length;

Also, if reader.Read() ever returns false, it means you've likely reached the end of your buffer and you need to get more data. (not shown in example).

[Fact]
public void ReadingFiles()
{
    var dir = new DirectoryInfo(@"C:\Users\Gonke\Dev\HowDoesJsonWork");
    var jsonFiles = dir.GetFiles("*.json");
    foreach (var file in jsonFiles)
    {
        using var jsonStream = file.Open(FileMode.Open, FileAccess.Read, FileShare.Read);
        var buffer = new byte[4096];
        var bytesRead = jsonStream.Read(buffer);
        var jsonReaderOptions = new JsonReaderOptions { AllowTrailingCommas = true, CommentHandling = JsonCommentHandling.Skip };
        var readerState = new JsonReaderState(jsonReaderOptions);
        var isFinalBlock = bytesRead < buffer.Length;
        var reader = new Utf8JsonReader(buffer.AsSpan(0, bytesRead), isFinalBlock, readerState);
        var readSucessful = reader.Read();

        Assert.True(readSucessful);
    }
}

I cannot add files with .json extension so here are the samples.
false.txt
null.txt
number.txt
string.txt
true.txt

[Gonkers]

An additional note for future readers, I was adapting this example and it uses a static false for the isFinalBlock parameter. I am filing a bug with the docs team to see if the example can be updated.