JsonSerializer.Deserialize throws random errors running in parallel on 64-bit .NET Core 3.1 or .NET 6 runtime on Windows 7

[fantaclaus]

Description

When JsonSerializer.Deserialize is invoked in parallel in multiple threads it throws random errors.

This behaviour is observed in 64-bit runtime of .NET Core 3.1, .NET 5 and .NET 6 on Windows 7.

Reproduction Steps

Create and run the following Console Application for .NET 6 code:

using System;
using System.Diagnostics;
using System.Linq;
using System.Text.Json;
using System.Threading.Tasks;

var data = Enumerable.Repeat("quick brown fox jumps over lazy dog", 1000);
var json = JsonSerializer.Serialize(data);

Parallel.For(0, 100000000, i =>
{
	try
	{
		JsonSerializer.Deserialize<object>(json);
	}
	catch (Exception x)
	{
		var msg = $"{x.GetType().FullName}: {x.Message}";

		Console.WriteLine(msg);
		Trace.WriteLine(msg);
	}
});

Try different count for Enumerable.Repeat from 100 to 1000 to get different kind of errors.

Expected behavior

No error messages in the console.

Actual behavior

A lot of random errors like these (not a complete list):

System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2586.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2434.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2890.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2890.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 1674.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3194.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 154.
System.Text.Json.JsonException: Expected end of string, but instead reached end of data. Path: $ | LineNumber: 0 | BytePositionInLine: 3801.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 1978.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2890.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3650.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3194.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2738.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3650.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3042.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3042.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 1826.

Regression?

• 32-bit version of .NET runtime works correctly on Windows 7.
• 32-bit and 64-bit version of .NET runtime work correctly on Windows 10.
• The code built for .NET Framework 4.8 with System.Text.Json v6.0.2 package works correctly on either 32-bit or 64-bit runtime on Windows 7.

Known Workarounds

No response

Configuration

• .NET Core 3.1, .NET 5, .NET 6 v6.0.2, 64-bit runtime
• Windows 7 Pro SP1 (64-bit)
• it is specific to this configuration

>dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.101
 Commit:    ef49f6213a

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.101\

Host (useful for support):
  Version: 6.0.1
  Commit:  3a25a7f1cc

.NET SDKs installed:
  3.1.416 [C:\Program Files\dotnet\sdk]
  5.0.211 [C:\Program Files\dotnet\sdk]
  5.0.301 [C:\Program Files\dotnet\sdk]
  5.0.303 [C:\Program Files\dotnet\sdk]
  6.0.101 [C:\Program Files\dotnet\sdk]

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When JsonSerializer.Deserialize is invoked in parallel in multiple threads it throws random errors.

This behaviour is observed in 64-bit runtime of .NET Core 3.1, .NET 5 and .NET 6 on Windows 7.

Reproduction Steps

Create and run the following Console Application for .NET 6 code:

using System;
using System.Diagnostics;
using System.Linq;
using System.Text.Json;
using System.Threading.Tasks;

var data = Enumerable.Repeat("quick brown fox jumps over lazy dog", 1000);
var json = JsonSerializer.Serialize(data);

Parallel.For(0, 100000000, i =>
{
	try
	{
		JsonSerializer.Deserialize<object>(json);
	}
	catch (Exception x)
	{
		var msg = $"{x.GetType().FullName}: {x.Message}";

		Console.WriteLine(msg);
		Trace.WriteLine(msg);
	}
});

Try different count for Enumerable.Repeat from 100 to 1000 to get different kind of errors.

Expected behavior

No error messages in the console.

Actual behavior

A lot of random errors like these (not a complete list):

System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2586.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2434.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2890.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2890.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 1674.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3194.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 154.
System.Text.Json.JsonException: Expected end of string, but instead reached end of data. Path: $ | LineNumber: 0 | BytePositionInLine: 3801.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 1978.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2890.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3650.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3194.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 2738.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3650.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3042.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 3042.
System.Text.Json.JsonException: 'q' is invalid after a value. Expected either ',', '}', or ']'. Path: $ | LineNumber: 0 | BytePositionInLine: 1826.

Regression?

• 32-bit version of .NET runtime works correctly on Windows 7.
• 32-bit and 64-bit version of .NET runtime work correctly on Windows 10.
• The code built for .NET Framework 4.8 with System.Text.Json v6.0.2 package works correctly on either 32-bit or 64-bit runtime on Windows 7.

Known Workarounds

No response

Configuration

• .NET Core 3.1, .NET 5, .NET 6 v6.0.2, 64-bit runtime
• Windows 7 Pro SP1 (64-bit)
• it is specific to this configuration

>dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.101
 Commit:    ef49f6213a

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.101\

Host (useful for support):
  Version: 6.0.1
  Commit:  3a25a7f1cc

.NET SDKs installed:
  3.1.416 [C:\Program Files\dotnet\sdk]
  5.0.211 [C:\Program Files\dotnet\sdk]
  5.0.301 [C:\Program Files\dotnet\sdk]
  5.0.303 [C:\Program Files\dotnet\sdk]
  6.0.101 [C:\Program Files\dotnet\sdk]

Other information

No response

Author:	fantaclaus
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[danmoseley]

@VSadov could this be #65292 ? It's Win 7 specific.

[danmoseley]

Also if it is, did the necessary fixes get into 7.0 Preview 2 for them to try? I see it will be in 6.0.4.

[krwq]

we've fixed bunch of issues similar to this in 7.0. I'd recommend to try preview and giving us feedback. I think the best workaround I can think of for previews versions is first running advanced serialization/deserialization scenario in single thread and after that running them in parallel since many of these bugs are related to how internal metadata is initialized

[MichalPetryka]

we've fixed bunch of issues similar to this in 7.0. I'd recommend to try preview and giving us feedback. I think the best workaround I can think of for previews versions is first running advanced serialization/deserialization scenario in single thread and after that running them in parallel since many of these bugs are related to how internal metadata is initialized

This issue says that it's specific to Windows 7 and .Net 7 does not support Windows 7.

[eiriktsarpalis]

I don't this is something we might want to service in .NET 6. Also, per dotnet/announcements#226 Windows 7 will not be supported in future releases so I'd be inclined to close this.