System.Text.Json 9.0.6 Error while using PipeWriter

[shargon]

Description

Related to: #116733

Using Kestrel with JSON sometimes produces malformed JSON that is impossible to parse.
I found a way to reproduce it.
Something is mutable, I didn't understand why, it doesn't happens in the first iteration.

Reproduction Steps

        var obj = new SortedDictionary<string, string>()
        {
            { "A" , "".PadLeft(10000,'\\')  },
            { "B" , "".PadLeft(10000,'\'')  },
            { "C" , "".PadLeft(10000,'"')  },
            { "D" , "".PadLeft(10000,'\n')  },
            { "E" , "".PadLeft(10000,'\\')  },
            { "F" , "".PadLeft(10000,'\'')  },
            { "G" , "".PadLeft(10000,'"')  },
            { "H" , "".PadLeft(10000,'\n')  }
        };

        var col = new Dictionary<string, SortedDictionary<string, string>>
        {
            { "dicA" , obj },
            { "dicB" ,obj}
        };

        var settings = new JsonSerializerOptions()
        {
            Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
            PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
            WriteIndented = false,
            NewLine = "\n",
            IncludeFields = false,
            IgnoreReadOnlyFields = false,
            IgnoreReadOnlyProperties = false,
            PropertyNameCaseInsensitive = true,
            DefaultBufferSize = ushort.MaxValue,
            ReadCommentHandling = JsonCommentHandling.Skip,
            NumberHandling = JsonNumberHandling.AllowReadingFromString,
            DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
        };

start:
        var options = new ParallelOptions
        {
            MaxDegreeOfParallelism = 10
        };

        var taskA = Parallel.ForEachAsync(Enumerable.Range(0, 1000), options, (i, c) =>
        {
            c.ThrowIfCancellationRequested();

            var json = JsonSerializer.Serialize(col, settings);

            try
            {
                var re = JsonSerializer.Deserialize<Dictionary<string, SortedDictionary<string, string>>>(json, settings);
            }
            catch (Exception ex)
            {
                Console.WriteLine($"ERROR: {ex}");
            }

            return ValueTask.CompletedTask;
        });

        var taskB = Task.Run(async () =>
        {
            var cts = new CancellationTokenSource();
            var pipe = new Pipe(new PipeOptions());
            var writer = pipe.Writer;
            var reader = pipe.Reader;
            using var memoryStream = new MemoryStream();

            var readTask = Task.Run(async () =>
            {
                while (true)
                {
                    ReadResult result = await reader.ReadAsync(cts.Token);
                    var buffer = result.Buffer;

                    foreach (var segment in buffer)
                        await memoryStream.WriteAsync(segment, cts.Token);

                    reader.AdvanceTo(buffer.End);

                    if (result.IsCompleted)
                        break;
                }

                await reader.CompleteAsync();
            });

            await JsonSerializer.SerializeAsync(writer, col, settings);
            await writer.CompleteAsync();

            await memoryStream.FlushAsync();
            memoryStream.Seek(0, SeekOrigin.Begin);

            var json = Encoding.UTF8.GetString(memoryStream.ToArray());
            try
            {
                var re = JsonSerializer.Deserialize<Dictionary<string, SortedDictionary<string, string>>>(json, settings);
            }
            catch (Exception ex)
            {
                Console.WriteLine($"ERROR: {ex}");
            }
        });

        Task.WaitAll(taskA, taskB);
        goto start;

Expected behavior

Good Json should be created

Actual behavior

ERROR: System.Text.Json.JsonException: Expected depth to be zero at the end of the JSON payload. There is an open JSON object or array that should be closed. Path: $.dicB.E | LineNumber: 0 | BytePositionInLine: 230108.
 ---> System.Text.Json.JsonReaderException: Expected depth to be zero at the end of the JSON payload. There is an open JSON object or array that should be closed. LineNumber: 0 | BytePositionInLine: 230108.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
   at System.Text.Json.Utf8JsonReader.Read()
   at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TDictionary& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
   at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TDictionary& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, T& value, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Deserialize(Utf8JsonReader& reader, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo`1 jsonTypeInfo, Nullable`1 actualByteCount)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo`1 jsonTypeInfo)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at R4S.Program.<>c__DisplayClass6_0.<<Main>b__1>d.MoveNext() in C:\Red4Sec\R4S\src\R4S\Program.cs:line 139

Regression?

No response

Known Workarounds

Avoid PipeWriter.

Configuration

No response

Other information

<PackageReference Include="System.Text.Json" Version="9.0.6" />

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.