Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Text.Json failing some large file tests #59678

Closed
bartonjs opened this issue Sep 27, 2021 · 32 comments
Closed

System.Text.Json failing some large file tests #59678

bartonjs opened this issue Sep 27, 2021 · 32 comments
Labels
area-System.Text.Json blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-mac-os-x macOS aka OSX test-failure
Milestone
9.0.0

Comments

@bartonjs
Copy link
Member

bartonjs commented Sep 27, 2021 •
edited by build-analysis bot
Loading

https://dev.azure.com/dnceng/public/_build/results?buildId=1387876&view=ms.vss-test-web.build-test-results-tab&runId=40401306&resultId=194463&paneView=debug

If I didn't know any better, I'd say "memory corruption".

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-59626-merge-ff748885001649f2ad/System.Text.Json.Tests/1/console.55b96e63.log?sv=2019-07-07&se=2021-10-17T15%3A49%3A24Z&sr=c&sp=rl&sig=jkQ9vlfitTdm9LAJkyCb4ZnZxrI5Gb%2F8qvt%2FSn0S27M%3D

Console log: 'System.Text.Json.Tests' from job ff748885-0016-49f2-ad26-20905cd95f3d workitem 0045d34b-f5a3-424a-8789-b58e57023e81 (osx.1015.amd64.open) executed on machine dci-mac-build-195
+ ./RunTests.sh --runtime-path /tmp/helix/working/A15608F4/p
----- start Mon Sep 27 11:49:52 EDT 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/A15608F4/p/dotnet exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/A15608F4/w/A1BD08B8/e /private/tmp/helix/working/A15608F4/w/A1BD08B8/e
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 2776 of 2815 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 12)
    System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(formatted: False, skipValidation: False) [FAIL]
      Assert.Equal() Failure
                                          ↓ (pos 1102164)
      Expected: ···qKioqKioqKioqKioqKiosKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq···
      Actual:   ···qKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq···
                                          ↑ (pos 1102164)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(738,0): at System.Text.Json.JsonTestHelper.AssertContentsAgainstJsonNet(String expectedValue, String value, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(714,0): at System.Text.Json.JsonTestHelper.AssertContents(String expectedValue, ArrayBufferWriter`1 buffer, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Utf8JsonWriterTests.cs(3202,0): at System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(Boolean formatted, Boolean skipValidation)
    System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(formatted: False, skipValidation: False, keyString: ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>") [FAIL]
      Assert.Equal() Failure
                                          ↓ (pos 484474)
      Expected: ···e\u003e\u003e\u003e\w003e\u003e\u003e\u003e\u003e\u003e\u003e···
      Actual:   ···e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e···
                                          ↑ (pos 484474)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(738,0): at System.Text.Json.JsonTestHelper.AssertContentsAgainstJsonNet(String expectedValue, String value, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(714,0): at System.Text.Json.JsonTestHelper.AssertContents(String expectedValue, ArrayBufferWriter`1 buffer, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Utf8JsonWriterTests.cs(5617,0): at System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(Boolean formatted, Boolean skipValidation, String keyString)
    System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync [FAIL]
      Assert.Equal() Failure
                                          ↓ (pos 12025)
      Expected: ···eeeeeeeeeeeeeeeeeeeegeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee···
      Actual:   ···eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee···
                                          ↑ (pos 12025)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(61,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(20,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(String expected, String actual)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(102,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TElement](Stream memoryStream, String expectedJson, Type type, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(71,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TElement](Object obj, Type type, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(53,0): at System.Text.Json.Serialization.Tests.StreamTests.RunTestAsync[TElement]()
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(25,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync()
        --- End of stack trace from previous location ---
  Finished:    System.Text.Json.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.Json.Tests  Total: 20247, Errors: 0, Failed: 3, Skipped: 0, Time: 58.695s
  • System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync
  • System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(formatted: False, skipValidation: False)
  • System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(formatted: False, skipValidation: False, keyString: ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>")

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers",
  "BuildRetry": false
}

Report

Build Definition Test Pull Request
646735 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100141
646752 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101172
646629 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100685
646704 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101117
646679 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101178
646615 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100094
646642 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100824
646471 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101171
646414 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101168
646500 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101140
646478 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101133
646303 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101106
646425 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101053
646408 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100951
646378 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101122
646397 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
646351 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101162
646235 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101161
646220 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101158
646197 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101037
646202 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101156
646116 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101141
645817 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100966
646046 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101137
645902 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101143
645931 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100946
645895 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100697
645856 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101141
645850 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101140
645791 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101138
645825 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101037
645838 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
645804 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101139
645757 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101133
645807 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #97041
645780 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101137
645795 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100951
645684 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101120
645607 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101087
645580 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101095
645577 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101129
645573 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100946
645473 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101122
645505 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100966
645489 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #101123
645396 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100141
645217 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101106
645199 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101110
645086 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100954
645056 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100949
645041 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100951
645012 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution
644951 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #94970
644913 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101101
644773 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #98838
644550 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101017
644343 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100898
644301 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101078
644318 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution
644104 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #101017
644152 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100587
644038 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100094
643951 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100765
642449 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution
642419 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100664
642197 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100765
642193 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101009
642160 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101008
642028 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution
641917 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #101002
641896 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100821
641548 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
641471 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100664
641482 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100938
641521 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution
641439 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100989
641128 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
641052 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100266
640998 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100890
640958 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100765
640906 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100520
640860 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100954
640857 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100954
640844 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100951
640748 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #99199
640730 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100729
640726 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100669
640634 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100743
640505 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
640242 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100812
639773 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100890
639582 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100890
639339 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100890
639263 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100907
638748 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100497
638373 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #100890
638196 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100881
638143 dotnet/runtime WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution #100877
637953 dotnet/runtime normal-System.Text.Json.Tests.WorkItemExecution #100812
633731 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #99278
Displaying 100 of 132 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
45 99 132

Known issue validation

Build: 🔎
Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.
@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Text.Json untriaged New issue has not been triaged by the area owner labels Sep 27, 2021
@ghost
Copy link

ghost commented Sep 27, 2021

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

https://dev.azure.com/dnceng/public/_build/results?buildId=1387876&view=ms.vss-test-web.build-test-results-tab&runId=40401306&resultId=194463&paneView=debug

If I didn't know any better, I'd say "memory corruption".

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-59626-merge-ff748885001649f2ad/System.Text.Json.Tests/1/console.55b96e63.log?sv=2019-07-07&se=2021-10-17T15%3A49%3A24Z&sr=c&sp=rl&sig=jkQ9vlfitTdm9LAJkyCb4ZnZxrI5Gb%2F8qvt%2FSn0S27M%3D

Console log: 'System.Text.Json.Tests' from job ff748885-0016-49f2-ad26-20905cd95f3d workitem 0045d34b-f5a3-424a-8789-b58e57023e81 (osx.1015.amd64.open) executed on machine dci-mac-build-195
+ ./RunTests.sh --runtime-path /tmp/helix/working/A15608F4/p
----- start Mon Sep 27 11:49:52 EDT 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/A15608F4/p/dotnet exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/A15608F4/w/A1BD08B8/e /private/tmp/helix/working/A15608F4/w/A1BD08B8/e
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 2776 of 2815 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 12)
    System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(formatted: False, skipValidation: False) [FAIL]
      Assert.Equal() Failure
                                          ↓ (pos 1102164)
      Expected: ···qKioqKioqKioqKioqKiosKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq···
      Actual:   ···qKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq···
                                          ↑ (pos 1102164)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(738,0): at System.Text.Json.JsonTestHelper.AssertContentsAgainstJsonNet(String expectedValue, String value, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(714,0): at System.Text.Json.JsonTestHelper.AssertContents(String expectedValue, ArrayBufferWriter`1 buffer, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Utf8JsonWriterTests.cs(3202,0): at System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(Boolean formatted, Boolean skipValidation)
    System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(formatted: False, skipValidation: False, keyString: ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>") [FAIL]
      Assert.Equal() Failure
                                          ↓ (pos 484474)
      Expected: ···e\u003e\u003e\u003e\w003e\u003e\u003e\u003e\u003e\u003e\u003e···
      Actual:   ···e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e···
                                          ↑ (pos 484474)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(738,0): at System.Text.Json.JsonTestHelper.AssertContentsAgainstJsonNet(String expectedValue, String value, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(714,0): at System.Text.Json.JsonTestHelper.AssertContents(String expectedValue, ArrayBufferWriter`1 buffer, Boolean skipSpecialRules)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Utf8JsonWriterTests.cs(5617,0): at System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(Boolean formatted, Boolean skipValidation, String keyString)
    System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync [FAIL]
      Assert.Equal() Failure
                                          ↓ (pos 12025)
      Expected: ···eeeeeeeeeeeeeeeeeeeegeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee···
      Actual:   ···eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee···
                                          ↑ (pos 12025)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(61,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(20,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(String expected, String actual)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(102,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TElement](Stream memoryStream, String expectedJson, Type type, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(71,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TElement](Object obj, Type type, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(53,0): at System.Text.Json.Serialization.Tests.StreamTests.RunTestAsync[TElement]()
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(25,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync()
        --- End of stack trace from previous location ---
  Finished:    System.Text.Json.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.Json.Tests  Total: 20247, Errors: 0, Failed: 3, Skipped: 0, Time: 58.695s
  • System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync
  • System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(formatted: False, skipValidation: False)
  • System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(formatted: False, skipValidation: False, keyString: ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>")
Author: bartonjs
Assignees: -
Labels:

area-System.Text.Json, untriaged
Milestone: -

@bartonjs bartonjs mentioned this issue Sep 27, 2021
Merged
@bartonjs
Copy link
Member Author

Interesting observation: In all three failures the expected character vs the actual char had a difference of 2. (q => s, u => w, e => g).

Maybe it really is some sort of memory corruption (something has a bad pointer somewhere, and added 2).

Secondary observation: They all feel like they have expected and actual backwards 😄.

@eiriktsarpalis eiriktsarpalis added this to the 7.0.0 milestone Sep 28, 2021
@eiriktsarpalis
Copy link
Member

Seems related to #45464

@eiriktsarpalis eiriktsarpalis added os-mac-os-x macOS aka OSX and removed untriaged New issue has not been triaged by the area owner labels Sep 28, 2021
@danmoseley danmoseley mentioned this issue Dec 8, 2021
Merged
@eiriktsarpalis eiriktsarpalis mentioned this issue Feb 14, 2022
Closed
@elinor-fung
Copy link
Member

The runtime-extra-platforms rolling build also hit the failure in Writing3MBBase64Bytes (on Windows Server 2022): https://dev.azure.com/dnceng/public/_build/results?buildId=1612202&view=ms.vss-test-web.build-test-results-tab&runId=44896660&resultId=189854&paneView=debug

@joperezr
Copy link
Member

joperezr commented Apr 4, 2022

Not sure if this is exactly related, but build https://dev.azure.com/dnceng/public/_build/results?buildId=1697303&view=logs&j=05e92ac1-194e-59cf-664a-fa72d1cdd19b&t=caea1d4b-c90c-5be1-e57d-c4635079c333 failed with HandleCollectionsAsync test failing in a very similar way. Here are the logs: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-50e485d6cc7b461f9a/System.Text.Json.Tests/1/console.6fad303f.log?sv=2019-07-07&se=2022-04-23T08%3A48%3A09Z&sr=c&sp=rl&sig=SNdOcc025qirLoyBCvPN7U%2BHl1uIgU94Zo64yUp0CLk%3D

@joperezr joperezr added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Apr 4, 2022
@runfoapp runfoapp bot mentioned this issue Apr 4, 2022
Closed
@danmoseley
Copy link
Member

In @bartonjs case, the second least significant bit is flipping 0->1 in every case. If it was a bad pointer I think we'd expect 1->0 also, but it's a sample of three.

In @joperezr case (again as @bartonjs points out, Expected and Actual are reversed - cc @eiriktsarpalis ) it is quite different, several bits flipped, and it's then repeated.

      Assert.Equal() Failure
                                       ↓ (pos 13390)
      Expected: ···xxxxxxxxxxxxxxxxxxxxooooooooooooooooooooooooooooooooooooooooo···
      Actual:   ···xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx···
                                       ↑ (pos 13390)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(61,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(55,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(20,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(String expected, String actual)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(87,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TCollection,TElement](String json, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(50,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TCollection,TElement](TCollection collection, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(30,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync[TCollection,TElement](TCollection collection, Int32 bufferSize, Witness`1 elementType)
        --- End of stack trace from previous location ---
    System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync<Collection<StreamTests+ImmutableStructWithStrings>, StreamTests+ImmutableStructWithStrings>(collection: [System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, ...], bufferSize: 16384, elementType: Witness`1 { }) [FAIL]
      Assert.Equal() Failure
                                       ↓ (pos 8986)
      Expected: ···ppppppppppppppppppppwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww···
      Actual:   ···ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp···
                                       ↑ (pos 8986)
      Stack Trace:
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(61,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)

It's important to record machine names in such cases. Last one above was dci-mac-build-365.local. Looks like we lost the other two? We have in the past send a box to the trash because it was regularly flipping 1 bit.

@danmoseley
Copy link
Member

Oh, @bartonjs case was dci-mac-build-195.

@carlossanlop
Copy link
Member

Might be related to this other issue? #65021

@carlossanlop
Copy link
Member

In the same CI job that Joe shared, there's another log file with a slightly different callstack, in case it's helpful:

Pipeline: Libraries Test Run release coreclr OSX x64 Release
Test method:

System.Text.Json.Serialization.Tests.StreamTests_Async.HandleCollectionsAsync
<ConcurrentStack<StreamTests+ImmutableStructWithStrings>, StreamTests+ImmutableStructWithStrings>
(collection: [System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings,
System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings,
System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings,
System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings,
System.Text.Json.Serialization.Tests.StreamTests+ImmutableStructWithStrings, ...],
bufferSize: 16384, elementType: Witness`1 { })
[FAIL]

Callstack:

      System.Text.Json.JsonException : '0x00' is invalid within a JSON string. The string should be correctly escaped. Path: $[14].MySecondString | LineNumber: 0 | BytePositionInLine: 439315.
      ---- System.Text.Json.JsonReaderException : '0x00' is invalid within a JSON string. The string should be correctly escaped. LineNumber: 0 | BytePositionInLine: 439315.
      Stack Trace:
        /_/src/libraries/System.Text.Json/src/System/Text/Json/ThrowHelper.Serialization.cs(265,0): at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.ReadCore.cs(79,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Helpers.cs(43,0): at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.String.cs(324,0): at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.String.cs(48,0): at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(75,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TCollection,TElement](String json, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(50,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TCollection,TElement](TCollection collection, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(30,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync[TCollection,TElement](TCollection collection, Int32 bufferSize, Witness`1 elementType)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----
        /_/src/libraries/System.Text.Json/src/System/Text/Json/ThrowHelper.cs(279,0): at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1365,0): at System.Text.Json.Utf8JsonReader.ConsumeStringAndValidate(ReadOnlySpan`1 data, Int32 idx)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1271,0): at System.Text.Json.Utf8JsonReader.ConsumeString()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1032,0): at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(800,0): at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(278,0): at System.Text.Json.Utf8JsonReader.Read()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Object/ObjectWithParameterizedConstructorConverter.cs(40,0): at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs(221,0): at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Collection/JsonCollectionConverter.cs(116,0): at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs(246,0): at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.ReadCore.cs(62,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)

@carlossanlop
Copy link
Member

@dotnet/area-system-text-json Happened once again in today's morning rolling build: https://dev.azure.com/dnceng/public/_build/results?buildId=1700046&view=results

 System.Text.Json.JsonReaderException : '0x00' is invalid within a JSON string. The string should be correctly escaped. LineNumber: 0 | BytePositionInLine: 392105.
      Stack Trace:
        /_/src/libraries/System.Text.Json/src/System/Text/Json/ThrowHelper.cs(279,0): at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1365,0): at System.Text.Json.Utf8JsonReader.ConsumeStringAndValidate(ReadOnlySpan`1 data, Int32 idx)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1293,0): at System.Text.Json.Utf8JsonReader.ConsumeString()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1032,0): at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(800,0): at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(278,0): at System.Text.Json.Utf8JsonReader.Read()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.cs(1077,0): at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, JsonReaderOptions readerOptions, MetadataDb& database, StackRowStack& stack)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.Parse.cs(689,0): at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedArrayPoolBytes, PooledByteBufferWriter extraPooledByteBufferWriter)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Document/JsonDocument.Parse.cs(261,0): at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 json, JsonDocumentOptions options)
        /_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(19,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(String expected, String actual)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(87,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TCollection,TElement](String json, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(54,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TCollection,TElement](TCollection collection, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(33,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync[TCollection,TElement](TCollection collection, Int32 bufferSize, Witness`1 elementType)

@danmoseley
Copy link
Member

Logging machine name from above instance:
Console log: 'System.Text.Json.Tests' from job dbec34c9-6493-4fa7-a9d3-a219dadba558 workitem 351aad77-de0d-4213-bf7f-90d96b9c35c1 (osx.1200.amd64.open) executed on machine dci-mac-build-387.local

so no evidence this is hardware specific.

@carlossanlop
Copy link
Member

@eiriktsarpalis @steveharter @layomia can we get the test disabled or a fix submitted? This happened again in the runtime-extra-platforms rolling build from this morning and is blocking a clean CI.

 System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync<ImmutableStack<String>, String>(collection: ["vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv"..., "oooooooooooooooooooooooooooooooooooooooooooooooooo"..., "ssssssssssssssssssssssssssssssssssssssssssssssssss"..., "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn"..., "wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww"..., ...], bufferSize: 16384, elementType: Witness`1 { }) [FAIL]
      System.Text.Json.JsonException : '0x00' is invalid within a JSON string. The string should be correctly escaped. Path: $[14] | LineNumber: 0 | BytePositionInLine: 216537.
      ---- System.Text.Json.JsonReaderException : '0x00' is invalid within a JSON string. The string should be correctly escaped. LineNumber: 0 | BytePositionInLine: 216537.
      Stack Trace:
        /_/src/libraries/System.Text.Json/src/System/Text/Json/ThrowHelper.Serialization.cs(265,0): at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.ReadCore.cs(79,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Helpers.cs(43,0): at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.String.cs(324,0): at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.String.cs(48,0): at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
           at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TCollection,TElement](String json, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(54,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TCollection,TElement](TCollection collection, JsonSerializerOptions options)
        /_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(33,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync[TCollection,TElement](TCollection collection, Int32 bufferSize, Witness`1 elementType)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----
        /_/src/libraries/System.Text.Json/src/System/Text/Json/ThrowHelper.cs(279,0): at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1365,0): at System.Text.Json.Utf8JsonReader.ConsumeStringAndValidate(ReadOnlySpan`1 data, Int32 idx)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1293,0): at System.Text.Json.Utf8JsonReader.ConsumeString()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1036,0): at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1801,0): at System.Text.Json.Utf8JsonReader.ConsumeNextToken(Byte marker)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(1689,0): at System.Text.Json.Utf8JsonReader.ConsumeNextTokenOrRollback(Byte marker)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(884,0): at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Reader/Utf8JsonReader.cs(272,0): at System.Text.Json.Utf8JsonReader.Read()
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Converters/Collection/JsonCollectionConverter.cs(198,0): at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs(221,0): at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
        /_/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.ReadCore.cs(62,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)

@eiriktsarpalis
Copy link
Member

FWIW this seems like a different failure compared to the original report. Is this impacting osx x64 builds only? Should we only disable the test for that platform?

@eiriktsarpalis eiriktsarpalis mentioned this issue Apr 6, 2022
Merged
@krwq
Copy link
Member

krwq commented Apr 6, 2022 •
edited
Loading

I'll add couple of other tests I've seen locally, first 3 look like the "not found" value is actually in the string and in all 3 cases it's using current culture for comparison:

    System.Text.Json.SourceGeneration.Tests.SerializationContextTests.SerializeObjectArray_WithCustomOptions [FAIL]
      Assert.Contains() Failure
      Not found: organizationName
      In value:  [{"activeOrUpcomingEvents":[{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"},{"id":10,"imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","name":"Just a name","campaignName":"The very new campaign","campaignManagedOrganizerName":"Name FamilyName","description":"The .NET Foundation works with Microsoft and the broader industry to increase the exposure of open source projects in the .NET community and the .NET Foundation. The .NET Foundation provides access to these resources to projects and looks to promote the activities of our communities.","startDate":"2001-02-03T04:05:06.789+00:00","endDate":"2002-02-03T04:05:06.789+00:00","offset":"00:00:00"}],"featuredCampaign":{"id":234235,"title":"Promoting Open Source","description":"Very nice campaign","imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","organizationName":"The Company XYZ","headline":"The Headline"},"isNewAccount":false,"hasFeaturedCampaign":true},{"id":234235,"title":"Promoting Open Source","description":"Very nice campaign","imageUrl":"https://www.dotnetfoundation.org/theme/img/carousel/foundation-diagram-content.png","organizationName":"The Company XYZ","headline":"The Headline"}]
      Stack Trace:
        D:\src\runtime\src\libraries\System.Text.Json\tests\System.Text.Json.SourceGeneration.Tests\SerializationContextTests.cs(339,0): at System.Text.Json.SourceGeneration.Tests.SerializationContextTests.SerializeObjectArray_WithCustomOptions()
  Finished:    System.Text.Json.SourceGeneration.Roslyn4.0.Tests

    System.Text.Json.Serialization.Tests.PolymorphicTests_Element.SimpleTestClassAsRootObject [FAIL]
      Assert.Contains() Failure
      Not found: "MyBooleanTrue":true
      In value:  {"MyInt16Array":[1],"MyInt32Array":[2],"MyInt64Array":[3],"MyUInt16Array":[4],"MyUInt32Array":[5],"MyUInt64Array":[6],"MyByteArray":"Bw==","MySByteArray":[8],"MyCharArray":["a"],"MyStringArray":["Hello"],"MyDecimalArray":[3.3],"MyBooleanTrueArray":[true],"MyBooleanFalseArray":[false],"MySingleArray":[1.1],"MyDoubleArray":[2.2],"MyDateTimeArray":["2019-01-30T12:01:02Z"],"MyEnumArray":[2],"MyStringList":["Hello"],"MyStringIEnumerable":["Hello"],"MyStringIList":["Hello"],"MyStringICollection":["Hello"],"MyStringIEnumerableT":["Hello"],"MyStringIListT":["Hello"],"MyStringICollectionT":["Hello"],"MyStringIReadOnlyCollectionT":["Hello"],"MyStringIReadOnlyListT":["Hello"],"MyStringISetT":["Hello"],"MyStringToStringKeyValuePair":{"Key":"myKey","Value":"myValue"},"MyStringToStringIDict":{"key":"value"},"MyStringToStringGenericDict":{"key":"value"},"MyStringToStringGenericIDict":{"key":"value"},"MyStringToStringGenericIReadOnlyDict":{"key":"value"},"MyStringToStringImmutableDict":{"key":"value"},"MyStringToStringIImmutableDict":{"key":"value"},"MyStringToStringImmutableSortedDict":{"key":"value"},"MyStringStackT":["World","Hello"],"MyStringQueueT":["Hello","World"],"MyStringHashSetT":["Hello"],"MyStringLinkedListT":["Hello"],"MyStringSortedSetT":["Hello"],"MyStringIImmutableListT":["Hello"],"MyStringIImmutableStackT":["Hello"],"MyStringIImmutableQueueT":["Hello"],"MyStringIImmutableSetT":["Hello"],"MyStringImmutableHashSetT":["Hello"],"MyStringImmutableArray":null,"MyStringImmutableListT":["Hello"],"MyStringImmutableStackT":["Hello"],"MyStringImmutablQueueT":["Hello"],"MyStringImmutableSortedSetT":["Hello"],"MyInt16":1,"MyInt32":2,"MyInt64":3,"MyUInt16":4,"MyUInt32":5,"MyUInt64":6,"MyByte":7,"MySByte":8,"MyChar":"a","MyString":"Hello","MyDecimal":3.3,"MyBooleanTrue":true,"MyBooleanFalse":false,"MySingle":1.1,"MyDouble":2.2,"MyDateTime":"2019-01-30T12:01:02Z","MyGuid":"5bb9d872-da8a-471e-aa70-08e19102683d","MyEnum":2,"MyStruct":{"One":1,"Two":3.14}}
      Stack Trace:
        D:\src\runtime\src\libraries\System.Text.Json\tests\System.Text.Json.Tests\Serialization\PolymorphicTests.cs(342,0): at System.Text.Json.Serialization.Tests.PolymorphicTests.SimpleTestClassAsRootObject()

    System.Text.Json.Serialization.Tests.PolymorphicTests_Element.NestedObjectAsRootObject [FAIL]
      Assert.Contains() Failure
      Not found: "ImmutableQueueT":["Hello","World"]
      In value:  {"Address":{"City":"MyCity"},"List":["Hello","World"],"Array":["Hello","Again"],"NullableInt":42,"NullableIntArray":[null,42,null],"IEnumerable":["Hello","World"],"IList":["Hello","World"],"ICollection":["Hello","World"],"IEnumerableT":["Hello","World"],"IListT":["Hello","World"],"ICollectionT":["Hello","World"],"IReadOnlyCollectionT":["Hello","World"],"IReadOnlyListT":["Hello","World"],"ISetT":["Hello","World"],"StackT":["World","Hello"],"QueueT":["Hello","World"],"HashSetT":["Hello","World"],"LinkedListT":["Hello","World"],"SortedSetT":["Hello","World"],"ImmutableArrayT":["Hello","World"],"IImmutableListT":["Hello","World"],"IImmutableStackT":["World","Hello"],"IImmutableQueueT":["Hello","World"],"IImmutableSetT":["Hello","World"],"ImmutableHashSetT":["Hello","World"],"ImmutableListT":["Hello","World"],"ImmutableStackT":["World","Hello"],"ImmutableQueueT":["Hello","World"],"ImmutableSortedSetT":["Hello","World"],"Object":{}}
      Stack Trace:
        D:\src\runtime\src\libraries\System.Text.Json\tests\System.Text.Json.Tests\Serialization\PolymorphicTests.cs(376,0): at System.Text.Json.Serialization.Tests.PolymorphicTests.<NestedObjectAsRootObject>g__Verify|9_0(String json)
        D:\src\runtime\src\libraries\System.Text.Json\tests\System.Text.Json.Tests\Serialization\PolymorphicTests.cs(393,0): at System.Text.Json.Serialization.Tests.PolymorphicTests.NestedObjectAsRootObject()

also in another run I've seen those 2 happen in a single run which was rather unusual and I think the stream was actually corrupted (somehow ended up with first byte being zero) - interestingly this only affected async related cases and same test using non-async code path was fine:

System.Text.Json.Serialization.Tests.CollectionTestsDynamic_AsyncStreamWithSmallBuffer.ReadAsyncEnumerableOfAsyncEnumerables [FAIL]
      System.Text.Json.JsonException : '0x00' is an invalid start of a value. Path: $ | LineNumber: 0 | BytePositionInLine: 0.
      ---- System.Text.Json.JsonReaderException : '0x00' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
      Stack Trace:
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\ThrowHelper.Serialization.cs(265,0): at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonConverterOfT.ReadCore.cs(79,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Helpers.cs(17,0): at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Stream.cs(574,0): at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Stream.cs(502,0): at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Stream.cs(358,0): at System.Text.Json.JsonSerializer.ReadAllAsync[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
        D:\src\runtime\src\libraries\System.Text.Json\tests\System.Text.Json.Tests\Serialization\JsonSerializerWrapper.Reflection.cs(157,0): at System.Text.Json.Serialization.Tests.JsonSerializerWrapper.AsyncStreamSerializerWrapper.DeserializeWrapper[T](Stream utf8Json, JsonSerializerOptions options)
        D:\src\runtime\src\libraries\System.Text.Json\tests\Common\CollectionTests\CollectionTests.AsyncEnumerable.cs(283,0): at System.Text.Json.Serialization.Tests.CollectionTests.ReadAsyncEnumerableOfAsyncEnumerables()
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\ThrowHelper.cs(279,0): at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(1111,0): at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(979,0): at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(894,0): at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(272,0): at System.Text.Json.Utf8JsonReader.Read()
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonConverterOfT.ReadCore.cs(27,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
    System.Text.Json.Serialization.Tests.CollectionTestsDynamic_AsyncStream.ReadAsyncEnumerableOfAsyncEnumerables [FAIL]
      System.Text.Json.JsonException : '0x00' is an invalid start of a value. Path: $ | LineNumber: 0 | BytePositionInLine: 0.
      ---- System.Text.Json.JsonReaderException : '0x00' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
      Stack Trace:
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\ThrowHelper.Serialization.cs(265,0): at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonConverterOfT.ReadCore.cs(79,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Helpers.cs(17,0): at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Stream.cs(574,0): at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Stream.cs(502,0): at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Stream.cs(358,0): at System.Text.Json.JsonSerializer.ReadAllAsync[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
        D:\src\runtime\src\libraries\System.Text.Json\tests\System.Text.Json.Tests\Serialization\JsonSerializerWrapper.Reflection.cs(157,0): at System.Text.Json.Serialization.Tests.JsonSerializerWrapper.AsyncStreamSerializerWrapper.DeserializeWrapper[T](Stream utf8Json, JsonSerializerOptions options)
        D:\src\runtime\src\libraries\System.Text.Json\tests\Common\CollectionTests\CollectionTests.AsyncEnumerable.cs(283,0): at System.Text.Json.Serialization.Tests.CollectionTests.ReadAsyncEnumerableOfAsyncEnumerables()
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace -----
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\ThrowHelper.cs(279,0): at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(1111,0): at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(979,0): at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(894,0): at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(272,0): at System.Text.Json.Utf8JsonReader.Read()
        D:\src\runtime\src\libraries\System.Text.Json\src\System\Text\Json\Serialization\JsonConverterOfT.ReadCore.cs(27,0): at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
  Finished:    System.Text.Json.Tests

I'll be replacing Assert.Contains in our tests into something else which can possibly provide us some more details

@danmoseley
Copy link
Member

danmoseley commented Apr 6, 2022 •
edited
Loading

be replacing Assert.Contains in our tests into something else which can possibly provide us some more details

@krwq perhaps you can put it in a shared place - AssertExtensions, we already did something very similar for Equals

https://github.com/dotnet/runtime/pull/67586/files#diff-9f9dc769fe4cfd311f1e57c7572eaeeb09c0c72ffe633131ed0fd8b3ed024994R505

If we build up several of these improved assertions and they show value, we could try to see whether xunit would be interested in any changes.

@krwq
Copy link
Member

krwq commented Apr 7, 2022

FWIW the AssertExtensions.Contains already exists and does exactly what I planned to add (which is doing Ordinal comparison by default). I'm currently running tests locally to see if StringComparison.CurrentCulture vs Ordinal make any difference (I want to make sure the failures above are not related to some globalization bug). I'm basically doing CurrentCulture comparison and rather than throwing on "not found" I'm also comparing using Ordinal and if I see any mismatch that will suggest to me this is likely some kind of globalization initialization error. If both checks fail it's likely memory corruption. I'll update once I get a repro (although it might take a day or two depending on the luck)

@krwq
Copy link
Member

krwq commented Apr 8, 2022

Ok, finally got lucky - random Assert.Contains failed where actualString.IndexOf(expectedSubstring, StringComparison.CurrentCulture) returned -1 but actualString.IndexOf(expectedSubstring, StringComparison.Ordinal) returned non-negative value.

IMO some issues we're seeing are some kind of globalization issue - maybe cultures didn't get initialized correctly when it happens on multiple threads. We'll likely need to try to write some standalone repro and try to attach debugger to see if we can see what exactly happened and then maybe we can guess what should be changed.

From JSON perspective we likely should change to ordinal compare anyway but it would be good to first nail down what the problem with CurrentCulture is

@krwq krwq mentioned this issue Apr 8, 2022
Closed
@krwq
Copy link
Member

krwq commented Apr 8, 2022

I'll locally do Ordinal compare for now to flush out other flaky issues and meantime I'm checking with @tarekgh if he's seen anything similar happening elsewhere...

@jeffhandley jeffhandley modified the milestones: 7.0.0, Future Aug 9, 2022
@eiriktsarpalis eiriktsarpalis mentioned this issue Dec 12, 2022
Closed
@eiriktsarpalis eiriktsarpalis added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Dec 12, 2022
@jkotas jkotas added the Known Build Error Use this to report build issues in the .NET Helix tab label Dec 12, 2022
@build-analysis build-analysis bot removed this from the Future milestone Nov 15, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 15, 2023
@eiriktsarpalis eiriktsarpalis added this to the Future milestone Nov 15, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Nov 15, 2023
@build-analysis build-analysis bot removed this from the Future milestone Nov 15, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 15, 2023
@missymessa missymessa mentioned this issue Nov 16, 2023
Closed
3 tasks
@eiriktsarpalis eiriktsarpalis added this to the Future milestone Nov 16, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Nov 16, 2023
@jeffhandley jeffhandley modified the milestones: Future, 9.0.0 Nov 17, 2023
@jeffhandley
Copy link
Member

Even if this has morphed from the original issue, we should take a look at this during 9.0.0 since it causes CI failures.

@lewing lewing mentioned this issue Apr 13, 2024
Open
@lewing
Copy link
Member

lewing commented Apr 13, 2024

I think there are other better checks that are more specific that are catching the things being caught there now like #100111 and #98578 so I'm closing this

@lewing lewing closed this as completed Apr 13, 2024
@lewing lewing reopened this Apr 16, 2024
@jeffschwMSFT
Copy link
Member

@lewing / @bartonjs this is our top reported known issue right now, I think that the search text is too board.

System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers

To Larry's point above, is there a more specific search text for the failure that this issue is tracking?

@lewing
Copy link
Member

lewing commented Apr 17, 2024 •
edited
Loading

Yeah I'm closing it again and I'll make mroe specific tests if any aren't caught by the others

@lewing lewing closed this as completed Apr 17, 2024
@lewing
Copy link
Member

lewing commented Apr 17, 2024 •
edited
Loading

the problem is there are failures of this specific test and the failures look like:

/datadisks/disk1/work/9B2B0894/w/B43409F0/e /datadisks/disk1/work/9B2B0894/w/B43409F0/e
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 7405 of 7466 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on [2 threads], stop on fail = off)
    System.Text.Json.Tests.Utf8JsonReaderTests.TestDepth(depth: 256) [SKIP]
      Takes very long to run on interpreter.
    System.Text.Json.Tests.Utf8JsonReaderTests.TestDepth(depth: 512) [SKIP]
      Takes very long to run on interpreter.
   System.Text.Json.Tests: [Long Running Test] 'System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes', Elapsed: 00:05:42
   System.Text.Json.Tests: [Long Running Test] 'System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers', Elapsed: 00:02:01
   System.Text.Json.Tests: [Long Running Test] 'System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers', Elapsed: 00:04:01
Killed
['System.Text.Json.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-101117-merge-608176b4786749e2b4/System.Text.Json.Tests/1/console.6c31de45.log?helixlogtype=result
which isn't easy to distinguish with build analysis

@lewing
Copy link
Member

lewing commented Apr 17, 2024

we should probably just skip the test on the interpreter

@lewing
Copy link
Member

lewing commented Apr 17, 2024

opened #101193

@lewing lewing mentioned this issue Apr 17, 2024
Open
@lewing
Copy link
Member

lewing commented Apr 17, 2024 •
edited
Loading

It is looking increasingly like the ooms were indeed only part of the problem and now we're hitting a bunch of different ways to timeout in the interpreter on desktop and wasm that this catches

@github-actions github-actions bot locked and limited conversation to collaborators May 18, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Text.Json blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-mac-os-x macOS aka OSX test-failure
Projects
None yet
Milestone
9.0.0
Development

No branches or pull requests
14 participants
@lewing @agocke @krwq @jeffhandley @carlossanlop @eiriktsarpalis @danmoseley @jkotas @jeffschwMSFT @bartonjs @tarekgh @joperezr @elinor-fung and others