[browser][MT] RuntimeError: memory access out of bounds #96546

pavelsavara · 2024-01-05T12:55:04Z

Error Blob

{
  "ErrorMessage": "RuntimeError: memory access out of bounds",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167
Log

[20:26:27] warn: worker sent an error! http://127.0.0.1:39745/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[20:26:27] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:39745/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:39745/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:39745/_framework/dotnet.js:3:33874
[20:26:27] warn: MONO_WASM: mono_wasm_exit failed unwind

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167
Error message validated: RuntimeError: memory access out of bounds
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/5/2024 12:55:21 PM UTC

Report

Build	Definition	Test	Pull Request
678721	dotnet/runtime	WasmTestOnChrome-System.Net.Http.Functional.Tests.WorkItemExecution
678427	dotnet/runtime	WasmTestOnChrome-System.Security.Cryptography.Tests.WorkItemExecution	#102351
671274	dotnet/runtime	WasmTestOnChrome-MT-System.Dynamic.Runtime.Tests.WorkItemExecution	#101938
666657	dotnet/runtime	WasmTestOnChrome-MT-System.Runtime.InteropServices.Tests.WorkItemExecution	#101717
661888	dotnet/runtime	WasmTestOnChrome-MT-System.Runtime.InteropServices.Tests.WorkItemExecution	#101716
661068	dotnet/runtime	WasmTestOnChrome-MT-System.Data.Common.Tests.WorkItemExecution	#101712
659851	dotnet/runtime	WasmTestOnChrome-MT-System.Buffers.Tests.WorkItemExecution
659061	dotnet/runtime	WasmTestOnChrome-MT-System.Runtime.InteropServices.Tests.WorkItemExecution
658853	dotnet/runtime	WasmTestOnChrome-MT-System.Linq.Expressions.Tests.WorkItemExecution
658682	dotnet/runtime	WasmTestOnChrome-MT-System.Runtime.InteropServices.Tests.WorkItemExecution	#101330
655218	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101318
655451	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution
653807	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101442
654744	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101330
654556	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101478
654532	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#100775
653837	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101445
653677	dotnet/runtime	WasmTestOnChrome-MT-System.Text.Json.Tests.WorkItemExecution	#100697
653288	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101329
653246	dotnet/runtime	WasmTestOnChrome-System.Net.Http.Functional.Tests.WorkItemExecution	#101408
653187	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101397
653143	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101342
652989	dotnet/runtime	WasmTestOnChrome-System.Runtime.InteropServices.JavaScript.Tests.WorkItemExecution	#101336
652546	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101312
652321	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101390
652318	dotnet/runtime	WasmTestOnChrome-System.Text.Json.Tests.WorkItemExecution	#101392
652294	dotnet/runtime	WasmTestOnV8-System.Text.Json.Tests.WorkItemExecution	#101391
651705	dotnet/runtime	WasmTestOnChrome-System.Net.Http.Functional.Tests.WorkItemExecution	#101133
651501	dotnet/runtime	WasmTestOnChrome-System.Data.Common.Tests.WorkItemExecution	#101106

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	2	29

The text was updated successfully, but these errors were encountered:

ghost · 2024-01-05T12:55:09Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "ErrorMessage": "RuntimeError: memory access out of bounds",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167
Log

[20:26:27] warn: worker sent an error! http://127.0.0.1:39745/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[20:26:27] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:39745/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:39745/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:39745/_framework/dotnet.js:3:33874
[20:26:27] warn: MONO_WASM: mono_wasm_exit failed unwind```

<table>
  <tr>
    <th align="left">Author:</th>
    <td>pavelsavara</td>
  </tr>
  <tr>
    <th align="left">Assignees:</th>
    <td>pavelsavara</td>
  </tr>
  <tr>
    <th align="left">Labels:</th>
    <td>

`arch-wasm`, `blocking-clean-ci`, `area-VM-threading-mono`, `Known Build Error`, `os-browser`

</td>
  </tr>
  <tr>
    <th align="left">Milestone:</th>
    <td>9.0.0</td>
  </tr>
</table>
</details>

pavelsavara · 2024-01-11T10:35:00Z

Another
Build
Log

[23:11:59] info: [2024-01-10T23:11:59.397Z] [PASS] System.Threading.ThreadPools.Tests.ThreadPoolTests.RunProcessorCountItemsInParallel
[23:11:59] info: [2024-01-10T23:11:59.400Z] [SKIP] System.Threading.ThreadPools.Tests.ThreadPoolTests.SetMinMaxThreadsTest_ChangedInDotNetCore
[23:12:00] warn: worker sent an error! http://127.0.0.1:33633/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[23:12:00] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:33633/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:33633/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:33633/_framework/dotnet.js:3:33874
[23:12:00] info: WASM EXIT 1
[23:12:00] info: Waiting to flush log messages with a timeout of 120 secs ..

pavelsavara · 2024-02-01T12:54:49Z

Another Log
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=549038&view=logs&jobId=63c2d0c8-fec2-5788-81c8-f3ac95e8841f

[11:44:47] fail: [0x066a4300-long 11:44:47.111] MONO_WASM: Unhandled error: {}
[11:44:47] fail: [0x066a4300-long 11:44:47.117] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[8732]:0x217b29
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2731]:0xd2e2b
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2815]:0xd597a
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2077]:0xaed21
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[324]:0x286c7
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[253]:0x25e62
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[244]:0x18884
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[283]:0x26af0
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[3319]:0xf9db8
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2587]:0xcbe17
[11:44:47] info: WASM EXIT 1

pavelsavara · 2024-02-12T08:25:23Z

Log

[08:55:30] info: Starting:    Microsoft.Extensions.Logging.Console.Tests.dll
[08:55:31] fail: [0x045f6788-emsc 08:55:31.650] MONO_WASM: preRunWorker() failed {}
[08:55:31] fail: [0x045f6788-emsc 08:55:31.652] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[649]:0x4a787
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[650]:0x4a7d6
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[652]:0x4a8ab
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[530]:0x4793a
                     at e.<computed> (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:226093)
                     at http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101299
                     at Ga (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101426)
                     at Ya (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101713)
                     at http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:212904
                     at iu (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:213061)
[08:55:31] info: [0x000dc84c-main 08:55:31.655] MONO_WASM: Dumping web worker info as seen by UI thread, it could be stale:
[08:55:31] info: 00 | 0x000dc84c-main: isRunning:false isAttached: true isEventLoop:false reuseCount:  1 - UI Thread
[08:55:31] info: 01 | 0x02ff8f90-norm: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 - Finalizer
[08:55:31] info: 02 | 0x032a8030-pool: isRunning: true isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[08:55:31] info: 03 | 0x035b0040-gate: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 - .NET TP Gate
[08:55:31] info: 04 | 0x03dbea48-pool: isRunning: true isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[08:55:31] info: 05 | 0x045f6788-emsc: isRunning: true isAttached:false isEventLoop:false reuseCount:  1 - pthread-assigned
[08:55:31] info: 06 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 07 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 08 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 09 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 10 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 11 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 12 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] fail: [0x000dc84c-main 08:55:31.657] MONO_WASM: Uncaught RuntimeError: memory access out of bounds
                 Error
                     at Object.Ge [as mono_exit] (http://127.0.0.1:44985/_framework/dotnet.js:3:20140)
                     at ft.config.exitOnUnhandledError.e.onerror (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:38634)
[08:55:31] fail: [out of order message from the browser]: http://127.0.0.1:44985/_framework/dotnet.native.js 7:1295 Uncaught ErrorEvent: Uncaught RuntimeError: memory access out of bounds

pavelsavara · 2024-02-17T15:15:27Z

During thread creation
Log
Build

[18:39:28] fail: [--06290040-emsc 18:39:28.771] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at pthread_attr_init (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[8640]:0x2144c1)
                     at mono_thread_info_attach (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[786]:0x4ef10)
                     at start_wrapper (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[2762]:0xd6903)
                     at Object.invokeEntryPoint (http://127.0.0.1:34537/_framework/dotnet.native.js:8:69009)
                     at handleMessage (http://127.0.0.1:34537/_framework/dotnet.native.worker.js:1:2306)

pavelsavara · 2024-02-18T15:05:17Z

Related #98630

pavelsavara · 2024-02-20T23:36:51Z

Log

[11:23:26] info: 001 | 0x03078030-pool: isRunning:false isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[11:23:26] fail: [0x03078030-pool 11:23:26.696] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at dlmalloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[8788]:0x21d1e6)
                     at dlcalloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[8796]:0x21f96a)
                     at monoeg_g_calloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[598]:0x49521)
                     at monoeg_malloc0 (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[599]:0x49544)
                     at monoeg_g_array_new (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[569]:0x48b40)
                     at mono_interp_transform_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[438]:0x407b6)
                     at do_transform_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[255]:0x25481)
                     at mono_interp_exec_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[254]:0x251fb)
                     at interp_entry (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[327]:0x28391)
                     at interp_entry_static_2 (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[365]:0x29411)

lewing · 2024-02-21T01:21:18Z

cc @vargaz @steveisok

pavelsavara · 2024-03-10T16:14:14Z

local run with nice stack trace

fail: [0x05a6fef0-jsww 15:29:09.596] MONO_WASM: memory access out of bounds
  RuntimeError: memory access out of bounds
      at dlfree (dlfree (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[17333]:0x34eda4))
      at bound_fn (http://127.0.0.1:61068/_framework/dotnet.runtime.js:4146:24)
      at sync_bound_fn (http://127.0.0.1:61068/_framework/dotnet.runtime.js:3980:13)
      at JSImport_INTERNAL_http_wasm_supports_streaming_request (https://dotnet/JSImport/INTERNAL_http_wasm_supports_streaming_request:4:86)
      at mono_wasm_invoke_jsimport (http://127.0.0.1:61068/_framework/dotnet.runtime.js:3893:5)
      at do_icall (do_icall (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[231]:0x29de6))
      at do_icall_wrapper (do_icall_wrapper (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[151]:0x26971))
      at mono_interp_exec_method (mono_interp_exec_method (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[142]:0x150ac))
      at interp_runtime_invoke (interp_runtime_invoke (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[185]:0x27d50))
      at mono_jit_runtime_invoke (mono_jit_runtime_invoke (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[9174]:0x200a97))

pavelsavara · 2024-03-26T16:34:42Z

another local with great log

MONO_WASM: memory access out of bounds
RuntimeError: memory access out of bounds
    at copy_object_no_checks.1 (copy_object_no_checks.1 (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2325]:0xa15c1))
    at simple_nursery_serial_scan_object (simple_nursery_serial_scan_object (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2321]:0x9f935))
    at simple_nursery_serial_drain_gray_stack (simple_nursery_serial_drain_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2324]:0xa1454))
    at sgen_drain_gray_stack (sgen_drain_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1721]:0x7c5ca))
    at finish_gray_stack (finish_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1828]:0x812be))
    at collect_nursery (collect_nursery (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1744]:0x7d6ee))
    at sgen_perform_collection_inner (sgen_perform_collection_inner (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1741]:0x7d11f))
    at sgen_perform_collection (sgen_perform_collection (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1740]:0x7d001))
    at sgen_ensure_free_space (sgen_ensure_free_space (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1739]:0x7cfac))
    at sgen_alloc_obj_nolock (sgen_alloc_obj_nolock (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1669]:0x7af3e))```

pavelsavara · 2024-03-26T17:22:53Z

another local

RuntimeError: memory access out of bounds
    at dlfree (dlfree (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[17410]:0x34e69e))
    at bound_fn (https://localhost:65453/_framework/dotnet.runtime.js:4217:24)
    at async_bound_fn (https://localhost:65453/_framework/dotnet.runtime.js:4027:9)
    at JSImport_INTERNAL_dynamic_import (https://dotnet/JSImport/INTERNAL_dynamic_import:4:64)
    at mono_wasm_invoke_jsimport_MT (https://localhost:65453/_framework/dotnet.runtime.js:3948:5)
    at do_icall (do_icall (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[227]:0x29b34))
    at do_icall_wrapper (do_icall_wrapper (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[147]:0x266c6))
    at mono_interp_exec_method (mono_interp_exec_method (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[138]:0x150fd))
    at interp_runtime_invoke (interp_runtime_invoke (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[181]:0x27aa5))
    at mono_jit_runtime_invoke (mono_jit_runtime_invoke (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[9240]:0x2001ac))```

pavelsavara · 2024-04-03T06:41:32Z

Most of the problems are fixed by #100304

There is new occurrence as fallout of OOM
Log

[19:33:04] info: [FAIL] System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(options: System.Text.Json.JsonWriterOptions, keyString: "mess><age")
[19:33:04] info: System.OutOfMemoryException : Out of memory
[19:33:04] info:    at System.GC.AllocateArray[Char](Int32 length, Boolean pinned)
[19:33:04] info:    at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
[19:33:04] info:    at System.Text.StringBuilder.AppendWithExpansion(Char value)
[19:33:04] info:    at System.Text.StringBuilder.Append(Char value)
[19:33:04] info:    at System.IO.StringWriter.Write(Char value)
[19:33:04] info:    at Newtonsoft.Json.JsonTextWriter.WriteEnd(JsonToken token)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.AutoCompleteClose(JsonContainerType type)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEndArray()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEnd(JsonContainerType type)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEnd()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.AutoCompleteAll()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.Close()
[19:33:04] info:    at Newtonsoft.Json.JsonTextWriter.Close()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.Dispose(Boolean disposing)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.System.IDisposable.Dispose()
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.GetExpectedLargeArrayOfStrings(Int32 length)
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.WriteJsonOnlyWritesToStreamOnDemand_FlushAsync()
[19:33:04] info: --- End of stack trace from previous location ---
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(JsonWriterOptions options, String keyString)
[19:33:04] info:    at System.Object.InvokeStub_Utf8JsonWriterTests.WriteNumbers(Object , Span`1 )
[19:33:04] info:    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.706] MONO_WASM: Unhandled error: RuntimeError: memory access out of bounds
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.707] MONO_WASM: Unhandled error: RuntimeError: memory access out of bounds
[19:33:04] info: [0x020cdfc0-dpty 19:33:04.710] MONO_WASM: Dumping web worker info as seen by UI thread, it could be stale:
[19:33:04] info: 001 | 0x020cdfc0-dpty: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 -
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.748] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at copy_object_no_checks.1 (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1115]:0x637d7)
                     at simple_nursery_serial_scan_object (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1113]:0x63003)
                     at simple_nursery_serial_drain_gray_stack (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1110]:0x626d8)
                     at finish_gray_stack (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[955]:0x56d6a)
                     at collect_nursery (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[927]:0x5481b)
                     at sgen_perform_collection (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[925]:0x540da)
                     at sgen_ensure_free_space (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[924]:0x53f75)
                     at sgen_alloc_obj_nolock (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[911]:0x533bc)
                     at mono_gc_alloc_string (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[3174]:0xf0e73)
                     at mono_string_new_size_checked (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[2645]:0xd1efe)
                     at mono_interp_exec_method (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[257]:0x208b2)
                     at interp_entry (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[330]:0x286f9)
                     at interp_entry_static_0 (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[366]:0x29702)
                     at wasm_native_to_interp_System_Runtime_InteropServices_JavaScript_JSSynchronizationContext_BackgroundJobHandler (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[232]:0x1676e)
                     at mono_background_exec (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[836]:0x50e85)
                     at Os (http://127.0.0.1:35135/_framework/dotnet.runtime.js:3:69709)
                     at callUserCallback (http://127.0.0.1:35135/_framework/dotnet.native.js:8:102608)
                     at http://127.0.0.1:35135/_framework/dotnet.native.js:8:111163
[19:33:04] info: WASM EXIT 1

kg · 2024-04-09T20:36:01Z

This incorrectly ate https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-100801-merge-8c5cd227f4844ef98c/Methodical_d2/1/console.e95f9cb2.log?helixlogtype=result from a non-MT lane, which looks like it could be a real issue.

pavelsavara · 2024-04-10T06:22:56Z

This incorrectly ate https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-100801-merge-8c5cd227f4844ef98c/Methodical_d2/1/console.e95f9cb2.log?helixlogtype=result from a non-MT lane, which looks like it could be a real issue.

I already opened #100757 for it

lewing · 2024-04-13T22:05:31Z

this appears to have been fixed with #100907 closing so that new failures are caught

lewing · 2024-04-18T00:35:46Z

hit again in #101179

pavelsavara · 2024-04-18T07:33:01Z

There are 4 more this morning, 3 of them are after OOM.
The 4th is in GC (probably on the edge of OOM too)
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-101122-merge-f513e5fd0d4a45d08e/WasmTestOnBrowser-System.Text.Json.Tests/1/console.93cb60d2.log?helixlogtype=result

Perhaps some alloc failed and returned NULL and we are not checking it ?

pavelsavara · 2024-04-18T07:42:33Z

Alternatively, are we missing some memory fence which would synchronize the memory between WASM threads ?

We do issue .grow(0) in mono_wasm_synchronization_context_pump but sgen_perform_collection could be doing lot of work and the memory could grow on another thread in the meantime.

[09:03:07] fail: [0x020ff9e8-dpty 09:03:07.808] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at copy_object_no_checks.1 (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1119]:0x677c9)
                     at simple_nursery_serial_scan_object (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1117]:0x66fe9)
                     at simple_nursery_serial_drain_gray_stack (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1114]:0x666be)
                     at finish_gray_stack (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[957]:0x5a74b)
                     at collect_nursery (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[930]:0x5813f)
                     at sgen_perform_collection (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[928]:0x579f4)
                     at sgen_ensure_free_space (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[927]:0x5788d)
                     at sgen_alloc_obj_nolock (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[914]:0x56d40)
                     at mono_gc_alloc_vector (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[3167]:0xfab94)
                     at mono_array_new_specific_internal (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[2710]:0xddfeb)
                     at mono_array_new_specific_checked (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[2711]:0xde022)
                     at mono_interp_exec_method (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[257]:0x21463)
                     at interp_entry (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[330]:0x297e8)
                     at interp_entry_static_0 (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[366]:0x2a830)
                     at wasm_native_to_interp_System_Runtime_InteropServices_JavaScript_JSSynchronizationContext_PumpHandler (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[232]:0x171f1)
                     at mono_wasm_synchronization_context_pump (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[206]:0x1556e)

kg · 2024-04-18T07:48:28Z

I think under normal circumstances we shouldn't need to .grow(0), the theory was that we specifically needed to do it if the debugger paused thread(s) while a grow operation happened, wasn't it?

Unchecked null pointer feels very plausible. It will fail silently and also corrupt the zero page (which will cause jiterp traces to break.)

pavelsavara · 2024-04-19T07:25:35Z

I saw Log

[13:54:11] fail: Error in mono_download_assets: RangeError: Start offset -1 is outside the bounds of the buffer
[13:54:11] fail: [--000eafbc-emsc 13:54:11.333] MONO_WASM: Start offset -1 is outside the bounds of the buffer
                 RangeError: Start offset -1 is outside the bounds of the buffer
                     at new Uint8Array (<anonymous>)
                     at http://127.0.0.1:41625/_framework/dotnet.runtime.js:3:77638
                     at Object.pa [as instantiate_asset] (http://127.0.0.1:41625/_framework/dotnet.runtime.js:3:77686)
                     at n (http://127.0.0.1:41625/_framework/dotnet.js:3:8629)
                     at async Promise.all (index 129)
[13:54:11] fail: [out of order message from the browser]: http://127.0.0.1:41625/_framework/dotnet.runtime.js 2:77637 Uncaught RangeError: Start offset -1 is outside the bounds of the buffer
[13:54:11] info: WASM EXIT 1

This translates to ICU asset data being loaded into memory via mono_wasm_load_bytes_into_heap_persistent on MT build in UI thread, while the mono is starting on deputy thread.

It seems that Module._sbrk returned -1 🤯

It seems that sbrk could be racing with other threads and loose.
https://github.com/emscripten-core/emscripten/blob/1ca2f3141ec99110d6c5c5690dcd1c65d08d97e5/system/lib/libc/sbrk.c#L75

Also I'm not sure on what version of emscripten we are now W.R.T. https://github.com/emscripten-core/emscripten/pull/20793/files

We are just lucky it was called from JS and that we got nice JS exception.
I bet the same could happen to our C code as well.
Which would mean we could get RuntimeError: memory access out of bounds unless all our code is handling -1 and NULL returns from sbrk and malloc

pavelsavara added arch-wasm WebAssembly architecture blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-VM-threading-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm labels Jan 5, 2024

pavelsavara added this to the 9.0.0 milestone Jan 5, 2024

pavelsavara self-assigned this Jan 5, 2024

pavelsavara mentioned this issue Jan 5, 2024

[browser][MT] MutexTests - operation does not support unaligned accesses #96191

Closed

pavelsavara removed their assignment Jan 6, 2024

fanyang-mono self-assigned this Jan 11, 2024

ilonatommy mentioned this issue Jan 26, 2024

[browser][MT] Multithreading and JavaScript async interop in .NET 9 #85592

Closed

31 tasks

pavelsavara assigned pavelsavara and unassigned fanyang-mono Jan 26, 2024

pavelsavara mentioned this issue Feb 17, 2024

[browser][MT] try to fix "Invalid atomic access index" #98608

Merged

pavelsavara mentioned this issue Mar 21, 2024

[browser][MT] Unhandled JS error #97546

Closed

ilonatommy mentioned this issue Mar 26, 2024

[ci] Fix Playwright test times out/fails because target closed #100074

Merged

pavelsavara mentioned this issue Mar 26, 2024

[browser] fix bit width from 32 to 8 for managed boolean #100304

Merged

pavelsavara mentioned this issue Apr 8, 2024

[wasm] Methodical_d2 test failure #100757

Closed

pavelsavara mentioned this issue Apr 11, 2024

Revert "[wasm] Bump chrome for testing - linux: 123.0.6312.58, windows: 123.0.6312.58" #100877

Merged

lewing closed this as completed Apr 13, 2024

lewing reopened this Apr 18, 2024

pavelsavara removed their assignment Apr 18, 2024

This was referenced Apr 19, 2024

[browser] load core assemblies first #100141

Merged

[wasm] Assertion: should not be reached at D:/a/_work/1/s/src/mono/mono/sgen/sgen-scan-object.h:93 in System.Reflection.Tests #95805

Closed

lewing removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[browser][MT] RuntimeError: memory access out of bounds #96546

[browser][MT] RuntimeError: memory access out of bounds #96546

pavelsavara commented Jan 5, 2024 •

edited by build-analysis bot

ghost commented Jan 5, 2024

Error Blob

Reproduction Steps

pavelsavara commented Jan 11, 2024 •

edited

pavelsavara commented Feb 1, 2024

pavelsavara commented Feb 12, 2024

pavelsavara commented Feb 17, 2024

pavelsavara commented Feb 18, 2024

pavelsavara commented Feb 20, 2024

lewing commented Feb 21, 2024

pavelsavara commented Mar 10, 2024

pavelsavara commented Mar 26, 2024

pavelsavara commented Mar 26, 2024

pavelsavara commented Apr 3, 2024

kg commented Apr 9, 2024

pavelsavara commented Apr 10, 2024

lewing commented Apr 13, 2024

lewing commented Apr 18, 2024

pavelsavara commented Apr 18, 2024 •

edited

pavelsavara commented Apr 18, 2024 •

edited

kg commented Apr 18, 2024

pavelsavara commented Apr 19, 2024

[browser][MT] RuntimeError: memory access out of bounds #96546

[browser][MT] RuntimeError: memory access out of bounds #96546

Comments

pavelsavara commented Jan 5, 2024 • edited by build-analysis bot

Error Blob

Reproduction Steps

Known issue validation

Report

Summary

ghost commented Jan 5, 2024

Error Blob

Reproduction Steps

pavelsavara commented Jan 11, 2024 • edited

pavelsavara commented Feb 1, 2024

pavelsavara commented Feb 12, 2024

pavelsavara commented Feb 17, 2024

pavelsavara commented Feb 18, 2024

pavelsavara commented Feb 20, 2024

lewing commented Feb 21, 2024

pavelsavara commented Mar 10, 2024

pavelsavara commented Mar 26, 2024

pavelsavara commented Mar 26, 2024

pavelsavara commented Apr 3, 2024

kg commented Apr 9, 2024

pavelsavara commented Apr 10, 2024

lewing commented Apr 13, 2024

lewing commented Apr 18, 2024

pavelsavara commented Apr 18, 2024 • edited

pavelsavara commented Apr 18, 2024 • edited

kg commented Apr 18, 2024

pavelsavara commented Apr 19, 2024

pavelsavara commented Jan 5, 2024 •

edited by build-analysis bot

pavelsavara commented Jan 11, 2024 •

edited

pavelsavara commented Apr 18, 2024 •

edited

pavelsavara commented Apr 18, 2024 •

edited