"Out of bounds memory access" on iOS 11.2.2

[bhollis]

We're using a modified fork of sql.js built for wasm using an emscripten version from about 7 months ago (can't recall the exact version). With the update to iOS 11.2.2 and the Meltdown/Spectre mitigations, this code now fails at sqlite3_open with "Out of bounds memory access".

I suspect this change may have affected something: https://trac.webkit.org/changeset/226461/webkit

I'm rebuilding with the latest emscripten now to see if it helps. I'm not sure if this is an emscripten issue or a sqlite issue.

[bhollis]

Rebuilding with 1.37.28 (which took some changes to work) still produces wasm that fails.

[bhollis]

Rebuilding with the latest sqlite3.c (3.21.0) still produces wasm that fails.

[juj]

You could try doing an asm.js build to see if that works - if it also fails, then it might be an Emscripten issue. (it can also be an Emscripten issue even if it doesn't, but might help at poking the issue from another angle). Also doing an -O0 build so that callstacks are preserved should show the function names instead of the "wasm function" blocks being printed - that might help debug as well.

To be sure, it would probably be best if you're able to run the original build that did work on iOS, and cross-check the same build against the old working iOS and new breaking iOS. If the only variable is the iOS version change, it would then be fair to say that it would be an iOS regression, although whether it's then a wasm execution regression or a consequence of some other web api regressing (and resulting in an eventual wasm trap) is then still a question.

[bhollis]

The asm.js build works - right now I'm trying to execute wasm, catching the error, and falling back to asm.js. The older build's asm.js didn't work but it may have been broken for unrelated reasons - I hadn't tested it in a bit.

We definitely only have this issue on iOS 11.2.2 - both the original build and the new one don't work there but continue to work on iOS 11.2.1. Weirdly, the patched desktop Safari runs the wasm fine.

[kripken]

This sounds more and more like an iOS regression. Another thing I'd check is to run the same workload on Chrome and Firefox on desktop. If, as seems likely, it works on desktop on Safari, Chrome, and Firefox, and just fails on iOS, I'd file a WebKit bug with this.

[seankibler]

Experiencing the exact issue as well. Have tested on multiple Apple devices against the exact same WASM build pre and post 11.2.2 update and all exhibit the same behavior; work fine pre 11.2.2 but not after. Have also confirmed desktop Safari is ok. Will also try an ASM build. Agree with others that this is likely a regression, perhaps from a rushed guard against Meltdown or Spectre, or maybe just coincidence.

[bhollis]

Yup, desktop Firefox and Chrome work fine. I'm mostly putting this here so people are aware, I'm pretty sure it's an Apple bug, and it's a hard one for wasm users to work around because you can't just feature detect it.

[jfbastien]

This is a known issue (rdar://problem/36236004) and is fixed in tip-of-tree WebKit.

[bhollis]

Awesome! I don't have radar access but I'm excited for the fix.

[elbalkini]

anyone has an update regarding this issue?

[bhollis]

I've been told the fix is included in the latest iOS betas.

[elbalkini]

i am on IOS 11.2.5 Beta and it is still not fixed. Same error message

[lgfaler]

Also seeing this same issue with our WASM. Falling back to ASM fixed the issue, but the decrease in performance makes that an unusable option for our application. Hoping for a fix soon.

[bvibber]

11.2.5beta6 doesn't fully resolve it for me in ogv.js; I filed https://bugs.webkit.org/show_bug.cgi?id=181781 in case there's another regression on top of the initial fix.

[bvibber]

11.2.5 released today, apparently without the fix.

[elbalkini]

It is totally frustrating how Apple is handeling their technology, Webassembly is broken, getusermedia is only on safari no support in the Webkit webview nor safari controller. everything is either Apple way or 'nil'.

[phoboslab]

Can confirm. The bug is still present in iOS 11.2.5.

Is there some way to detect the faulty WebAssembly implementation other than just probing the UA with /iPhone OS 11_2_(2|5)/.test(navigator.userAgent)? Preferably without having to load the WASM blob, execute it and catch the error first.

[bvibber]

@phoboslab you can feature-detect the failure with a minimal wasm module like this:

• source: https://github.com/brion/min-wasm-fail
• live copy: https://brionv.com/misc/min-wasm-fail/

(You could do this synchronously by loading from an array in the source instead of loading a small .wasm file.)

Base error seems to be that it has trouble loading/storing memory. There may be a cleaner test, but this works for me confirming the fail on 11.2.5 and doesn't fail elsewhere: store a non-zero i32 value at memory index 4, then load it back. If you get 0, the failure is live.

(Note this doesn't throw the out of bounds exception, which looks like a side effect of having wrong values loaded/stored such as pointers.)

[bvibber]

I can confirm this is resolved for me in iOS 11.3 beta 1.

[elbalkini]

@Brion thanks for the update, this is much appreciated.

[steipete]

We hit the problem in our Web PDF Renderer for iOS 11.2.2 and 11.2.5 and I can confirm that 11.3b1 indeed resolves this issue. Our fallback is to use the asm.js build on these specific versions.

[elbalkini]

any ideas when IOS 11.3 will be public to all ios users?

[shader13]

Did you mean 11.3b1 is 11.3 development beta1?
I have updated to this version, but I still get the issue using Unity WebGL WASM:
Unhandled Promise Rejection: Error: Out of executable memory in function at index

[steipete]

Apple rarely gives exact release dates for iOS point releases, but has given a general time frame of "this spring." So far, only the first beta has been released to developers, not the general public. Given the features in this release, we expect at least four or five beta releases, possibly many more. We doubt iOS 11.3 will release before March. (source)

[bvibber]

Error: Out of executable memory in function at index

That may be a distinct issue from the "out of bounds" failures, but is certainly also worrying if it's a regression. It sounds like that could be an issue with compiling a large amount of JS or wasm code in general... Note some devices supported by iOS 11 have low memory ceilings like the first gen iPad Air, which may also make them more susceptible to that.

If it looks like a regression where it worked prior to 11.2.2 on the same device, then definitely file a bug with bugs.webkit.org or direct in Apple's internal tracker.

[jfbastien]

JSC usable executable memory is set here on a per-processor type basis: https://trac.webkit.org/browser/webkit/trunk/Source/JavaScriptCore/jit/ExecutableAllocator.cpp

[caiiiycuk]

Same problem for me:

12381ms |  exception thrown: Error: Out of bounds memory access
(evaluating 'Module["asm"]["_main"].apply(null,
arguments)'),<?>.wasm-function[__ZN4game18Scene_SplashScreen4initEv]@[wasm
code]
<?>.wasm-function[__ZN4game18Scene_SplashScreen6createEv]@[wasm code]
<?>.wasm-function[__ZN4game5CMain4initEv]@[wasm code]
<?>.wasm-function[__ZN4game11AppDelegate29applicationDidFinishLaunchingEv]@[wasm
code]
<?>.wasm-function[__ZN7cocos2d11Application3runEv]@[wasm code]
<?>.wasm-function[_main]@[wasm code]
wasm-stub@[wasm code]
_main@[native code]

callMain
http://192.168.1.37/loader.js:456:30
fn@http://192.168.1.37/loader.js:465:32

emscripten: 1.38.10 (commit 8e5456b)
iphone: 6, 8, 10
ios 11.4.1

[kripken]

cc @jfbastien, see last comment - this seems to still be an issue.

[kmiller68]

@caiiiycuk Do you have a repro case? If so, can you open a bug on bugs.webkit.org and CC me (keith_miller)?

[caiiiycuk]

@kmiller68 I think I have only build in binary form (wasm). I can't provide sources because of NDA, is it ok for you?

[kmiller68]

Sure, I can take a look either way.

[caiiiycuk]

I figured out the reason of this exception. I think it is not webkit bug and not emscripten bug, I use incorrect link settings. I've use -s TOTAL_MEMORY=48Mb -s ALLOW_MEMORY_GROWTH=1 -s WASM_MEM_MAX=128Mb. I thought that means that memory will grow from 48Mb but not bigger then 128Mb, but actually memory will grow as follow: 48Mb, 96Mb, 192Mb... And when code tries to increase memory from 98Mb to 192Mb it fails silently and error "exception thrown: Error: Out of bounds memory access" will occurs after a while.

It is happen in all browsers, I found this on iphone because I think it's slower. My code do some concurrent tasks, and I think they stay alive more time on iphone (so need more memory).

In FF:

exception thrown: RuntimeError: index out of bounds,_free@http://localhost/oom.js:17190:1
_fclose@http://localhost/oom.js:17631:1
_main@http://localhost/oom.js:34018:1

In Chrome:

exception thrown: RuntimeError: memory access out of bounds,RuntimeError: memory access out of bounds
    at _free (wasm-function[45]:1508)
    at _fclose (wasm-function[46]:167)
    at _main (wasm-function[81]:10986)

In Safari:

exception thrown: Error: Out of bounds memory access (evaluating 'Module["asm"]["_main"].apply(null, arguments)'),<?>.wasm-function[_free]@[wasm code]
<?>.wasm-function[_fclose]@[wasm code]
<?>.wasm-function[_main]@[wasm code]
wasm-stub@[wasm code]
_main@[native code

The problem is that this error is not clear (in -O3 --profling), I expect to see "Out of memory" in moment when memory can't grow. But error "exception thrown: Error: Out of bounds memory access" occurs not in moment of failed grow.

Super simple test case to reproduce attached. Compile with:

emcc --preload-file "res_test@" --no-heap-copy -O3 --profiling -s TOTAL_MEMORY=48Mb -s ALLOW_MEMORY_GROWTH=1 -s WASM_MEM_MAX=128Mb --std=c++11 oom.cpp -o oom.html

oom.zip

[caiiiycuk]

@kripken Should I create issue to improve error message for this case?

[kripken]

Thanks @caiiiycuk, I think what's wrong is that we didn't respect the wasm mem max in growth mode. Fix in #6937 - does your app work ok with that? (or, at least show a proper error on malloc failing when reaching the wasm mem max limit)

[caiiiycuk]

Sorry, I can't verify because I am on vocations now, but when I return (after 15th of August) I will check for sure. Thanks!

[kripken]

(@caiiiycuk That PR was merged, so when you get back you can just test it on incoming now.)

[caiiiycuk]

Just checked, problem is solved 👍

[kripken]

Great, thanks @caiiiycuk.

Ok, I think all bugs discussed here are known to be fixed, closing.

[ghost]

Hello everyone,

I tested and believe the issue is happening again on iOS 13. Is any of you aware of that issue as well?

@kripken @Brion @caiiiycuk

[ghost]

@kripken @Brion @caiiiycuk

I was able to debug the console in Safari on iOS 13 and here is the output for the sql.js GUI:

The issue happen when we try to execute sqlite3_prepare_v2:

iOS 13 is set to rollout in less than a month, there is still going to be one beta to be released but I'm afraid nothing is going to happen if they are not aware of this issue / if it is intended and we don't fix it.

Note that this is not the simulator but an actual device, WebAssembly is not supported in the iOS Simulator.

[ghost]

Ok, we're not alone: mono/mono#15989

A ticket just got opened on Webkit: https://bugs.webkit.org/show_bug.cgi?id=201026

Hopefully this fixes it 🤞

[bvibber]

I'm not seeing any problems with my projects, but I can confirm that I see the same failure with the sqlite demo on iOS developer beta 8 but not in iOS 12 or Safari Technology Preview on desktop.

The stack trace seems really short for 'maximum call stack exceeded' though? (Note the stack limit for Safari in iOS seems to be about 8192 call levels on both 12 and 13.)

[caiiiycuk]

I also usually have problems on ios when my binaries become big. I believe this is somehow related to OOM problems in Safari browser. So, I trying to build smallest possible binaries (with Oz + other optimizations) and this work in most cases.

[ghost]

I also usually have problems on ios when my binaries become big. I believe this is somehow related to OOM problems in Safari browser. So, I trying to build smallest possible binaries (with Oz + other optimizations) and this work in most cases.

Indeed, thanks to you and the -Oz option I was able not only able to reduce by 50% the size of the binary but also make it work on iOS 13! 😄

However, the fact that the issue is not happening on iOS 12.4 makes me believe it's a regression and WebKit should take care of it.