New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: wasm code causes out of memory error on Chrome and Firefox for Android #27462

Open
termonio opened this Issue Sep 3, 2018 · 23 comments

Comments

Projects
None yet
10 participants
@termonio
Copy link

termonio commented Sep 3, 2018

I am very excited that Go ships now with Webassembly support. I ran wasm code generated by Go 1.11 on Chrome and Firefox on desktops (MacOS and Linux) and on Chrome, Firefox and Safari on iOS devices. Running Go generated wasm code on Android devices failed though.

Minimal example

  • Go code
    • GOOS=js GOARCH=wasm go build -o test.wasm wasm.go
package main

import (
    "fmt"
)

func main() {
    fmt.Println("hello")
}
  • index.html
    • wasm_exec.js from go/misc/wasm
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width,initial-scale=1.0">
    </head>
    <body>
        <script src="wasm_exec.js"></script>
        <script>
            (async function() {
                const wasmFile = "test.wasm"
                let run
                const go = new Go()
                try {
                    const { instance } = await WebAssembly.instantiateStreaming(fetch(wasmFile), go.importObject)
                    document.querySelector('#info').innerHTML = "ready"
                    run = go.run(instance)
                } catch (err) {
                    document.querySelector('#info').innerHTML = err
                    console.log(err)
                }
            })()
        </script>
        <div id="info"></div>
    </body>
</html>
  • files are served via Nginx using adjusted mime.types

Expected behavior

  • page displays "ready" after the wasm file has been loaded and the console shows "hello"
    • this works on Desktops and iOS devices

Actual behavior

  • on Android devices the above code fails with "RangeError: WebAssembly Instantiation: Out of memory: wasm memory" (Chrome) and "out of memory" (Firefox)
    • tested on Chrome for Android (68.0.3440.91) and Firefox for Android (61.0.2). Several devices from different manufacturers were tested
@agnivade

This comment has been minimized.

Copy link
Member

agnivade commented Sep 3, 2018

Since the code is same for all desktop, iOS and Android devices, I doubt there is much we can do here.

@neelance has some optimizations in mind for 1.12. But unless you have some specific suggestions, this just falls in the category of general optimizations which will happen anyways.

@agnivade agnivade changed the title Wasm code generated by Go 1.11 causes out of memory error on Chrome and Firefox for Android cmd/compile: wasm code causes out of memory error on Chrome and Firefox for Android Sep 3, 2018

@termonio

This comment has been minimized.

Copy link

termonio commented Sep 3, 2018

Given the market share of Android devices, this would be a major drawback ...

@thesyncim

This comment has been minimized.

Copy link

thesyncim commented Sep 3, 2018

can you try this?

wasm-opt test.wasm -O -o testO.wasm

@agnivade

This comment has been minimized.

Copy link
Member

agnivade commented Sep 3, 2018

btw @termonio - You should also build with -ldflags='-s -w' for slightly smaller binaries. It most probably will not help with overall memory instantiation, but can help with payload size.

@termonio

This comment has been minimized.

Copy link

termonio commented Sep 3, 2018

@thesyncim: wasm-opt test.wasm -O -o testO.wasm reduces the file size from 2.4MB to 2.3MB. The optimized file works on Desktops and iOS but still not on Android.

@agnivade: Your hint has a similar effect. The payload is reduced to 2.3MB, but can't be run on Android.

For someone who wants to reproduce this issue: iOS devices need a polyfill (omitted in my minimal code example above).

if (!WebAssembly.instantiateStreaming) {
    WebAssembly.instantiateStreaming = async (resp, importObject) => {
        const source = await (await resp).arrayBuffer()
        return await WebAssembly.instantiate(source, importObject)
    }
}
@thesyncim

This comment has been minimized.

Copy link

thesyncim commented Sep 3, 2018

you can try other optimization levels like -O2 or -O4 for binary size consider using -Os or -Oz

@termonio

This comment has been minimized.

Copy link

termonio commented Sep 3, 2018

I did try other optimization levels but got core dumps (not further investigated yet).

@thesyncim

This comment has been minimized.

Copy link

thesyncim commented Sep 3, 2018

@termonio try to run with -d (debug option) (in my case running with -d avoid core dumps, also -O4 requires a lot of memory)

@termonio

This comment has been minimized.

Copy link

termonio commented Sep 3, 2018

@thesyncim: running with -d yields an output and further optimization makes the file sizes shrink down to 1.9MB. Won't run on Android though (tried -O2, -O4, -Os, and -Oz).

@thesyncim

This comment has been minimized.

Copy link

thesyncim commented Sep 3, 2018

@termonio sorry to hear that, just one last thought, did you try to run the optimization on top of @agnivade sugestion -ldflags='-s -w'?

@termonio

This comment has been minimized.

Copy link

termonio commented Sep 3, 2018

@thesyncim : Yes indeed. I tried all 12 combinations (with and without -ldflags='-s -w' and for each in addition wasm-opt for optimization levels -O, -O2, -O4, -Os, and -Oz). I don't think this issue can be resolved with general optimization. (I guess it is how much memory can be allocated on an Android device for Webassembly. This seems to be smaller than on other platforms. new WebAssembly.Memory({initial: x}) fails on my Android device when x is around 9000 64kB pages.)

@termonio

This comment has been minimized.

Copy link

termonio commented Sep 3, 2018

I looked into the instance that is returned from WebAssembly.instantiateStreaming(fetch(wasmFile), go.importObject) when running on a desktop browser. instance.exports.mem hold a WebAssembly.Memory object. The allocated ArrayBuffer holds 1073741824 Bytes = 1GB after instantiation! I am wondering whether that much memory is really needed.

@neelance

This comment has been minimized.

Copy link
Member

neelance commented Sep 3, 2018

The solution most likely depends on https://github.com/WebAssembly/design/blob/master/FutureFeatures.md#finer-grained-control-over-memory. However, even currently it is not necessary for the WebAssembly host and operating system to physically allocate the full amount of memory, since most of it is not used. For example on Chrome on OS X, the operating system reports a much lower memory usage than the 1GB that WebAssembly requests.

@termonio

This comment has been minimized.

Copy link

termonio commented Sep 3, 2018

Why does WebAssembly allocate this 1GB of (mostly unused) memory in the first place? Is there a way to limit the amount of memory it can request? (WebAssembly.Memory({initial: x, maximum: y}) comes to mind but my attempts to populate the importObject with preallocated memory did not succeed.)
Edit: Dumping a wasm file with wasm-dump shows that memory is indeed set to 16384 64kB pages (=1GB): memory[0] pages: initial=16384. I am wondering whether this can be changed to a more reasonable size.

@cherrymui

This comment has been minimized.

Copy link
Contributor

cherrymui commented Sep 4, 2018

The initial 1 GB memory is control by this line:

https://go.googlesource.com/go/+/go1.11/src/cmd/link/internal/wasm/asm.go#310

I don't think we will change this setting in Go 1.11. But you can modify the source and rebuild the toolchain.

@agnivade

This comment has been minimized.

Copy link
Member

agnivade commented Sep 4, 2018

@neelance - I see a TODO there to use lower initial memory size. I believe the challenge to set the correct initial memory size is to somehow analyze the code being compiled and come up with the base minimum memory the code would need ?

Is it possible to hoist this from being hardcoded in the binary to being set from the importObject ? So that the user has control over the value. Or do we not want to expose more knobs ?

@termonio

This comment has been minimized.

Copy link

termonio commented Sep 4, 2018

I wrote a small tool that can patch the memory section of a .wasm binary. This allows for easy experimenting with smaller initial page sizes without building a modified tool chain. It seems as if during instantiation quite a bit of memory is allocated by the WebAssembly runtime. When starting with 4096 pages (256MB) the runtime grows the memory on my desktop machine to 745865216 bytes (about 710MB, more than 11000 pages). As I have trouble allocating more than 7500 pages on my Android devices, this approach alone won't help to make Go generated .wasm binaries run on Android. I am surprised that the instantiation is that expensive but I can understand now why the initial memory was set to 1GB ...

@bradfitz bradfitz added this to the Unplanned milestone Sep 6, 2018

@bradfitz bradfitz added the Performance label Sep 6, 2018

twifkak added a commit to twifkak/amppackager that referenced this issue Dec 13, 2018

Basic test benchmark for WebAssembly integration.
This is not nearly as efficient as a native build of the transform
binary, due to bugs such as http://crbug.com/853685 and
golang/go#27462, but serves as a starting
point for further investigation.

There have been a few optimizations already:
 - Keeping one long-lived process open to amortize the bootstrapping
   cost across requests.
 - Batching the "requests" to the wasm code via Promise.all. (This can
   be seen by adding a log statement just before the call to exports.run
   in wasm_exec.js.)

Another potential problem to investigate the cost of is the lack of
cross-heap GC.

twifkak added a commit to ampproject/amppackager that referenced this issue Dec 15, 2018

Basic test benchmark for WebAssembly integration. (#216)
This is not nearly as efficient as a native build of the transform
binary, due to bugs such as http://crbug.com/853685 and
golang/go#27462, but serves as a starting
point for further investigation.

There have been a few optimizations already:
 - Keeping one long-lived process open to amortize the bootstrapping
   cost across requests.
 - Batching the "requests" to the wasm code via Promise.all. (This can
   be seen by adding a log statement just before the call to exports.run
   in wasm_exec.js.)

Another potential problem to investigate the cost of is the lack of
cross-heap GC.
@Yaoir

This comment has been minimized.

Copy link

Yaoir commented Jan 8, 2019

I also ran into this problem. My Go/WebAssembly app may be helpful for analysis or testing bug fixes in the wasm compiler: https://github.com/Yaoir/VideoPoker-Go-WebAssembly

I tried the wams tool, and found it seemed to alter reliability, but if reliability increased in one browser, it decreased or entirely broke the app in another browser or on another device. Overall, nothing was solved.

I worked with the JavaScript glue code that's in the HTML file to load sequentially rather than using the streaming JavaScript calls. I still got "out of memory" errors. I also put console.log() statements between calls to fetch, compile, and load to see where things went wrong. I did not find any solutions, but learned that the "out of memory" error occurs during the linking phase of the fetch/compile/load sequence.

I noticed that if the app isn't entirely broken on a combination of browser and device, it sometimes it can be made to work by clearing the browser cache, restarting the browser app, and loading the WebAssembly app fresh, into a browser that has not loaded any other pages already. But reloading the page one or more times may bring up the error, and reloading after getting an error may actually work! For a while, I had Firefox for Android running the app successfully every other time the page was reloaded.

@twifkak

This comment has been minimized.

Copy link

twifkak commented Jan 9, 2019

I hacked on the runtime library to reduce the initial allocation to ~80MB: master...twifkak:small

A couple of notes:

  • I have no idea if this is a good idea. The comments about keeping the heap contiguous might apply here, too? I just looked around for things that call growMemory and then lowered some constants.
  • If you use @termonio's wams tool after compilation, then you don't need to build a custom Go compiler. Only patch your GOROOT's runtime dir, which will get compiled into the wasm binary.
  • No idea if this has good/bad/no effect on long-term memory usage. I'm still trying to figure that out.
  • You probably want to remove the printlns from my patch.

Update: It seems this is bad for processes that create a lot of flyweight objects. runtime/mem_js.go needs a free list or some such. Update 2: I wrote a free list. It requires GODEBUG=gcstoptheworld=1.

@free1139

This comment has been minimized.

Copy link

free1139 commented Jan 10, 2019

It works. Thank you! @twifkak

@twifkak

This comment has been minimized.

Copy link

twifkak commented Jan 12, 2019

I updated my fork to implement a free list, so that Go can reclaim freed memory in wasm. Using this, combined with GODEBUG=gcstoptheworld=1, I was able to run a pretty allocation-intensive workload (parsing a bunch of HTML files) with <124MB. (Using the concurrent GC, it crashes.)

Update: With GOGC=20 (arbitrary first guess), memory is <80MB.

@olso

This comment has been minimized.

Copy link

olso commented Jan 16, 2019

It works! Thank you so much @twifkak.

I'm writing article/opinion/tutorial where my goal is to create a simple game that can be run on mobile because I need touch interaction.

You saved me so much time! ❤️

I'll share the publication once its done!

@twifkak

This comment has been minimized.

Copy link

twifkak commented Jan 16, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment