Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wasm: large memory usage with hard-coded map/array #42979

Open
tbhartman opened this issue Dec 3, 2020 · 1 comment
Open

wasm: large memory usage with hard-coded map/array #42979

tbhartman opened this issue Dec 3, 2020 · 1 comment

Comments

@tbhartman
Copy link

@tbhartman tbhartman commented Dec 3, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.4 windows/amd64

Does this issue reproduce with the latest release?

Unknown; due to my employer's security policy, I cannot install latest on my machine.

What operating system and processor architecture are you using (go env)?

go env Output
*Some output redacted*
$ go env
set GO111MODULE=
set GOARCH=wasm
set GOBIN=
set GOCACHE=C:\Users\redacted\AppData\Local\go-build
set GOENV=C:\Users\redacted\AppData\Roaming\go\env
set GOEXE=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GONOPROXY=
set GONOSUMDB=
set GOOS=js
set GOPATH=C:\Users\redacted\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set GOWASM=
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=C:\Users\redacted\wasm\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-fPIC -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\redacted\AppData\Local\Temp\1\go-build312734978=/tmp/go-build -gno-record-gcc-switches

What did you do?

Files available at: https://gist.github.com/tbhartman/c496b9cfa9eb3ec1c736fd0849a30e98

I built a Wasm module. Initially, I was trying to use the Goldmark markdown parser, but upon loading the module in the browser, the browser page quickly crashed with a memory error. I identified that the crash was reproduced by only importing "github.com/yuin/goldmark/util", which defines a couple of const maps by hard-coding several thousand elements.

I have created a minimum working example that reproduces the issue (see the gist) by declaring an array (in my example, a string array []string) and hard-coding the filling of that map/array:

ss := make([]string, 3000)
ss[0] = "hi"
ss[1] = "hi"
ss[2] = "hi"
// many lines omitted...
ss[2997] = "hi"
ss[2998] = "hi"
ss[2999] = "hi"

When run in the browser, even after the main() function returned, the browser page memory usage would increase greatly (in my case, over 12 GB). If the page did not crash, once the memory peaked, it would gradually reduce back to normal usage (<100MB).

Performing the same operation in a loop, I never saw the page memory usage increase above 100MB:

ss := make([]string, 3000)
for i := 0; i < 3000; i++ {
    s[i] = "hi"
}

In another test, I broke up the 3000 hard-coded lines into 300 goroutines operating asynchronously, each of which handled 10 of the assignments. In this case, the page memory never increased above 100 MB. (See the example in the gist.)

In an intermediate case, the 3000 hard-coded lines were broken into 6 asynchronous goroutines. In this case, the memory peaked around 2 GB.

What did you expect to see?

A lower memory usage (or, in my original case, no crash).

What did you see instead?

A very high memory usage or a crash.

@dmitshur dmitshur added this to the Backlog milestone Dec 3, 2020
@dmitshur dmitshur changed the title Wasm large memory usage with hard-coded map/array wasm: large memory usage with hard-coded map/array Dec 3, 2020
@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Dec 3, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants