Large .mem file with LLVM upstream/wasm2js

[cwoffenden]

Building asm.js with the new LLVM can result in a much larger .mem file. Here's a standalone test:

#include <stdio.h>

char buf[4 * 1024 * 1024];

int main() {
    for (int n = sizeof buf; n > 0; n--) {
        buf[n] = n & 63;
    }
    printf("%d\n", buf[0]);
    return 0;
}

Compile using emcc test.c -Os -s WASM=0 -o out.js. With latest-upstream the resulting out.js.mem is 4196767 bytes, and with Fastcomp (latest) it's 973 bytes.

[kripken]

Thanks @cwoffenden! Yeah, this looks bad.

This has a huge zero-initialized segment up front, and ends in a few bytes of non-zero. As an optimization in emscripten we trim zeros from the end, but not from the beginning or middle. We could trim the beginning with some work, but I'm not sure how generally useful that would be. For the middle, we just need the linker to handle things.

@sbc100 can wasm-ld reorder data sections so that all the zeros are at the end, perhaps?

[sbc100]

Hmm, I thought binaryen would trim all zero and split the data segment in smaller ones? But maybe that only happens for wasm output, not for js + .mem output?

I'll take a look. lld should already put all bss at the end of the data segment.

[kripken]

Binaryen does it for wasm, yeah, using the segment mechanism there. For wasm2js we just have a flat binary file. I guess in theory we could implement in JS something like segments, basically a little compression scheme. But if it's natural to do this in wasm-ld then I think that would be more generally useful. If not I can look at the JS option.

[tlively]

@sbc100 I can take a look at this in the next few days if you'd like. I would like to do some similar optimizations for eliding bss sections when using passive segments, and it would make sense to tackle this at the same time. I'll assign this to myself for now.

[kripken]

Great! I verified the testcase here now has a 847 byte mem init file, even smaller than fastcomp :)

[aheejin]

Funny, it's not my commit but it says I closed this because of the reflected commit in my personal fork..? How can one prevent this, should I just not have a fork? :(

[tlively]

Not a problem. I think you're the first person with permissions to close this issue to have acquired the commit in a repo you control. GitHub automatically closes issues when it sees commits with language like "Fixes #8999". I deliberately put that in, but I could have used different wording to avoid having the issue automatically closed.

[aheejin]

No I think putting that word in the commit is good; you don't need to close the issue separately. :)

[kripken]

Yeah, this seems like a feature, not a bug :)

But I am surprised github will close an issue in repo X because of a commit in repo Y... even if the user with the commit is an admin in repo X. But it does work out conveniently I guess!

[aheejin]

Yes I think so, but I guess Github can

1. Only count commits to non-fork repos
2. Make the author of the commit as the person who closes the issue, not the owner of the repo

[kripken]

Yeah, this could be much better, I agree.

Meanwhile I found the docs for this,

For example, including Closes example_user/example_repo#76 will close the referenced issue in that repository, provided you have push access to that repository.

That does seem to not take into account forks...