Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Measure size increase for enabling EH on modern C++ codebases #20

Open
lukewagner opened this issue May 17, 2017 · 14 comments
Open

Measure size increase for enabling EH on modern C++ codebases #20

lukewagner opened this issue May 17, 2017 · 14 comments
Assignees

Comments

@lukewagner
Copy link
Member

This is a request from people working on the producer side to get some real-world measurements of the size increase of enabling exception handling in large C++ codebases, particularly those that heavily use the common RAII style which will end up giving a large percentage of functions one or more catch_all blocks.

I'm not sure what the criteria is for what's an acceptable increase, but if it's significant then we should probably reconsider some of the strategies that would allow sharing code between the normal and unwinding exit paths. It won't feel great if we do all this work to add EH to wasm and the general advice immediately becomes "don't enable EH".

@jfbastien
Copy link
Member

Good question. Additionally, I'd like to understand how much of this size is metadata versus code.

@aheejin aheejin self-assigned this Oct 20, 2019
@lukewagner
Copy link
Member Author

Hah, I had this same question again today and when I did a quick search to see if anyone had already asked/answered this question I found this old issue :) Just thought I'd bump this b/c I'm still quite interested to see any measurements here. Thanks!

@aheejin
Copy link
Member

aheejin commented Apr 22, 2020

We made end-to-end support work for small tests and are starting to test larger programs now, so we don't have reliable data on this yet. I think they will be available in a few to several months (depending on the size of programs), and I'll update this issue on that.

@lars-t-hansen lars-t-hansen mentioned this issue Sep 7, 2021
11 tasks
@aheejin
Copy link
Member

aheejin commented Sep 15, 2021

We don't have results from a complete benchmark suite or something at this point, but now we have some data points.

Note that some amount of code size increase is inevitable because there is simply more code to compile. -fignore-exceptions, the baseline, compiles away all landing pads, including catch clauses and destructor calls for stack objects when an exception is thrown.

We compiled Binaryen using Wasm and compared code size increases with the baseline (-fignore-exceptions). The LTO version of emcc -Oz showed a 9.1% increase in code size of wasm-opt.wasm, the main code optimizer in Binaryen. The old Emscripten-style exception handling showed 42.8% increase in the same setup. So 9.1% vs. 42.8% is a clear win for code size using Wasm EH. Other tools in Binaryen showed similar numbers. Non-LTO version also showed a big difference:

Platform / Build Code Size Increase
Wasm EH, emcc -Oz (latest), non-LTO 13.7%
Wasm EH, emcc -Oz (latest), LTO 9.1%
Emscripten EH, emcc -Oz (latest), non-LTO 58.9%
Emscripten EH, emcc -Oz (latest), LTO 42.8%

We also checked how native platforms perform for the same wasm-opt binary. All code size increase are the ratio compared to the baseline (-fignore-exceptions). Binaries are statically linked as in the Wasm binaries.

Platform / Build Code Size Increase
Linux (x86), clang -Oz (12.0.1), non-LTO 5.7%
Linux (x86), clang -Oz (12.0.1), LTO 7.1%
Windows (x86), clang -Oz (latest), non-LTO 20.0%
Windows (x86), clang -Oz (latest), LTO 22.0%

Wasm EH's numbers are still a little higher than Linux x86’s, but the code size highly depends on how many optimizations the toolchain can do, and given that we have spent very little time on optimizing EH code in our toolchain so far, I think we have still have much room for improvement. But the bottom line is that the current numbers already show the advantage of using the new proposal over the old Emscripten-style EH.

Most (about 97.5%) of the code size increases comes from the code section. The rest 2.5% is from increase from the data section, which contains LSDA info (= exception table for each function that has landingpads).

We also asked a partner we are working with, who has a large cross-platform application they are porting to the web. Their application uses exceptions heavily and is much bigger than Binaryen. With -O2 and non-LTO builds, they reported a 13.4% code size increase. It looks like they haven't tried LTO build yet so they may benefit from that more. We asked them to also build with Emscripten EH but there was some problem so they couldn't build it, but we can say the size was much larger.

@lars-t-hansen
Copy link
Contributor

@aheejin, thank you! If I read this correctly, "code size" for wasm is uncompressed wasm bytecode size. I agree that 10-15% increase for bytecode that uses exception handling pervasively (as would C++) is reasonable, especially as this is the ceiling.

Also interesting to me would be the changes in the size of machine code and in-memory metadata in V8, as this can matter just as much as bytecode size, especially on mobile and in other size-constrained environments. Once we have an implementation in SpiderMonkey we'll collect and publish these numbers too, but in the mean time any numbers from V8 would be welcome.

@donalffons
Copy link

donalffons commented Sep 19, 2021

Hi, I just came across this thread. I am working on a project called OpenCascade.js, which is converting the rather large OpenCascade project to WebAssembly using Emscripten. The current beta version of OpenCascade.js has ~47 MB (without exception support).
If there is an easy way to test Wasm EH using some Emscripten compiler / linker flags (which ones?) - and if it's interesting for you, I would be happy to run some builds and compare the results. It might also be interesting to look at performance - I have experienced huge performance drops with Emscripten's implementation and would be curious if those go away with Wasm EH.

@dschuff
Copy link
Member

dschuff commented Sep 20, 2021

Emscripten's docs now cover enabling exceptions in emscripten. To test with Chrome, you can go to chrome://flags and turn on #enable-experimental-webassembly-features

@donalffons
Copy link

donalffons commented Sep 21, 2021

Thanks a lot for pointing me at the docs! Didn't know (and expect) that this was already documented.

I made the following (custom) builds of the library

  1. Disabled EH
  2. Emscripten EH
  3. Wasm EH

Those two builds are custom builds of the library with a reduced feature set (hence the smaller file size than previously mentioned). Unfortunately, I wasn't able to create a full build of the library since I was running into an LLVM error - I might file an issue with them about this.

The difference in file size between Wasm EH and Emscripten EH is (main JS + WASM + worker JS):

  1. Disabled EH: 16,337,122 bytes
  2. Emscripten EH: 23,203,693 bytes (+42% vs disabled EH)
  3. Wasm EH: 20,221,475 bytes (+24% vs disabled EH)

Difference Wasm EH vs Emscripten EH: -2,982,218 bytes => ~13% smaller.

For performance testing, I tried loading a couple of different 3D models in the STEP file format (~10MB, ~100MB and ~500MB) using this prototype of a 3D viewer. In terms of performance I wasn't able to notice any difference between disabled EH, full Emscipten EH, full Wasm EH and "selective Emscripten EH" (i.e. using -sEXCEPTION_CATCHING_ALLOWED=[...] only where I know that I need it). Last time I investigated Exceptions, I had a 5 times increase in load times when using "full Emscripten EH" vs "selective Emscripten EH". So that's definitely a good thing, but might also be related to some different optimization settings I am using right now (LTO?!). (This is wrong, updated test results below)

The results are great! Can't wait for this to arrive in all major runtimes.

@dschuff
Copy link
Member

dschuff commented Sep 21, 2021

Thanks, please do file a bug (if the LLVM bug tracker gives you trouble, you can file it at https://github.com/emscripten-core/emscripten/issues).
I would find it quite surprising if you don't see much difference between "full" emscripten EH and disabled EH. Although I guess it is plausible that LTO could determine that no exceptions can be thrown on many codepaths, and remove the EH code around many calls.

@donalffons
Copy link

donalffons commented Sep 21, 2021

I just updated my previous post with the results of the base line build - with disabled EH.

I also re-checked the performance results: Turns out that NextJS did some caching magic and I was running all my tests with the same build 😑. Here are the updated and corrected performance test results, measured with a manual stopwatch with an accuracy of ~2 seconds or so. The table shows total loading times in seconds for different models.

Panzerkampfwagen:
2021-09-21_10-32

DS60 - I-Temp
2021-09-21_10-31

1100 GSXR Streetbike
2021-09-21_10-26

  Panzerkampfwagen (17MB) DS60 - I-Temp - Step (100.5MB) 1100 GSXR Streetbike (440MB)
0 Disabled EH 49 uncaught exception uncaught exception
1 Emscripten EH 98 233 1394
2 Wasm EH Aw, Snap! Aw, Snap! Aw, Snap!
3 Selective Emscripten EH 48 88 455

Wasm EH doesn't seem to work in my case. Tested with chrome 92.0.4515.159 with #enable-experimental-webassembly-features enabled.

@donalffons
Copy link

Just had another idea: The Wasm EH version does work without problems (and seemingly fast) when I'm having Chrome's DevTools open during the execution. Since V8's TurboFan compiler is disabled when DevTools are open, this might be an indication that the problem lies there.
I am going to file a bug report about this over at bugs.chromium.org (and link it here).

@aheejin
Copy link
Member

aheejin commented Sep 21, 2021

@donalffons Thanks for your experiments! So apparently there are two problems:

One is the compilation error with LLVM when you are building the library

Unfortunately, I wasn't able to create a full build of the library since I was running into an LLVM error - I might file an issue with them about this.

and the other is a runtime error?

Just had another idea: The Wasm EH version does work without problems (and seemingly fast) when I'm having Chrome's DevTools open during the execution. Since V8's TurboFan compiler is disabled when DevTools are open, this might be an indication that the problem lies there.

For the compilation error, I'd appreciate if you let us know about it at https://github.com/emscripten-core/emscripten/issues. It'd be also great if you CC me there. You can also file a LLVM bug report at https://bugs.llvm.org/, but I think the former is easier to use if you haven't used the LLVM Bugzilla or don't have an account there.

About the runtime error on TurboFan, you can file a bug at https://bugs.chromium.org/p/v8. Please select "WebAssembly" for the component, and you can paste the link here or CC my Chromium account: aheejin@chromium.org

@donalffons
Copy link

@aheejin, here is a link to the issue on the V8 tracker: https://bugs.chromium.org/p/v8/issues/detail?id=12255 (with an example deployment)

About the other (LLVM related) issue: I'm still looking into that. Somehow, I wasn't able to reproduce to reproduce that today - so maybe that was a fluke?! I'll let you know if I find something.

@swang206
Copy link

C++ EH is a mistake. Herbception please https://www.youtube.com/watch?v=ARYP83yNAWk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants