Octane-mandreel-latency inappropriately penalizes lazy parsing #29

bhackett1024 opened this Issue May 7, 2015 · 0 comments


None yet
1 participant

AFAIK most web browsers these days use lazy parsing to some degree, where a function is not parsed until it first executes. This is an important feature for improving browser latency and memory usage. The pause time sampling done for mandreel-latency test ends up measuring all the time spent lazy parsing during its first sample. Since there is so much code in the test, parsing alone can take hundreds of milliseconds. For a browser with off thread JIT compilation and incremental GC (like Firefox) the remaining pause times will be tiny, so the benchmark effectively only measures the time spent in that first sample. This time can be easily gamed: by avoiding lazy parsing and doing more work up front (which will hurt browser latency and memory usage) the benchmark score sees a huge boost. v8 seems to be doing this now, as of [1], though the associated bug is not public so I can't really tell what's going on. [2] is a bug I filed against Firefox for this issue, and has more details.

[1] https://chromium.googlesource.com/v8/v8.git/+/be9570027f34f8dbf60a7b530fe2499ff52c1a3d
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1162272

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment