Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Octane-mandreel-latency inappropriately penalizes lazy parsing #29
AFAIK most web browsers these days use lazy parsing to some degree, where a function is not parsed until it first executes. This is an important feature for improving browser latency and memory usage. The pause time sampling done for mandreel-latency test ends up measuring all the time spent lazy parsing during its first sample. Since there is so much code in the test, parsing alone can take hundreds of milliseconds. For a browser with off thread JIT compilation and incremental GC (like Firefox) the remaining pause times will be tiny, so the benchmark effectively only measures the time spent in that first sample. This time can be easily gamed: by avoiding lazy parsing and doing more work up front (which will hurt browser latency and memory usage) the benchmark score sees a huge boost. v8 seems to be doing this now, as of , though the associated bug is not public so I can't really tell what's going on.  is a bug I filed against Firefox for this issue, and has more details.