Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[Speed Comparison] Avoiding JIT warmup #1860
So, I was adding the new Cherow parser to the comparison benchmark, and I noticed that the results didn't match up with my unscientific benchmark "in the wild", which turned out much slower. I believe this is because the benchmark calls the parser many many times, giving the JIT compiler a lot of opportunity to optimize the code, while I only parse a few scripts, and all of them only once.
So, I modified the benchmark to create a new Web Worker for every benchmark cycle, in an attempt to make the code run in a fresh instance of the JIT compiler. (While still only measuring the parse time.) And sure enough, the parse times went up by around 2x, and closer together.
Do you think this is a fair(er) benchmark (or do you think warm JIT is more realistic for most use cases)? (We could also have both, of course.) It might also be useful for the (non-comparison) benchmark suite.
Thanks for looking into this. The Web Workers approach is definitely interesting, feel free to submit that as a pull request and I'd be happy to review it!
As for the differences in the result, it certainly requires a more in-depth analysis. Based on my experience, I highly recommend separating between factual observations (e.g. differences in elapsed execution time) and assumptions (e.g. this is caused by a certain JIT compiler behavior). Taken out of context, mixing the two can result in a lot of future misunderstanding.
Also, the assumption of that JIT compiler behavior needs to be validated anyway. This can be done by looking at the graph and generated code, particularly interesting if it is V8's TurboFan. There could be other reasons as well, e.g. perhaps it is because of the garbage collection?
Please be careful with your particular use of cold JIT vs warm JIT. IMO, it has nothing do with running in a separate thread (via Web Workers) or not. In the compiler field, warm is usually associated with the JIT compiler getting enough work (e.g. due to intensive loops) to start switching to a different optimization mode. A single-threaded JIT compiler can get warm/hot enough after doing a particular processing over and over again.
Instead of distinguishing by cold and warm JIT, I suggest identifying the two different modes as the main browser thread vs its own isolated thread (due to Web Workers).
Looking forward to reviewing your PR. Thank you!
Hi @ariya, thanks for the response.
Yes. this is exactly what I mean by it as well. The single-threaded benchmark I refer to as "warm", and the "cold" benchmark starts a new thread (Web Worker) for every call to
Good point. In fact, it's theoretically possible for a JIT compiler to optimize across threads, in which case the benchmark would be broken since it no longer tests a "cold JIT".
On the other hand, if all the page says is "Run in a single thread" / "Run each parse in a new thread", it might be unclear to some people why you would want to do that. Do you feel that adding an explanation of what the benchmark is supposed to simulate (parsing a file for the 100th time vs parsing it for the 1st time) is appropriate, or would you rather leave that question to someone else to answer?
(This is what I meant earlier by separating between what is being done vs the assumption of what happens.)
Here is a suggestion: just make it a check box that says something along the line of
It's factual (describing exactly what the code does) and it doesn't add any assumption or interpretation.
@twiss I'm closing this issue for now. Feel free to reopen if you think you want to submit a pull request.
referenced this issue
Nov 30, 2017
Right, I forgot about the interpreter stage.
I've sent a PR including your suggestion. However, I still feel that it will be lost on many people why you would want to do this. The description makes it sound like it may have to do something with Web Workers. They might think they should turn the checkbox off because they're not using those. Or, when they observe that with this box checked, the measured times double, they might assume something like "well, maybe Web Workers have some sort of overhead" and again think they should turn it off.
In reality, the Web Workers are just an implementation detail of "separate threads", which in turn is just an implementation detail of "without-an-initially-hot-JIT" which is an implementation detail of "parsing for the first time". There are other possible implementations, e.g. creating iframes instead of Web Workers. I'm not saying that we shouldn't mention Web Workers, just that we should also mention what it's simulating. Something like:
@twiss I'm fine with the elaboration of the meaning of the checkbox. I think it goes a long way of explaining it, and that's a good thing
Minor nitpick: instead of the active "we start..." form, better use the same passive style as the second paragraph ("all benchmark cycles are run..."). We (as a pronoun) is not effective in a documentation, it creates a lot of ambiguity w.r.t who is being referred to.