Support running Speedometer 3.0 benchmarks on Chrome #147
Conversation
This version supports running Speedometer 3.0 benchmarks located at https://github.com/WebKit/Speedometer on Chrome. There are two hacks in this change, which can possibly be done better: 1) I didn't find yet an easy way for running-ng terminate Chrome automatically after the benchmark iterations complete. As a workaround, I relied on timeouts to kill Chrome, and used benchmark.is_passed() in minheap.py and runbms.py to figure out if the timeout should be regarded as a success or failure. This leads benchmarks to take longer to complete, because one needs to configure a long enough timeout that guarantees completing the N iterations. 2) I didn't know what is the best way to enforce deleting Chrome's cache between iterations. At the moment the hack I implemented is by adding `rm -rf /tmp/chrome-expr-cache;` in get_log_prologue, and configuring the benchmark to use the same cache directory.
| s = '' | ||
| for j in self.js_args: | ||
| s += j + " " | ||
| cmd.append("--js-flags={}".format(s)) |
There was a problem hiding this comment.
The above can be simplified to cmd.append("--js-flags={}".format(" ".join(self.js_args)))
| @@ -196,6 +196,8 @@ def hz_to_ghz(hzstr: str) -> str: | |||
| def get_log_prologue(runtime: Runtime, bm: Benchmark) -> str: | |||
| output = "\n-----\n" | |||
| output += "mkdir -p PLOTTY_WORKAROUND; timedrun; " | |||
| # Chrome workaround to remove the cache directory between iterations. | |||
| output += "rm -rf /tmp/chrome-expr-cache; " | |||
There was a problem hiding this comment.
This doesn't actually run anything. It gets added to the log file, but that's it. You want to use system to execute commands.
Generally, you probably don't want to do this for all runs (which is the case if it gets put inside get_log_prologue), and probably should avoid hardcoded paths (tempfile.TemporaryDirectory) would be a better choice.
There was a problem hiding this comment.
Note that running has already created a temporary directory for each run.
So you might be able to create a subfolder under it as the temporary directory for Chrome.
There was a problem hiding this comment.
So what might be the cleanest is to use the plugin API to create the temporary folder under runbms_dir for Chrome to use. The plugin object will be created by an instance of Chrome, and will only run when Chrome is used for example.
The plugin object can also delete the folder when the benchmark finishes.
There was a problem hiding this comment.
As you said, there are two general problems.
- There are some additional setup/teardown required.
- Unlike other existing benchmarks, the VM doesn't exit when Speedometer finishes.
1 is easier to solve. You can register a plugin https://github.com/anupli/running-ng/blob/815cf54d96b3b50e9c1b3001e63dd6292bd51373/src/running/command/runbms.py#L459C13-L459C20 to do the setup/cleanup. The plugin API is here https://github.com/anupli/running-ng/blob/815cf54d96b3b50e9c1b3001e63dd6292bd51373/src/running/plugin/runbms/__init__.py
This is similar to how we reuse the modifier API to set heap sizes, which is cleaner than some special case code just to set heap sizes.
2 is trickier. Is it possible for this to be implemented on the VM side so that you can patch the benchmark to call a VM intrinsic (which might not exist) to shutdown the VM?
This version supports running Speedometer 3.0 benchmarks located at https://github.com/WebKit/Speedometer on Chrome.
There are two hacks in this change, which can possibly be done better:
I didn't find yet an easy way for running-ng terminate Chrome automatically after the benchmark iterations complete. As a workaround, I relied on timeouts to kill Chrome, and used benchmark.is_passed() in minheap.py and runbms.py to figure out if the timeout should be regarded as a success or failure. This leads benchmarks to take longer to complete, because one needs to configure a long enough timeout that guarantees completing the N iterations.
I didn't know what is the best way to enforce deleting Chrome's cache between iterations. At the moment the hack I implemented is by adding
rm -rf /tmp/chrome-expr-cache;in get_log_prologue, and configuring the benchmark to use the same cache directory.