Run benchmarks on a fresh install.

[vext01]

Have krun(?) reinstall the machine, install the packages, build the VMs, and run the benchmarks.

[vext01]

Pre-seeding may help: https://wiki.debian.org/DebianInstaller/Preseed

[ltratt]

Yes, between that and the "hands off" documentation, it looks plausible.

[ltratt]

I'm now of the opinion that we might have to set this as not being a blocker for the initial version of the paper, because I'm not even sure how to do it in all situations. @snim2, might you be able to look into the Linux situation? @vext01 and I will do the same for OpenBSD.

[snim2]

I haven't had a thorough look at this yet, but one problem that I came across in PR #74 is that some of the package names contain non-ascii characters, and it's a real pain to try to compare the unicode that comes out of a Json file with a str in such a way that the two can be compared.

[ltratt]

Having talked to @snim2 about this, I don't think it should be krun's job to install packages. I think it should take the machine it's given, record the installed packages and move on.

[A real problem here is the huge difference in OS package granularity. bencher3 (Debian) has 1425 packages installed; bencher6 (OpenBSD) has 116. I know we haven't installed the packages that HHVM would need on bencher6, but still... the difference is gob-smacking and is some way beyond krun's remit.]

[vext01]

I agree that krun should not install packages.

I was under the impression there would be some kind of a "pre-run-script" config option that would deal with reinstalling the OS and reinstalling packages.

A separate issue is how Krun should resume given that the disk it was running on has just been formatted.

[snim2]

Since we have agreed not to have Krun reinstall the machine, I'm closing this issue.

"Resuming given that the disk it was running on has just been formatted" is a separate issue, I think.