Support "repeat N times and show the average"

[DarwinAwardWinner]

Startup times can be unstable. It would be nice to be able to repeat the startup several times and take the average to get a more stable estimate (and maybe also the standard deviation).

[DarwinAwardWinner]

I think the first step is to add :nruns 1 to the value returned by esup-profile-sexp, and then implement a esup-merge-results function that takes a list of results for the same sexp and merges them into averages (and standard deviations?).

[jschaf]

I'll brush up on my stats and look into implementations. I think we'd get better results by just running Esup n times, but having the Emacs Frame appear multiple times would get old.

Looking into the issue, I don't think we can just re-benchmark sexps in the same emacs. It won't work for code that only executes once, like require. As an example, eval the following code:

(loop for i from 1 to 5
      collect (benchmark-run (require 'ruby-mode)) into results
      finally return results)

;; This returns the below on my machine:
((0.007 0 0.0) (0.0 0 0.0) (0.0 0 0.0) (0.0 0 0.0) (0.0 0 0.0))

Only the first iteration takes any time.

[DarwinAwardWinner]

The default should probably be to run just once, but for people that want better statistics, running multiple times could be an option.

But honestly, this probably isn't a high-priority thing.

[DarwinAwardWinner]

By the way, another consideration is that some people auto-compile their init files, so it would make sense to have an option to run the init once and throw away the results and then do the real run, to get the results for an already-compiled init.

[jschaf]

So, are you suggesting that we compare the byte-compiled vs interpreted
version? There's emacs-init-time which shows the total init time. But
emacs-init-time includes everything for Emacs. Esup time + Emacs -q time
are usually way less than emacs-init-time and I'm not totally sure why.
I'll take a peek at the internals and see if I can figure something out.

On Mon, Aug 18, 2014 at 7:46 PM, Ryan C. Thompson notifications@github.com
wrote:

By the way, another consideration is that some people auto-compile their
init files, so it would make sense to have an option to run the init once
and throw away the results and then do the real run, to get the results for
an already-compiled init.

—
Reply to this email directly or view it on GitHub
#4 (comment).

[DarwinAwardWinner]

No, I'm just suggesting that it would be useful to be able to run the init process once and throw away the result, and then run it for real a second time and keep the results.

[jschaf]

Okay, I'll take a look next week.

On Mon, Aug 25, 2014 at 6:43 PM, Ryan C. Thompson notifications@github.com
wrote:

No, I'm just suggesting that it would be useful to be able to run the init
process once and throw away the result, and then run it for real a second
time and keep the results.

—
Reply to this email directly or view it on GitHub
#4 (comment).

[jschaf]

So, my week estimate was waaaay off.

I don't think I can implement this without rewriting significant parts of the code and I'm not sure that the added complexity would justify getting an average.

I'm gonna close it, but if there's significant interest I'll take another look.

[DarwinAwardWinner]

Yeah, that's pretty much the conclusion I came to after looking at it. No problem.