Multiple measurements per scenario

[GoogleCodeExporter]

Multiple measurements per scenario (same measurer). For purposes of console
display, they can be averaged (?), but the xml result sent to the server
should contain all measurements; it's the job of the visualization package
to decide how to summarize all the data.

Original issue reported on code.google.com by kevin...@gmail.com on 7 Jan 2010 at 10:32

[GoogleCodeExporter]

Original comment by kevinb@google.com on 11 Jan 2010 at 10:15

• Added labels: Milestone-0.5

[GoogleCodeExporter]

I was playing with the visualization of this. I wanted to render a scatter plot 
on the 
server, but unfortunately I ran into an AppEngine bug.
  http://code.google.com/p/googleappengine/issues/detail?id=2645
Should that bug ever get fixed, I've attached the code that'll build the 
appropriate 
scatter plot. Until then, I'm going to experiment with <canvas>

Original comment by jessewil...@google.com on 16 Jan 2010 at 6:44

Attachments:

• scatter.png
• scatterplot.patch

[GoogleCodeExporter]

I've adopted the <canvas> tag to render box plots, but I don't know how I feel 
about 
the UI.

I want something that is:
 - useful in describing the variability of a measurement
 - non intimidating for non-statisticians
 - pretty

What I've got is not particularly stellar in any category, but it's a start. 
Advice is 
welcome; screenshots are attached.

PS - I also updated the client to create multiple measurements; this code is 
horribly 
lame and should be shot.

Original comment by jessewil...@google.com on 17 Jan 2010 at 9:32

Attachments:

• [Picture 75.png](https://storage.googleapis.com/google-code-attachments/caliper/issue-3/comment-3/Picture 75.png)
• [Picture 74.png](https://storage.googleapis.com/google-code-attachments/caliper/issue-3/comment-3/Picture 74.png)

[GoogleCodeExporter]

It works!

Original comment by limpbizkit on 17 Jan 2010 at 8:51

• Changed state: Started

Attachments:

• [Picture 76.png](https://storage.googleapis.com/google-code-attachments/caliper/issue-3/comment-4/Picture 76.png)

[GoogleCodeExporter]

Done.

The only thing I'm unhappy with is the console UI's fidelity in displaying 
multiple 
measurements. Currently it's just displaying the median; but we could do 
something 
fancy (at a risk of being complex)

  foo                     [--|-]----|
  bar                     [-|--]
  baz                        |-[--|--]
  quux             |---[---|--]--|

Original comment by limpbizkit on 18 Jan 2010 at 8:34

• Changed state: Fixed

[GoogleCodeExporter]

Here's a question: why have a console UI at all?  It'll continually be that one 
more 
thing we have to think about (as it is in this example).

Original comment by kevinb@google.com on 19 Jan 2010 at 4:30

[GoogleCodeExporter]

We'll definitely want a way to get good plaintext from benchmark reports, since 
that's 
what can get pasted into code reviews, bug reports, and email messages. I was 
thinking 
the web app should even have a "copy as text" button somewhere.

Original comment by jessewil...@google.com on 19 Jan 2010 at 5:49

[GoogleCodeExporter]

I'm not sure that pasting around text results is such a good idea!  If a url is 
pasted instead, the information can 
be visualized better, can be seen in context with other runs, etc.  I think 
that's the use case we should spend 
time supporting a pleasant, streamlined way.

Original comment by kevinb@google.com on 19 Jan 2010 at 6:30

[GoogleCodeExporter]

> Here's a question: why have a console UI at all?

because there are lots of people who live in the shell.


personally, i'd make a few changes:

1. i wouldn't bother with ASCII-art graphs.

2. i might show scale factors like "2.3x" instead (though this might lead to me
wanting a way to @Annotate the method that represents my baseline).

3. i would always use the same time units. switching between ms and us is all 
well
and good, but it's a pain for this kind of use because it makes the output less
directly comparable.

4. i'd include a timestamp showing when the run was performed, and i'd include a
duration showing how long the benchmark run took.

5. i might include a hash of the test source, so i could easily distinguish 
results i
shouldn't necessarily be comparing.


as for copy & paste, i'll always want to copy & paste some text form, because 
it's
the only thing i trust to last and because it's an extra level of indirection 
(which
is fine for "show more detail" but not for an overview in a check-in comment). 
the
web's a nice optional extra, but that's all.

Original comment by e...@google.com on 19 Jan 2010 at 7:11