Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Don't always mix test score into summary + bug fix in summary score calc
Some tests, such as empty loops, don't represent a structure that
would ever appear in real code but which can be very heavily optimized
by some compilers, thus skewing summary scores unrealistically.  This
change provides a way for tests to be opted-out of the summary score.

Also, fix a bug where the geometric mean would use a power calculated
for *all* tests, not just the ones actually included in the summary
score (which has been inaccurate ever since incomplete results were
auto-skipped).
  • Loading branch information
Geoffrey Broadwell committed Aug 8, 2014
1 parent 87e809d commit 58ca1fa
Showing 1 changed file with 7 additions and 2 deletions.
9 changes: 7 additions & 2 deletions analyze
Expand Up @@ -283,6 +283,7 @@ sub compute_scores {
$score{$_} = 1.0 for @compilers;

my %test_score;
my $summarized = 0;
TEST: for my $test (@$tests) {
my $peak_rate = $test->{compare}{peak_rate};

Expand All @@ -298,18 +299,22 @@ sub compute_scores {
my $reference = $peak_rate->{$standard}{rate};
return unless $reference;

# Determine if test should be included in summary
my $summarize = $test->{conf}{summarize} // 1;
$summarized++ if $summarize;

for my $compiler (@compilers) {
my $rate = $peak_rate->{$compiler}{rate};
$score{$compiler} = undef, next
unless defined $rate && defined $score{$compiler};

my $relative = $rate / $reference;
$score{$compiler} *= $relative;
$score{$compiler} *= $relative if $summarize;
$test_score{$test->{name}}{$compiler} = $relative * 100;
}
}

my $power = 1 / @$tests;
my $power = 1 / ($summarized || 1);
for my $compiler (@compilers) {
next unless defined $score{$compiler};
$score{$compiler} **= $power;
Expand Down

0 comments on commit 58ca1fa

Please sign in to comment.