Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementing extended options for console formatter #153

Merged
merged 2 commits into from Oct 29, 2017
Merged

Implementing extended options for console formatter #153

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
850ee97
Implementing extended options for console formatter
lwalter Oct 20, 2017
8d87ccf
Modifying extended statistics config key name, cleaning up output log…
lwalter Oct 23, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Expand Up @@ -63,6 +63,12 @@ Provides you with the following **statistical data**:
* **median** - when all measured times are sorted, this is the middle value (or average of the two middle values when the number of samples is even). More stable than the average and somewhat more likely to be a typical value you see. (the lower the better)
* **99th %** - 99th percentile, 99% of all run times are less than this

In addition, you can optionally output an extended set of statistics.
* **minimum** - the smallest (fastest) run time measured for the job
* **maximum** - the biggest (slowest) run time measured for the job
* **sample size** - the number of run time measurements taken
* **mode** - the run time(s) that occur the most. Often one value, but can be multiple values if they occur the same amount of times. If no value occurs at least twice, this value will be nil.

Benchee does not:

* Keep results of previous runs and compare them (yet), if you want that have a look at [benchfella](https://github.com/alco/benchfella) or [bmark](https://github.com/joekain/bmark) until benchee gets that feature :)
Expand Down
39 changes: 20 additions & 19 deletions lib/benchee/configuration.ex
View file
Open in desktop
Expand Up @@ -13,30 +13,31 @@ defmodule Benchee.Configuration do
}

defstruct [
parallel: 1,
time: 5,
warmup: 2,
formatters: [Console],
parallel: 1,
time: 5,
warmup: 2,
formatters: [Console],
print: %{
benchmarking: true,
configuration: true,
fast_warning: true
benchmarking: true,
configuration: true,
fast_warning: true
},
inputs: nil,
inputs: nil,
# formatters should end up here but known once are still picked up at
# the top level for now
formatter_options: %{
console: %{
comparison: true
comparison: true,
extended_statistics: false
}
},
unit_scaling: :best,
unit_scaling: :best,
# If you/your plugin/whatever needs it your data can go here
assigns: %{},
before_each: nil,
after_each: nil,
before_scenario: nil,
after_scenario: nil
assigns: %{},
before_each: nil,
after_each: nil,
before_scenario: nil,
after_scenario: nil
]

@type t :: %__MODULE__{
Expand Down Expand Up @@ -140,7 +141,7 @@ defmodule Benchee.Configuration do
configuration: true
},
formatter_options: %{
console: %{comparison: true}
console: %{comparison: true, extended_statistics: false}
},
unit_scaling: :best,
assigns: %{},
Expand Down Expand Up @@ -168,7 +169,7 @@ defmodule Benchee.Configuration do
configuration: true
},
formatter_options: %{
console: %{comparison: true}
console: %{comparison: true, extended_statistics: false}
},
unit_scaling: :best,
assigns: %{},
Expand Down Expand Up @@ -196,7 +197,7 @@ defmodule Benchee.Configuration do
configuration: true
},
formatter_options: %{
console: %{comparison: true}
console: %{comparison: true, extended_statistics: false}
},
unit_scaling: :best,
assigns: %{},
Expand Down Expand Up @@ -233,7 +234,7 @@ defmodule Benchee.Configuration do
configuration: true
},
formatter_options: %{
console: %{comparison: false},
console: %{comparison: false, extended_statistics: false},
some: "option"
},
unit_scaling: :smallest,
Expand Down
100 changes: 89 additions & 11 deletions lib/benchee/formatters/console.ex
View file
Open in desktop
Expand Up @@ -19,6 +19,10 @@ defmodule Benchee.Formatters.Console do
@deviation_width 11
@median_width 15
@percentile_width 15
@minimum_width 15
@maximum_width 15
@sample_size_width 15
@mode_width 25

@doc """
Formats the benchmark statistics using `Benchee.Formatters.Console.format/1`
Expand Down Expand Up @@ -59,7 +63,7 @@ defmodule Benchee.Formatters.Console do
...> scenarios: scenarios,
...> configuration: %Benchee.Configuration{
...> formatter_options: %{
...> console: %{comparison: false}
...> console: %{comparison: false, extended_statistics: false}
...> },
...> unit_scaling: :best
...> }
Expand Down Expand Up @@ -120,20 +124,26 @@ defmodule Benchee.Formatters.Console do
iex> scenarios = [
...> %Benchee.Benchmark.Scenario{
...> job_name: "My Job", run_time_statistics: %Benchee.Statistics{
...> average: 200.0, ips: 5000.0,std_dev_ratio: 0.1, median: 190.0, percentiles: %{99 => 300.1}
...> average: 200.0, ips: 5000.0,std_dev_ratio: 0.1, median: 190.0, percentiles: %{99 => 300.1},
...> minimum: 100.1, maximum: 200.2, sample_size: 10_101, mode: 333.2
...> }
...> },
...> %Benchee.Benchmark.Scenario{
...> job_name: "Job 2", run_time_statistics: %Benchee.Statistics{
...> average: 400.0, ips: 2500.0, std_dev_ratio: 0.2, median: 390.0, percentiles: %{99 => 500.1}
...> average: 400.0, ips: 2500.0, std_dev_ratio: 0.2, median: 390.0, percentiles: %{99 => 500.1},
...> minimum: 200.2, maximum: 400.4, sample_size: 20_202, mode: [612.3, 554.1]
...> }
...> }
...> ]
iex> configuration = %{comparison: false, unit_scaling: :best}
iex> configuration = %{comparison: false, unit_scaling: :best, extended_statistics: true}
iex> Benchee.Formatters.Console.format_scenarios(scenarios, configuration)
["\nName ips average deviation median 99th %\n",
"My Job 5 K 200 μs ±10.00% 190 μs 300.10 μs\n",
"Job 2 2.50 K 400 μs ±20.00% 390 μs 500.10 μs\n"]
"Job 2 2.50 K 400 μs ±20.00% 390 μs 500.10 μs\n",
"\nExtended statistics: \n",
"\nName minimum maximum sample size mode\n",
"My Job 100.10 μs 200.20 μs 10.10 K 333.20 μs\n",
"Job 2 200.20 μs 400.40 μs 20.20 K 612.30 μs, 554.10 μs\n"]

```

Expand All @@ -147,9 +157,76 @@ defmodule Benchee.Formatters.Console do

[column_descriptors(label_width) |
scenario_reports(sorted_scenarios, units, label_width)
++ comparison_report(sorted_scenarios, units, label_width, config)]
++ comparison_report(sorted_scenarios, units, label_width, config)
++ extended_statistics_report(
sorted_scenarios, units, label_width, config)]
end

@spec extended_statistics_report(
[Scenario.t], unit_per_statistic, integer, map) :: [String.t]
defp extended_statistics_report(_, _, _, %{extended_statistics: false}) do
[]
end
defp extended_statistics_report(scenarios, units, label_width, _config) do
[
descriptor("Extended statistics"),
extended_column_descriptors(label_width) |
extended_statistics(scenarios, units, label_width)
]
end

@spec extended_statistics([Scenario.t], unit_per_statistic, integer)
:: [String.t]
defp extended_statistics(scenarios, units, label_width) do
Enum.map(scenarios, fn(scenario) ->
format_scenario_extended(scenario, units, label_width)
end)
end

@spec format_scenario_extended(Scenario.t, unit_per_statistic, integer)
:: String.t
defp format_scenario_extended(%Scenario{
job_name: name,
run_time_statistics: %Statistics{
minimum: minimum,
maximum: maximum,
sample_size: sample_size,
mode: mode
}
},
%{run_time: run_time_unit},
label_width) do
"~*s~*ts~*ts~*ts~*ts\n"
|> :io_lib.format([
-label_width, name,
@minimum_width, run_time_out(minimum, run_time_unit),
@maximum_width, run_time_out(maximum, run_time_unit),
@sample_size_width, Count.format(sample_size),
@mode_width, mode_out(mode, run_time_unit)])
|> to_string
end

@spec mode_out([number], Benchee.Conversion.Unit.t) :: String.t
defp mode_out(modes, _run_time_unit) when is_nil(modes) do
"None"
end
defp mode_out(modes, run_time_unit) when is_list(modes) do
Enum.map_join(modes, ", ", fn(mode) -> run_time_out(mode, run_time_unit) end)
end
defp mode_out(mode, run_time_unit) when is_number(mode) do
run_time_out(mode, run_time_unit)
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iirc this misses a case, mode can be nil if everything appears exactly the same amount of times (aka modeless). Also needs a test case :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I've added a case where it just outputs the string "None" to be displayed. Is this sufficient or should we remove the mode column altogether maybe?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope None is perfect 👍


@spec extended_column_descriptors(integer) :: String.t
defp extended_column_descriptors(label_width) do
"\n~*s~*s~*s~*s~*s\n"
|> :io_lib.format([-label_width, "Name", @minimum_width, "minimum",
@maximum_width, "maximum", @sample_size_width, "sample size",
@mode_width, "mode"])
|> to_string
end

@spec column_descriptors(integer) :: String.t
defp column_descriptors(label_width) do
"\n~*s~*s~*s~*s~*s~*s\n"
|> :io_lib.format([-label_width, "Name", @ips_width, "ips",
Expand Down Expand Up @@ -202,8 +279,8 @@ defmodule Benchee.Formatters.Console do
Count.format({Count.scale(ips, unit), unit})
end

defp run_time_out(average, unit) do
Duration.format({Duration.scale(average, unit), unit})
defp run_time_out(run_time, unit) do
Duration.format({Duration.scale(run_time, unit), unit})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

end

defp deviation_out(std_dev_ratio) do
Expand All @@ -221,7 +298,7 @@ defmodule Benchee.Formatters.Console do
end
defp comparison_report([scenario | other_scenarios], units, label_width, _) do
[
comparison_descriptor(),
descriptor("Comparison"),
reference_report(scenario, units, label_width) |
comparisons(scenario, units, label_width, other_scenarios)
]
Expand Down Expand Up @@ -256,7 +333,8 @@ defmodule Benchee.Formatters.Console do
|> to_string
end

defp comparison_descriptor do
"\nComparison: \n"
@spec descriptor(String.t) :: String.t
defp descriptor(header_str) do
"\n#{header_str}: \n"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I like the extraction of descriptor :)

end
end
7 changes: 5 additions & 2 deletions test/benchee/configuration_test.exs
View file
Open in desktop
Expand Up @@ -36,7 +36,10 @@ defmodule Benchee.ConfigurationTest do
test "it can be adjusted with a map" do
user_options = %{
time: 10,
formatter_options: %{custom: %{option: true}}
formatter_options: %{
custom: %{option: true},
console: %{extended_statistics: true}
}
}

result = deep_merge(@default_config, user_options)
Expand All @@ -45,7 +48,7 @@ defmodule Benchee.ConfigurationTest do
time: 10,
formatter_options: %{
custom: %{option: true},
console: %{comparison: true}
console: %{comparison: true, extended_statistics: true}
}
}

Expand Down