diff --git a/.tool-versions b/.tool-versions index e4c0cdc1..92183972 100644 --- a/.tool-versions +++ b/.tool-versions @@ -1,2 +1,2 @@ -elixir 1.7.1 +elixir 1.7.3-otp-21 erlang 21.0 diff --git a/.travis.yml b/.travis.yml index 2628a8cc..56142cd8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,29 +1,16 @@ language: elixir elixir: - - 1.4.5 - - 1.5.3 - 1.6.6 - 1.7.4 otp_release: - - 19.3 - 20.3 - 21.1 - -matrix: - exclude: - - elixir: 1.4.5 - otp_release: 21.1 - - elixir: 1.5.3 - otp_release: 21.1 - - elixir: 1.7.4 - otp_release: 19.3 - before_script: - MIX_ENV=test mix compile --warnings-as-errors - travis_wait mix dialyzer --plt script: - mix credo --strict - - if [[ "$TRAVIS_ELIXIR_VERSION" == "1.6"* ]]; then mix format --check-formatted; fi + - mix format --check-formatted - mix dialyzer --halt-exit-status - mix safe_coveralls.travis after_script: diff --git a/README.md b/README.md index c8ad5724..696a1c28 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ Add benchee to your list of dependencies in `mix.exs`: ```elixir defp deps do - [{:benchee, "~> 0.11", only: :dev}] + [{:benchee, "~> 0.13", only: :dev}] end ``` @@ -162,7 +162,7 @@ The available options are the following (also documented in [hexdocs](https://he * `warmup` - the time in seconds for which a benchmarking job should be run without measuring times before "real" measurements start. This simulates a _"warm"_ running system. Defaults to 2. * `time` - the time in seconds for how long each individual benchmarking job should be run for measuring the execution times (run time performance). Defaults to 5. * `memory_time` - the time in seconds for how long [memory measurements](measuring-memory-consumption) should be conducted. Defaults to 0 (turned off). -* `inputs` - a map from descriptive input names to some different input, your benchmarking jobs will then be run with each of these inputs. For this to work your benchmarking function gets the current input passed in as an argument into the function. Defaults to `nil`, aka no input specified and functions are called without an argument. See [Inputs](#inputs). +* `inputs` - a map or list of two element tuples. If a map, they keys are descriptive input names and values are the actual input values. If a list of tuples, the first element in each tuple is the input name, and the second element in each tuple is the actual input value. Your benchmarking jobs will then be run with each of these inputs. For this to work your benchmarking function gets the current input passed in as an argument into the function. Defaults to `nil`, aka no input specified and functions are called without an argument. See [Inputs](#inputs). * `formatters` - list of formatters either as a module implementing the formatter behaviour, a tuple of said module and options it should take or formatter functions. They are run when using `Benchee.run/2` or you can invoktem them through `Benchee.Formatter.output/1`. Functions need to accept one argument (which is the benchmarking suite with all data) and then use that to produce output. Used for plugins. Defaults to the builtin console formatter `Benchee.Formatters.Console`. See [Formatters](#formatters). * `pre_check` - whether or not to run each job with each input - including all given before or after scenario or each hooks - before the benchmarks are measured to ensure that your code executes without error. This can save time while developing your suites. Defaults to `false`. * `parallel` - the function of each benchmarking job will be executed in `parallel` number processes. If `parallel: 4` then 4 processes will be spawned that all execute the _same_ function for the given time. When these finish/the time is up 4 new processes will be spawned for the next job/function. This gives you more data in the same time, but also puts a load on the system interfering with benchmark results. For more on the pros and cons of parallel benchmarking [check the wiki](https://github.com/PragTob/benchee/wiki/Parallel-Benchmarking). Defaults to 1 (no parallel execution). @@ -218,18 +218,26 @@ A full example, including an example of the console output, can be found ### Inputs -`:inputs` is a very useful configuration that allows you to run the same benchmarking jobs with different inputs. You specify the inputs as a map from name (String or atom) to the actual input value. Functions can have different performance characteristics on differently shaped inputs - be that structure or input size. +`:inputs` is a very useful configuration that allows you to run the same benchmarking jobs with different inputs. You specify the inputs as either a map from name (String or atom) to the actual input value or a list of tuples where the first element in each tuple is the name and the second element in the tuple is the value. Functions can have different performance characteristics on differently shaped inputs - be that structure or input size. One of such cases is comparing tail-recursive and body-recursive implementations of `map`. More information in the [repository with the benchmark](https://github.com/PragTob/elixir_playground/blob/master/bench/tco_blog_post_focussed_inputs.exs) and the [blog post](https://pragtob.wordpress.com/2016/06/16/tail-call-optimization-in-elixir-erlang-not-as-efficient-and-important-as-you-probably-think/). ```elixir map_fun = fn(i) -> i + 1 end inputs = %{ - "Small (1 Thousand)" => Enum.to_list(1..1_000), + "Small (1 Thousand)" => Enum.to_list(1..1_000), "Middle (100 Thousand)" => Enum.to_list(1..100_000), - "Big (10 Million)" => Enum.to_list(1..10_000_000), + "Big (10 Million)" => Enum.to_list(1..10_000_000) } +# Or inputs could also look like this: +# +# inputs = [ +# {"Small (1 Thousand)", Enum.to_list(1..1_000)}, +# {"Middle (100 Thousand)", Enum.to_list(1..100_000)}, +# {"Big (10 Million)", Enum.to_list(1..10_000_000)} +# ] + Benchee.run %{ "map tail-recursive" => fn(list) -> MyMap.map_tco(list, map_fun) end, diff --git a/lib/benchee.ex b/lib/benchee.ex index 41fd6e08..d576d702 100644 --- a/lib/benchee.ex +++ b/lib/benchee.ex @@ -20,35 +20,27 @@ for {module, moduledoc} <- [{Benchee, elixir_doc}, {:benchee, erlang_doc}] do alias Benchee.Formatter @doc """ - Run benchmark jobs defined by a map and optionally provide configuration - options. + Runs the given benchmarks, calculates statistics based on the results and + outputs results with the configured formatters. - Runs the given benchmarks and prints the results on the console. - - * jobs - a map from descriptive benchmark job name to a function to be - executed and benchmarked - * configuration - configuration options to alter what Benchee does, see - `Benchee.Configuration.init/1` for documentation of the available options. + Benchmarks are defined as a map where the keys are a name for the given + function and the values are the functions to benchmark. Users can configure + the run by passing a keyword list as the second argument. For more + information on configuration see `Benchee.Configuration.init/1`. ## Examples - Benchee.run(%{"My Benchmark" => fn -> 1 + 1 end, - "My other benchmrk" => fn -> "1" ++ "1" end}, time: 3) - # Prints a summary of the benchmark to the console - + Benchee.run( + %{ + "My Benchmark" => fn -> 1 + 1 end, + "My other benchmrk" => fn -> [1] ++ [1] end + }, + warmup: 2, + time: 3 + ) """ - def run(jobs, config \\ []) - - def run(jobs, config) when is_list(config) do - do_run(jobs, config) - end - - def run(config, jobs) when is_map(jobs) do - # pre 0.6.0 way of passing in the config first and as a map - do_run(jobs, config) - end - - defp do_run(jobs, config) do + @spec run(map, keyword) :: any + def run(jobs, config \\ []) when is_list(config) do config |> Benchee.init() |> Benchee.system() @@ -68,11 +60,11 @@ for {module, moduledoc} <- [{Benchee, elixir_doc}, {:benchee, erlang_doc}] do defdelegate init(), to: Benchee.Configuration defdelegate init(config), to: Benchee.Configuration defdelegate system(suite), to: Benchee.System + defdelegate benchmark(suite, name, function), to: Benchee.Benchmark + defdelegate benchmark(suite, name, function, printer), to: Benchee.Benchmark defdelegate measure(suite), to: Benchee.Benchmark defdelegate measure(suite, printer), to: Benchee.Benchmark - defdelegate benchmark(suite, name, function), to: Benchee.Benchmark defdelegate statistics(suite), to: Benchee.Statistics defdelegate load(suite), to: Benchee.ScenarioLoader - defdelegate benchmark(suite, name, function, printer), to: Benchee.Benchmark end end diff --git a/lib/benchee/benchmark.ex b/lib/benchee/benchmark.ex index 26c30e07..7ed18410 100644 --- a/lib/benchee/benchmark.ex +++ b/lib/benchee/benchmark.ex @@ -4,10 +4,9 @@ defmodule Benchee.Benchmark do Exposes `benchmark/4` and `measure/3` functions. """ - alias Benchee.Benchmark.{Scenario, ScenarioContext, Runner} + alias Benchee.Benchmark.{Runner, Scenario, ScenarioContext} alias Benchee.Output.BenchmarkPrinter, as: Printer - alias Benchee.Suite - alias Benchee.Utility.DeepConvert + alias Benchee.{Suite, Utility.DeepConvert} @type job_name :: String.t() | atom @no_input :__no_input @@ -48,19 +47,6 @@ defmodule Benchee.Benchmark do %Suite{suite | scenarios: List.flatten([scenarios | new_scenarios])} end - defp build_scenarios_for_job(job_name, function, config) - - defp build_scenarios_for_job(job_name, function, nil) do - [ - build_scenario(%{ - job_name: job_name, - function: function, - input: @no_input, - input_name: @no_input - }) - ] - end - defp build_scenarios_for_job(job_name, function, %{inputs: nil}) do [ build_scenario(%{ diff --git a/lib/benchee/benchmark/repeated_measurement.ex b/lib/benchee/benchmark/repeated_measurement.ex index ff96fa35..9926bf84 100644 --- a/lib/benchee/benchmark/repeated_measurement.ex +++ b/lib/benchee/benchmark/repeated_measurement.ex @@ -21,7 +21,7 @@ defmodule Benchee.Benchmark.RepeatedMeasurement do # with too high variance. Therefore determine an n how often it should be # executed in the measurement cycle. - alias Benchee.Benchmark.{Hooks, Runner, Scenario, ScenarioContext, Measure} + alias Benchee.Benchmark.{Hooks, Measure, Runner, Scenario, ScenarioContext} alias Benchee.Utility.RepeatN @minimum_execution_time 10 diff --git a/lib/benchee/benchmark/runner.ex b/lib/benchee/benchmark/runner.ex index 370acea9..c2eed0ce 100644 --- a/lib/benchee/benchmark/runner.ex +++ b/lib/benchee/benchmark/runner.ex @@ -4,12 +4,8 @@ defmodule Benchee.Benchmark.Runner do # This module actually runs our benchmark scenarios, adding information about # run time and memory usage to each scenario. - alias Benchee.Benchmark - alias Benchee.Benchmark.{Scenario, ScenarioContext, Measure, Hooks, RepeatedMeasurement} - alias Benchee.Configuration - alias Benchee.Conversion - alias Benchee.Statistics - alias Benchee.Utility.Parallel + alias Benchee.{Benchmark, Configuration, Conversion, Statistics, Utility.Parallel} + alias Benchmark.{Hooks, Measure, RepeatedMeasurement, Scenario, ScenarioContext} @doc """ Executes the benchmarks defined before by first running the defined functions diff --git a/lib/benchee/configuration.ex b/lib/benchee/configuration.ex index c3b312a6..1694dcba 100644 --- a/lib/benchee/configuration.ex +++ b/lib/benchee/configuration.ex @@ -4,12 +4,12 @@ defmodule Benchee.Configuration do """ alias Benchee.{ - Suite, Configuration, Conversion.Duration, Conversion.Scale, - Utility.DeepConvert, - Formatters.Console + Formatters.Console, + Suite, + Utility.DeepConvert } defstruct parallel: 1, @@ -17,7 +17,7 @@ defmodule Benchee.Configuration do warmup: 2, memory_time: 0.0, pre_check: false, - formatters: [Console], + formatters: [{Console, %{comparison: true, extended_statistics: false}}], percentiles: [50, 99], print: %{ benchmarking: true, @@ -27,16 +27,7 @@ defmodule Benchee.Configuration do inputs: nil, save: false, load: false, - # formatters should end up here but known once are still picked up at - # the top level for now - formatter_options: %{ - console: %{ - comparison: true, - extended_statistics: false - } - }, unit_scaling: :best, - # If you/your plugin/whatever needs it your data can go here assigns: %{}, before_each: nil, after_each: nil, @@ -51,12 +42,11 @@ defmodule Benchee.Configuration do warmup: number, memory_time: number, pre_check: boolean, - formatters: [(Suite.t() -> Suite.t())], + formatters: [(Suite.t() -> Suite.t()) | {atom, map}], print: map, - inputs: %{Suite.key() => any} | nil, + inputs: %{Suite.key() => any} | [{String.t(), any}] | nil, save: map | false, load: String.t() | [String.t()] | false, - formatter_options: map, unit_scaling: Scale.scaling_strategy(), assigns: map, before_each: fun | nil, @@ -263,9 +253,7 @@ defmodule Benchee.Configuration do ...> warmup: 0.2, ...> formatters: [&IO.puts/1], ...> print: [fast_warning: false], - ...> console: [comparison: false], ...> inputs: %{"Small" => 5, "Big" => 9999}, - ...> formatter_options: [some: "option"], ...> unit_scaling: :smallest) %Benchee.Suite{ configuration: @@ -273,7 +261,7 @@ defmodule Benchee.Configuration do parallel: 2, time: 1_000_000_000.0, warmup: 200_000_000.0, - inputs: %{"Small" => 5, "Big" => 9999}, + inputs: [{"Big", 9999}, {"Small", 5}], save: false, load: false, formatters: [&IO.puts/1], @@ -282,13 +270,6 @@ defmodule Benchee.Configuration do fast_warning: false, configuration: true }, - formatter_options: %{ - console: %{ - comparison: false, - extended_statistics: false - }, - some: "option" - }, percentiles: [50, 99], unit_scaling: :smallest, assigns: %{}, @@ -309,7 +290,6 @@ defmodule Benchee.Configuration do config |> standardized_user_configuration |> merge_with_defaults - |> formatter_options_to_tuples |> convert_time_to_nano_s |> update_measure_memory |> save_option_conversion @@ -319,45 +299,23 @@ defmodule Benchee.Configuration do defp standardized_user_configuration(config) do config - |> DeepConvert.to_map([:formatters]) - |> translate_formatter_keys + |> DeepConvert.to_map([:formatters, :inputs]) |> force_string_input_keys end - # backwards compatible translation of formatter keys to go into - # formatter_options now - @formatter_keys [:console, :csv, :json, :html] - defp translate_formatter_keys(config) do - {formatter_options, config} = Map.split(config, @formatter_keys) - DeepMerge.deep_merge(%{formatter_options: formatter_options}, config) - end - - alias Benchee.Formatters.{Console, CSV, JSON, HTML} - - # backwards compatible formatter definition without leaving the burden on every formatter - defp formatter_options_to_tuples(config) do - update_in(config, [Access.key(:formatters), Access.all()], fn - Console -> formatter_configuration_from_options(config, Console, :console) - CSV -> formatter_configuration_from_options(config, CSV, :csv) - JSON -> formatter_configuration_from_options(config, JSON, :json) - HTML -> formatter_configuration_from_options(config, HTML, :html) - formatter -> formatter - end) - end - - defp formatter_configuration_from_options(config, module, legacy_option_key) do - if Map.has_key?(config.formatter_options, legacy_option_key) do - {module, config.formatter_options[legacy_option_key]} - else - module - end - end - defp force_string_input_keys(config = %{inputs: inputs}) do standardized_inputs = - for {name, value} <- inputs, into: %{} do - {to_string(name), value} - end + inputs + |> Enum.reduce([], fn {name, value}, acc -> + normalized_name = to_string(name) + + if List.keymember?(acc, normalized_name, 0) do + acc + else + [{normalized_name, value} | acc] + end + end) + |> Enum.reverse() %{config | inputs: standardized_inputs} end @@ -400,22 +358,13 @@ defmodule Benchee.Configuration do """) end - defp save_option_conversion(config = %{save: false}) do - config - end + defp save_option_conversion(config = %{save: false}), do: config defp save_option_conversion(config = %{save: save_values}) do save_options = Map.merge(save_defaults(), save_values) - - tagged_save_options = %{ - tag: save_options.tag, - path: save_options.path - } - - %__MODULE__{ - config - | formatters: config.formatters ++ [{Benchee.Formatters.TaggedSave, tagged_save_options}] - } + tagged_save_options = %{tag: save_options.tag, path: save_options.path} + formatters = config.formatters ++ [{Benchee.Formatters.TaggedSave, tagged_save_options}] + %__MODULE__{config | formatters: formatters} end defp save_defaults do diff --git a/lib/benchee/conversion.ex b/lib/benchee/conversion.ex index ff6d9e5e..8d5fea0a 100644 --- a/lib/benchee/conversion.ex +++ b/lib/benchee/conversion.ex @@ -6,7 +6,7 @@ defmodule Benchee.Conversion do """ alias Benchee.Benchmark.Scenario - alias Benchee.Conversion.{Duration, Count, Memory} + alias Benchee.Conversion.{Count, Duration, Memory} @doc """ Takes scenarios and a given scaling_strategy, returns the best units for the diff --git a/lib/benchee/formatter.ex b/lib/benchee/formatter.ex index 8ff483fe..8bca47a5 100644 --- a/lib/benchee/formatter.ex +++ b/lib/benchee/formatter.ex @@ -32,8 +32,6 @@ defmodule Benchee.Formatter do """ @callback write(any, options) :: :ok | {:error, String.t()} - @typep module_configuration :: module | {module, options} - @doc """ Format and output all configured formatters and formatting functions. @@ -54,7 +52,7 @@ defmodule Benchee.Formatter do {parallelizable, serial} = formatters |> Enum.map(&normalize_module_configuration/1) - |> Enum.split_with(&is_formatter_module?/1) + |> Enum.split_with(&is_tuple/1) # why do we ignore this suite? It shouldn't be changed anyway. # We assign it because dialyzer would complain otherwise :D @@ -65,25 +63,38 @@ defmodule Benchee.Formatter do suite end - @default_opts %{} - defp normalize_module_configuration(module_configuration) - defp normalize_module_configuration({module, opts}), do: {module, DeepConvert.to_map(opts)} + defp normalize_module_configuration(formatter) when is_function(formatter, 1), do: formatter - defp normalize_module_configuration(formatter) when is_atom(formatter) do - {formatter, @default_opts} + defp normalize_module_configuration({module, opts}) do + normalize_module_configuration(module, DeepConvert.to_map(opts)) end - defp normalize_module_configuration(formatter), do: formatter + defp normalize_module_configuration(module) when is_atom(module) do + normalize_module_configuration(module, %{}) + end - defp is_formatter_module?({formatter, _options}) when is_atom(formatter) do - module_attributes = formatter.module_info(:attributes) + defp normalize_module_configuration(module, opts) do + if formatter_module?(module) do + {module, opts} + else + raise_behaviour_not_implemented(module) + end + end - module_attributes + defp formatter_module?(module) do + :attributes + |> module.module_info() |> Keyword.get(:behaviour, []) |> Enum.member?(Benchee.Formatter) end - defp is_formatter_module?(_), do: false + @spec raise_behaviour_not_implemented(atom) :: no_return() + defp raise_behaviour_not_implemented(module) do + raise """ + The module you're attempting to use as a formatter - #{module} - does + not implement the `Benchee.Formatter` behaviour. + """ + end @doc """ Output a suite with a given formatter and options. @@ -105,7 +116,6 @@ defmodule Benchee.Formatter do # Invokes `format/2` and `write/2` as defined by the `Benchee.Formatter` # behaviour. The output for all formatters is generated in parallel, and then # the results of that formatting are written in sequence. - @spec parallel_output(Suite.t(), [module_configuration]) :: Suite.t() defp parallel_output(suite, module_configurations) do module_configurations |> Parallel.map(fn {module, options} -> {module, options, module.format(suite, options)} end) diff --git a/lib/benchee/formatters/console.ex b/lib/benchee/formatters/console.ex index 652ec3d8..54df9a8d 100644 --- a/lib/benchee/formatters/console.ex +++ b/lib/benchee/formatters/console.ex @@ -70,7 +70,17 @@ defmodule Benchee.Formatters.Console do |> Map.merge(options) scenarios - |> Enum.group_by(fn scenario -> scenario.input_name end) + |> Enum.reduce([], fn scenario, grouped -> + case List.keyfind(grouped, scenario.input_name, 0) do + {_, group} -> + new_tuple = {scenario.input_name, [scenario | group]} + List.keyreplace(grouped, scenario.input_name, 0, new_tuple) + + _ -> + [{scenario.input_name, [scenario]} | grouped] + end + end) + |> Enum.reverse() |> Enum.map(fn {input, scenarios} -> scenarios |> Statistics.sort() diff --git a/lib/benchee/formatters/console/memory.ex b/lib/benchee/formatters/console/memory.ex index edc787a9..3baaa65c 100644 --- a/lib/benchee/formatters/console/memory.ex +++ b/lib/benchee/formatters/console/memory.ex @@ -9,9 +9,9 @@ defmodule Benchee.Formatters.Console.Memory do alias Benchee.{ Benchmark.Scenario, Conversion, - Conversion.Unit, Conversion.Count, Conversion.Memory, + Conversion.Unit, Formatters.Console.Helpers, Statistics } diff --git a/lib/benchee/formatters/console/run_time.ex b/lib/benchee/formatters/console/run_time.ex index d8602de0..dd8b6043 100644 --- a/lib/benchee/formatters/console/run_time.ex +++ b/lib/benchee/formatters/console/run_time.ex @@ -10,8 +10,8 @@ defmodule Benchee.Formatters.Console.RunTime do Benchmark.Scenario, Conversion, Conversion.Count, - Conversion.Unit, Conversion.Duration, + Conversion.Unit, Formatters.Console.Helpers, Statistics } diff --git a/lib/benchee/output/benchmark_printer.ex b/lib/benchee/output/benchmark_printer.ex index ee93ac15..b3816a8b 100644 --- a/lib/benchee/output/benchmark_printer.ex +++ b/lib/benchee/output/benchmark_printer.ex @@ -72,7 +72,7 @@ defmodule Benchee.Output.BenchmarkPrinter do defp inputs_out(inputs) do inputs - |> Map.keys() + |> Enum.map(fn {name, _} -> name end) |> Enum.join(", ") end diff --git a/mix.exs b/mix.exs index 7f228d08..5be64fd3 100644 --- a/mix.exs +++ b/mix.exs @@ -51,13 +51,12 @@ defmodule Benchee.Mixfile do [ {:deep_merge, "~> 0.1"}, {:ex_guard, "~> 1.3", only: :dev}, - {:credo, "< 0.9.3", only: :dev}, - # newer versions break in older elixirs + {:credo, "~> 1.0.0", only: :dev}, {:ex_doc, "~> 0.18.0", only: :dev}, {:earmark, "~> 1.0", only: :dev}, {:excoveralls, "~> 0.7", only: :test}, {:inch_ex, "~> 0.5", only: :docs}, - {:dialyxir, "~> 0.5", only: :dev, runtime: false} + {:dialyxir, "~> 1.0.0-rc.4", only: :dev, runtime: false} ] end diff --git a/mix.lock b/mix.lock index a349b475..e3879dd3 100644 --- a/mix.lock +++ b/mix.lock @@ -1,13 +1,14 @@ %{ "bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm"}, "certifi": {:hex, :certifi, "2.4.2", "75424ff0f3baaccfd34b1214184b6ef616d89e420b258bb0a5ea7d7bc628f7f0", [:rebar3], [{:parse_trans, "~>3.3", [hex: :parse_trans, repo: "hexpm", optional: false]}], "hexpm"}, - "credo": {:hex, :credo, "0.9.2", "841d316612f568beb22ba310d816353dddf31c2d94aa488ae5a27bb53760d0bf", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:poison, ">= 0.0.0", [hex: :poison, repo: "hexpm", optional: false]}], "hexpm"}, + "credo": {:hex, :credo, "1.0.0", "aaa40fdd0543a0cf8080e8c5949d8c25f0a24e4fc8c1d83d06c388f5e5e0ea42", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"}, "deep_merge": {:hex, :deep_merge, "0.2.0", "c1050fa2edf4848b9f556fba1b75afc66608a4219659e3311d9c9427b5b680b3", [:mix], [], "hexpm"}, - "dialyxir": {:hex, :dialyxir, "0.5.1", "b331b091720fd93e878137add264bac4f644e1ddae07a70bf7062c7862c4b952", [:mix], [], "hexpm"}, - "earmark": {:hex, :earmark, "1.2.6", "b6da42b3831458d3ecc57314dff3051b080b9b2be88c2e5aa41cd642a5b044ed", [:mix], [], "hexpm"}, + "dialyxir": {:hex, :dialyxir, "1.0.0-rc.4", "71b42f5ee1b7628f3e3a6565f4617dfb02d127a0499ab3e72750455e986df001", [:mix], [{:erlex, "~> 0.1", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm"}, + "earmark": {:hex, :earmark, "1.3.0", "17f0c38eaafb4800f746b457313af4b2442a8c2405b49c645768680f900be603", [:mix], [], "hexpm"}, + "erlex": {:hex, :erlex, "0.1.6", "c01c889363168d3fdd23f4211647d8a34c0f9a21ec726762312e08e083f3d47e", [:mix], [], "hexpm"}, "ex_doc": {:hex, :ex_doc, "0.18.4", "4406b8891cecf1352f49975c6d554e62e4341ceb41b9338949077b0d4a97b949", [:mix], [{:earmark, "~> 1.1", [hex: :earmark, repo: "hexpm", optional: false]}], "hexpm"}, "ex_guard": {:hex, :ex_guard, "1.3.3", "941acb390979a1ef40ed0ef10d07035ce7d98b9d0089cecca132f9c0744ac620", [:mix], [{:fs, "~> 0.9", [hex: :fs, repo: "hexpm", optional: false]}], "hexpm"}, - "excoveralls": {:hex, :excoveralls, "0.10.2", "fb4abd5b8a1b9d52d35e1162e7e2ea8bfb84b47ae07c38d39aa8ce64be0b0794", [:mix], [{:hackney, "~> 1.13", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"}, + "excoveralls": {:hex, :excoveralls, "0.10.3", "b090a3fbcb3cfa136f0427d038c92a9051f840953ec11b40ee74d9d4eac04d1e", [:mix], [{:hackney, "~> 1.13", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"}, "fs": {:hex, :fs, "0.9.2", "ed17036c26c3f70ac49781ed9220a50c36775c6ca2cf8182d123b6566e49ec59", [:rebar], [], "hexpm"}, "hackney": {:hex, :hackney, "1.14.3", "b5f6f5dcc4f1fba340762738759209e21914516df6be440d85772542d4a5e412", [:rebar3], [{:certifi, "2.4.2", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "6.0.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "1.0.1", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "1.0.2", [hex: :mimerl, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "1.1.4", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm"}, "idna": {:hex, :idna, "6.0.0", "689c46cbcdf3524c44d5f3dde8001f364cd7608a99556d8fbd8239a5798d4c10", [:rebar3], [{:unicode_util_compat, "0.4.1", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm"}, diff --git a/test/benchee/benchmark/runner_test.exs b/test/benchee/benchmark/runner_test.exs index 477ca8ab..13f8f9d5 100644 --- a/test/benchee/benchmark/runner_test.exs +++ b/test/benchee/benchmark/runner_test.exs @@ -1,9 +1,10 @@ defmodule Benchee.Benchmark.RunnerTest do use ExUnit.Case, async: true + import Benchee.TestHelpers import ExUnit.CaptureIO - alias Benchee.{Suite, Benchmark, Configuration} - alias Benchee.Benchmark.Scenario + + alias Benchee.{Benchmark, Benchmark.Scenario, Configuration, Suite} alias Benchee.Test.FakeBenchmarkPrinter, as: TestPrinter @config %Configuration{ diff --git a/test/benchee/benchmark_test.exs b/test/benchee/benchmark_test.exs index 70891048..1f66321f 100644 --- a/test/benchee/benchmark_test.exs +++ b/test/benchee/benchmark_test.exs @@ -1,11 +1,16 @@ defmodule Benchee.BenchmarkTest do use ExUnit.Case, async: true - alias Benchee.Benchmark - alias Benchee.Configuration - alias Benchee.Benchmark.{Scenario, ScenarioContext} + + alias Benchee.{ + Benchmark, + Benchmark.Scenario, + Benchmark.ScenarioContext, + Configuration, + Suite + } + alias Benchee.Test.FakeBenchmarkPrinter, as: TestPrinter alias Benchee.Test.FakeBenchmarkRunner, as: TestRunner - alias Benchee.Suite describe ".benchmark" do test "can add jobs with atom keys but converts them to string" do diff --git a/test/benchee/configuration_test.exs b/test/benchee/configuration_test.exs index 339a38ef..40cf9b08 100644 --- a/test/benchee/configuration_test.exs +++ b/test/benchee/configuration_test.exs @@ -9,103 +9,60 @@ defmodule Benchee.ConfigurationTest do @default_config %Configuration{} - describe ".init/1" do - test "it crashes for values that are going to be ignored" do - assert_raise KeyError, fn -> - init(runntime: 2) - end + describe "init/1" do + test "crashes for values that are going to be ignored" do + assert_raise KeyError, fn -> init(runntime: 2) end end - test "it converts input keys to strings" do - suite = init(inputs: %{"map" => %{}, list: []}) - - assert %Suite{ - configuration: %{inputs: %{"list" => [], "map" => %{}}} - } = suite + test "converts maps to lists and input keys to strings" do + assert %Suite{configuration: %{inputs: [{"list", []}, {"map", %{}}]}} = + init(inputs: %{"map" => %{}, list: []}) end - test "it loses duplicated inputs keys after normalization" do - suite = init(inputs: %{"map" => %{}, map: %{}}) + test "doesn't convert input lists to maps and retains the order of input lists" do + assert %Suite{configuration: %{inputs: [{"map", %{}}, {"list", []}]}} = + init(inputs: [{"map", %{}}, {:list, []}]) + end - assert %Suite{configuration: %{inputs: inputs}} = suite - assert %{"map" => %{}} == inputs + test "loses duplicated inputs keys after normalization" do + assert %Suite{configuration: %{inputs: [{"map", %{}}]}} = + init(inputs: %{"map" => %{}, map: %{}}) end test "uses information from :save to setup the external term formattter" do - suite = init(save: [path: "save_one.benchee", tag: "master"]) - - assert suite.configuration.formatters == [ - {Benchee.Formatters.Console, %{comparison: true, extended_statistics: false}}, - {Benchee.Formatters.TaggedSave, %{path: "save_one.benchee", tag: "master"}} - ] + assert %Suite{ + configuration: %{ + formatters: [ + {Benchee.Formatters.Console, %{comparison: true, extended_statistics: false}}, + {Benchee.Formatters.TaggedSave, %{path: "save_one.benchee", tag: "master"}} + ] + } + } = init(save: [path: "save_one.benchee", tag: "master"]) end test ":save tag defaults to date" do - suite = init(save: [path: "save_one.benchee"]) - - [_, {_, etf_options}] = suite.configuration.formatters - - assert etf_options.tag =~ ~r/\d\d\d\d-\d\d?-\d\d?--\d\d?-\d\d?-\d\d?/ - assert etf_options.path == "save_one.benchee" - end - - test "takes formatter_options to build tuple list" do - suite = - init( - formatter_options: %{console: %{foo: :bar}}, - formatters: [Benchee.Formatters.Console] - ) - - assert [{Benchee.Formatters.Console, %{foo: :bar}}] = suite.configuration.formatters - end - - test "formatters already specified as a tuple are left alone" do - suite = - init( - formatter_options: %{console: %{foo: :bar}}, - formatters: [{Benchee.Formatters.Console, %{a: :b}}] - ) - - assert [{Benchee.Formatters.Console, %{a: :b}}] == suite.configuration.formatters - end - - test "legacy formatter options default to just the module if no options are given" do - suite = init(formatters: [Benchee.Formatter.CSV]) + assert %Suite{configuration: %{formatters: [_, {_, %{tag: tag, path: "save_one.benchee"}}]}} = + init(save: [path: "save_one.benchee"]) - assert [Benchee.Formatter.CSV] == suite.configuration.formatters + assert tag =~ ~r/\d\d\d\d-\d\d?-\d\d?--\d\d?-\d\d?-\d\d?/ end end describe ".deep_merge behaviour" do test "it can be adjusted with a map" do - user_options = %{ - time: 10, - formatter_options: %{ - custom: %{option: true}, - console: %{extended_statistics: true} - } - } + user_options = %{time: 10} result = deep_merge(@default_config, user_options) - expected = %Configuration{ - time: 10, - formatter_options: %{ - custom: %{option: true}, - console: %{ - comparison: true, - extended_statistics: true - } - } - } + expected = %Configuration{time: 10} assert expected == result end test "it just replaces when given another configuration" do - other_config = %Configuration{formatter_options: %{some: %{value: true}}} + other_config = %Configuration{} result = deep_merge(@default_config, other_config) - expected = %Configuration{formatter_options: %{some: %{value: true}}} + expected = %Configuration{} assert ^expected = result end diff --git a/test/benchee/formatter_test.exs b/test/benchee/formatter_test.exs index 00d0d4cb..371ba3b5 100644 --- a/test/benchee/formatter_test.exs +++ b/test/benchee/formatter_test.exs @@ -1,6 +1,6 @@ defmodule Benchee.FormatterTest do use ExUnit.Case, async: true - alias Benchee.{Suite, Formatter, Test.FakeFormatter} + alias Benchee.{Formatter, Suite, Test.FakeFormatter} describe "output/1" do test "calls `write/1` with the output of `format/1` on each module" do diff --git a/test/benchee/formatters/console/memory_test.exs b/test/benchee/formatters/console/memory_test.exs index 0273fdce..eaf1714c 100644 --- a/test/benchee/formatters/console/memory_test.exs +++ b/test/benchee/formatters/console/memory_test.exs @@ -2,7 +2,7 @@ defmodule Benchee.Formatters.Console.MemoryTest do use ExUnit.Case, async: true doctest Benchee.Formatters.Console.Memory - alias Benchee.{Formatters.Console.Memory, Statistics, Benchmark.Scenario} + alias Benchee.{Benchmark.Scenario, Formatters.Console.Memory, Statistics} @console_config %{ comparison: true, diff --git a/test/benchee/formatters/console/run_time_test.exs b/test/benchee/formatters/console/run_time_test.exs index faa9cc98..d55dd6a5 100644 --- a/test/benchee/formatters/console/run_time_test.exs +++ b/test/benchee/formatters/console/run_time_test.exs @@ -2,7 +2,7 @@ defmodule Benchee.Formatters.Console.RunTimeTest do use ExUnit.Case, async: true doctest Benchee.Formatters.Console.RunTime - alias Benchee.{Formatters.Console.RunTime, Statistics, Benchmark.Scenario} + alias Benchee.{Benchmark.Scenario, Formatters.Console.RunTime, Statistics} @console_config %{ comparison: true, diff --git a/test/benchee/formatters/console_test.exs b/test/benchee/formatters/console_test.exs index 1d9ab6f7..78bfe851 100644 --- a/test/benchee/formatters/console_test.exs +++ b/test/benchee/formatters/console_test.exs @@ -3,8 +3,7 @@ defmodule Benchee.Formatters.ConsoleTest do doctest Benchee.Formatters.Console import ExUnit.CaptureIO - alias Benchee.{Formatters.Console, Suite, Statistics, Benchmark.Scenario} - alias Benchee.Formatter + alias Benchee.{Benchmark.Scenario, Formatter, Formatters.Console, Statistics, Suite} @config %Benchee.Configuration{ title: "A comprehensive benchmarking of inputs" @@ -68,7 +67,7 @@ defmodule Benchee.Formatters.ConsoleTest do end end - describe ".format" do + describe "format/2" do @header_regex ~r/Name.+ips.+average.+deviation.+median.+99th %.*/ test "with multiple inputs and just one job" do scenarios = [ @@ -116,6 +115,52 @@ defmodule Benchee.Formatters.ConsoleTest do assert result_2 =~ ~r/Job.+2\.5.+400.+15\.00%.+395.+500\.1/ end + test "retains the order of scenarios" do + scenarios = [ + %Scenario{ + name: "Job", + input_name: "Other Arg", + input: "Other Arg", + run_time_statistics: %Statistics{ + average: 400.0, + ips: 2_500.0, + std_dev_ratio: 0.15, + median: 395.0, + percentiles: %{99 => 500.1}, + sample_size: 200 + }, + memory_usage_statistics: %Statistics{} + }, + %Scenario{ + name: "Job", + input_name: "My Arg", + input: "My Arg", + run_time_statistics: %Statistics{ + average: 200.0, + ips: 5_000.0, + std_dev_ratio: 0.1, + median: 195.5, + percentiles: %{99 => 400.1}, + sample_size: 200 + }, + memory_usage_statistics: %Statistics{} + } + ] + + [other_arg, my_arg] = + Console.format(%Suite{scenarios: scenarios, configuration: @config}, @options) + + [input_header, header, result] = my_arg + assert input_header =~ "My Arg" + assert header =~ @header_regex + assert result =~ ~r/Job.+5.+200.+10\.00%.+195\.5.+400\.1/ + + [input_header_2, header_2, result_2] = other_arg + assert input_header_2 =~ "Other Arg" + assert header_2 =~ @header_regex + assert result_2 =~ ~r/Job.+2\.5.+400.+15\.00%.+395.+500\.1/ + end + test "with multiple inputs and two jobs" do scenarios = [ %Scenario{ diff --git a/test/benchee/formatters/tagged_save_test.exs b/test/benchee/formatters/tagged_save_test.exs index c08f855e..848b257a 100644 --- a/test/benchee/formatters/tagged_save_test.exs +++ b/test/benchee/formatters/tagged_save_test.exs @@ -1,12 +1,15 @@ defmodule Benchee.Formatters.TaggedSaveTest do use ExUnit.Case - alias Benchee.{Suite, Statistics} - alias Benchee.Benchmark.Scenario - alias Benchee.Formatter - alias Benchee.Formatters.TaggedSave + alias Benchee.{ + Benchmark.Scenario, + Formatter, + Formatters.TaggedSave, + Statistics, + Suite + } - import Benchee.Formatters.TaggedSave + import TaggedSave import Benchee.Benchmark, only: [no_input: 0] import ExUnit.CaptureIO import Benchee.TestHelpers, only: [suite_without_scenario_tags: 1] diff --git a/test/benchee/output/benchmark_printer_test.exs b/test/benchee/output/benchmark_printer_test.exs index 43c91818..56a9e550 100644 --- a/test/benchee/output/benchmark_printer_test.exs +++ b/test/benchee/output/benchmark_printer_test.exs @@ -1,10 +1,10 @@ defmodule Benchee.Output.BenchmarkPrintertest do use ExUnit.Case, async: true + + alias Benchee.{Benchmark, Benchmark.Scenario, Configuration} + import ExUnit.CaptureIO import Benchee.Output.BenchmarkPrinter - alias Benchee.Benchmark.Scenario - alias Benchee.Benchmark - alias Benchee.Configuration @system_info %{ elixir: "1.4", diff --git a/test/benchee/scenario_loader_test.exs b/test/benchee/scenario_loader_test.exs index b2039eba..27484633 100644 --- a/test/benchee/scenario_loader_test.exs +++ b/test/benchee/scenario_loader_test.exs @@ -1,8 +1,7 @@ defmodule Benchee.ScenarioLoaderTest do use ExUnit.Case import Benchee.ScenarioLoader - alias Benchee.{Suite, Configuration} - alias Benchee.Benchmark.{Scenario} + alias Benchee.{Benchmark.Scenario, Configuration, Suite} test "`load` indeed loads scenarios into the suite" do scenarios = [%Scenario{tag: "old"}] diff --git a/test/benchee/statistics_test.exs b/test/benchee/statistics_test.exs index e024e7e2..4e3140e7 100644 --- a/test/benchee/statistics_test.exs +++ b/test/benchee/statistics_test.exs @@ -1,6 +1,6 @@ defmodule Benchee.StatistcsTest do use ExUnit.Case, async: true - alias Benchee.{Statistics, Suite, Benchmark.Scenario, Configuration} + alias Benchee.{Benchmark.Scenario, Configuration, Statistics, Suite} doctest Benchee.Statistics @sample_1 [600, 470, 170, 430, 300] diff --git a/test/benchee_test.exs b/test/benchee_test.exs index 6fe693df..99593b98 100644 --- a/test/benchee_test.exs +++ b/test/benchee_test.exs @@ -1,13 +1,17 @@ defmodule BencheeTest do use ExUnit.Case, async: true + + alias Benchee.{ + Conversion.Duration, + Formatter, + Formatters.Console, + Statistics, + Suite, + Test.FakeFormatter + } + import ExUnit.CaptureIO import Benchee.TestHelpers - alias Benchee.Test.FakeFormatter - alias Benchee.Statistics - alias Benchee.Formatter - alias Benchee.Formatters.Console - alias Benchee.Suite - alias Benchee.Conversion.Duration doctest Benchee @@ -75,23 +79,6 @@ defmodule BencheeTest do assert Regex.match?(body_regex("Magic"), output) end - test "integration high level README example old school map config" do - output = - capture_io(fn -> - list = Enum.to_list(1..10_000) - map_fun = fn i -> [i, i * i] end - - map_config = Enum.into(@test_configuration, %{}) - - Benchee.run(map_config, %{ - "flat_map" => fn -> Enum.flat_map(list, map_fun) end, - "map.flatten" => fn -> list |> Enum.map(map_fun) |> List.flatten() end - }) - end) - - readme_sample_asserts(output) - end - test "integration high level README example but with formatter options" do output = capture_io(fn -> @@ -229,7 +216,7 @@ defmodule BencheeTest do @test_configuration, time: 0.01, warmup: 0, - console: [comparison: false] + formatters: [{Console, %{comparison: false}}] ) ) end) @@ -277,7 +264,7 @@ defmodule BencheeTest do readme_sample_asserts(output) end - test "formatters can be supplied with the Formatter.output/3 function" do + test "formatters can be supplied as a function with arity 1" do output = capture_io(fn -> list = Enum.to_list(1..10_000) @@ -418,6 +405,35 @@ defmodule BencheeTest do assert length(occurences) == 3 end + test "inputs can also be a list of 2-tuples" do + output = + capture_io(fn -> + map_fun = fn i -> [i, i * i] end + + inputs = [ + inputs: [ + {"small list", Enum.to_list(1..100)}, + {"medium list", Enum.to_list(1..1_000)}, + {"bigger list", Enum.to_list(1..10_000)} + ] + ] + + configuration = Keyword.merge(@test_configuration, inputs) + + Benchee.run( + %{ + "flat_map" => fn input -> Enum.flat_map(input, map_fun) end, + "map.flatten" => fn input -> input |> Enum.map(map_fun) |> List.flatten() end + }, + configuration + ) + end) + + assert String.contains?(output, ["small list", "medium list", "bigger list"]) + occurences = Regex.scan(body_regex("flat_map"), output) + assert length(occurences) == 3 + end + test "multiple inputs with very fast functions" do output = capture_io(fn ->