Merge 336d11e into ee9f723

.travis.yml

@@ -1,29 +1,16 @@
 language: elixir
 elixir:
-  - 1.4.5
-  - 1.5.3
   - 1.6.6
   - 1.7.4
 otp_release:
-  - 19.3
   - 20.3
   - 21.1
-
-matrix:
-  exclude:
-  - elixir: 1.4.5
-    otp_release: 21.1
-  - elixir: 1.5.3
-    otp_release: 21.1
-  - elixir: 1.7.4
-    otp_release: 19.3
-
 before_script:
   - MIX_ENV=test mix compile --warnings-as-errors
   - travis_wait mix dialyzer --plt
 script:
   - mix credo --strict
-  - if [[ "$TRAVIS_ELIXIR_VERSION" == "1.6"* ]]; then mix format --check-formatted; fi
+  - mix format --check-formatted
   - mix dialyzer --halt-exit-status
   - mix safe_coveralls.travis
 after_script:

README.md

@@ -94,7 +94,7 @@ Add benchee to your list of dependencies in `mix.exs`:
 
 ```elixir
 defp deps do
-  [{:benchee, "~> 0.11", only: :dev}]
+  [{:benchee, "~> 0.13", only: :dev}]
 end
 ```
 
@@ -162,7 +162,7 @@ The available options are the following (also documented in [hexdocs](https://he
 * `warmup` - the time in seconds for which a benchmarking job should be run without measuring times before "real" measurements start. This simulates a _"warm"_ running system. Defaults to 2.
 * `time` - the time in seconds for how long each individual benchmarking job should be run for measuring the execution times (run time performance). Defaults to 5.
 * `memory_time` - the time in seconds for how long [memory measurements](measuring-memory-consumption) should be conducted. Defaults to 0 (turned off).
-* `inputs` - a map from descriptive input names to some different input, your benchmarking jobs will then be run with each of these inputs. For this to work your benchmarking function gets the current input passed in as an argument into the function. Defaults to `nil`, aka no input specified and functions are called without an argument. See [Inputs](#inputs).
+* `inputs` - a map or list of two element tuples. If a map, they keys are descriptive input names and values are the actual input values. If a list of tuples, the first element in each tuple is the input name, and the second element in each tuple is the actual input value. Your benchmarking jobs will then be run with each of these inputs. For this to work your benchmarking function gets the current input passed in as an argument into the function. Defaults to `nil`, aka no input specified and functions are called without an argument. See [Inputs](#inputs).
 * `formatters` - list of formatters either as a module implementing the formatter behaviour, a tuple of said module and options it should take or formatter functions. They are run when using `Benchee.run/2` or you can invoktem them through `Benchee.Formatter.output/1`. Functions need to accept one argument (which is the benchmarking suite with all data) and then use that to produce output. Used for plugins. Defaults to the builtin console formatter `Benchee.Formatters.Console`. See [Formatters](#formatters).
 * `pre_check` - whether or not to run each job with each input - including all given before or after scenario or each hooks - before the benchmarks are measured to ensure that your code executes without error. This can save time while developing your suites. Defaults to `false`.
 * `parallel` - the function of each benchmarking job will be executed in `parallel` number processes. If `parallel: 4` then 4 processes will be spawned that all execute the _same_ function for the given time. When these finish/the time is up 4 new processes will be spawned for the next job/function. This gives you more data in the same time, but also puts a load on the system interfering with benchmark results. For more on the pros and cons of parallel benchmarking [check the wiki](https://github.com/PragTob/benchee/wiki/Parallel-Benchmarking). Defaults to 1 (no parallel execution).
@@ -218,18 +218,26 @@ A full example, including an example of the console output, can be found
 
 ### Inputs
 
-`:inputs` is a very useful configuration that allows you to run the same benchmarking jobs with different inputs. You specify the inputs as a map from name (String or atom) to the actual input value. Functions can have different performance characteristics on differently shaped inputs - be that structure or input size.
+`:inputs` is a very useful configuration that allows you to run the same benchmarking jobs with different inputs. You specify the inputs as either a map from name (String or atom) to the actual input value or a list of tuples where the first element in each tuple is the name and the second element in the tuple is the value. Functions can have different performance characteristics on differently shaped inputs - be that structure or input size.
 
 One of such cases is comparing tail-recursive and body-recursive implementations of `map`. More information in the [repository with the benchmark](https://github.com/PragTob/elixir_playground/blob/master/bench/tco_blog_post_focussed_inputs.exs) and the [blog post](https://pragtob.wordpress.com/2016/06/16/tail-call-optimization-in-elixir-erlang-not-as-efficient-and-important-as-you-probably-think/).
 
 ```elixir
 map_fun = fn(i) -> i + 1 end
 inputs = %{
-  "Small (1 Thousand)"    => Enum.to_list(1..1_000),
+  "Small (1 Thousand)" => Enum.to_list(1..1_000),
   "Middle (100 Thousand)" => Enum.to_list(1..100_000),
-  "Big (10 Million)"      => Enum.to_list(1..10_000_000),
+  "Big (10 Million)" => Enum.to_list(1..10_000_000)
 }
 
+# Or inputs could also look like this:
+#
+# inputs = [
+#   {"Small (1 Thousand)", Enum.to_list(1..1_000)},
+#   {"Middle (100 Thousand)", Enum.to_list(1..100_000)},
+#   {"Big (10 Million)", Enum.to_list(1..10_000_000)}
+# ]
+
 Benchee.run %{
   "map tail-recursive" =>
     fn(list) -> MyMap.map_tco(list, map_fun) end,

lib/benchee.ex

@@ -68,11 +68,11 @@ for {module, moduledoc} <- [{Benchee, elixir_doc}, {:benchee, erlang_doc}] do
     defdelegate init(), to: Benchee.Configuration
     defdelegate init(config), to: Benchee.Configuration
     defdelegate system(suite), to: Benchee.System
+    defdelegate benchmark(suite, name, function), to: Benchee.Benchmark
+    defdelegate benchmark(suite, name, function, printer), to: Benchee.Benchmark
     defdelegate measure(suite), to: Benchee.Benchmark
     defdelegate measure(suite, printer), to: Benchee.Benchmark
-    defdelegate benchmark(suite, name, function), to: Benchee.Benchmark
     defdelegate statistics(suite), to: Benchee.Statistics
     defdelegate load(suite), to: Benchee.ScenarioLoader
-    defdelegate benchmark(suite, name, function, printer), to: Benchee.Benchmark
   end
 end

lib/benchee/benchmark.ex

@@ -4,7 +4,7 @@ defmodule Benchee.Benchmark do
   Exposes `benchmark/4` and `measure/3` functions.
   """
 
-  alias Benchee.Benchmark.{Scenario, ScenarioContext, Runner}
+  alias Benchee.Benchmark.{Runner, Scenario, ScenarioContext}
   alias Benchee.Output.BenchmarkPrinter, as: Printer
   alias Benchee.Suite
   alias Benchee.Utility.DeepConvert

lib/benchee/benchmark/repeated_measurement.ex

@@ -21,7 +21,7 @@ defmodule Benchee.Benchmark.RepeatedMeasurement do
   # with too high variance. Therefore determine an n how often it should be
   # executed in the measurement cycle.
 
-  alias Benchee.Benchmark.{Hooks, Runner, Scenario, ScenarioContext, Measure}
+  alias Benchee.Benchmark.{Hooks, Measure, Runner, Scenario, ScenarioContext}
   alias Benchee.Utility.RepeatN
 
   @minimum_execution_time 10

lib/benchee/benchmark/runner.ex

@@ -4,12 +4,8 @@ defmodule Benchee.Benchmark.Runner do
   # This module actually runs our benchmark scenarios, adding information about
   # run time and memory usage to each scenario.
 
-  alias Benchee.Benchmark
-  alias Benchee.Benchmark.{Scenario, ScenarioContext, Measure, Hooks, RepeatedMeasurement}
-  alias Benchee.Configuration
-  alias Benchee.Conversion
-  alias Benchee.Statistics
-  alias Benchee.Utility.Parallel
+  alias Benchee.{Benchmark, Configuration, Conversion, Statistics, Utility.Parallel}
+  alias Benchmark.{Hooks, Measure, RepeatedMeasurement, Scenario, ScenarioContext}
 
   @doc """
   Executes the benchmarks defined before by first running the defined functions

lib/benchee/configuration.ex

@@ -3,13 +3,15 @@ defmodule Benchee.Configuration do
   Functions to handle the configuration of Benchee, exposes `init/1` function.
   """
 
+  alias Benchee.Formatters.{Console, CSV, HTML, JSON}
+
   alias Benchee.{
-    Suite,
     Configuration,
     Conversion.Duration,
     Conversion.Scale,
-    Utility.DeepConvert,
-    Formatters.Console
+    Formatters.Console,
+    Suite,
+    Utility.DeepConvert
   }
 
   defstruct parallel: 1,
@@ -53,7 +55,7 @@ defmodule Benchee.Configuration do
           pre_check: boolean,
           formatters: [(Suite.t() -> Suite.t())],
           print: map,
-          inputs: %{Suite.key() => any} | nil,
+          inputs: %{Suite.key() => any} | [{Suite.key(), any}] | nil,
           save: map | false,
           load: String.t() | [String.t()] | false,
           formatter_options: map,
@@ -273,7 +275,7 @@ defmodule Benchee.Configuration do
             parallel: 2,
             time: 1_000_000_000.0,
             warmup: 200_000_000.0,
-            inputs: %{"Small" => 5, "Big" => 9999},
+            inputs: [{"Big", 9999}, {"Small", 5}],
             save: false,
             load: false,
             formatters: [&IO.puts/1],
@@ -319,9 +321,9 @@ defmodule Benchee.Configuration do
 
   defp standardized_user_configuration(config) do
     config
-    |> DeepConvert.to_map([:formatters])
-    |> translate_formatter_keys
-    |> force_string_input_keys
+    |> DeepConvert.to_map([:formatters, :inputs])
+    |> translate_formatter_keys()
+    |> standardize_inputs()
   end
 
   # backwards compatible translation of formatter keys to go into
@@ -332,8 +334,6 @@ defmodule Benchee.Configuration do
     DeepMerge.deep_merge(%{formatter_options: formatter_options}, config)
   end
 
-  alias Benchee.Formatters.{Console, CSV, JSON, HTML}
-
   # backwards compatible formatter definition without leaving the burden on every formatter
   defp formatter_options_to_tuples(config) do
     update_in(config, [Access.key(:formatters), Access.all()], fn
@@ -353,16 +353,26 @@ defmodule Benchee.Configuration do
     end
   end
 
-  defp force_string_input_keys(config = %{inputs: inputs}) do
+  defp standardize_inputs(config = %{inputs: inputs}) do
     standardized_inputs =
-      for {name, value} <- inputs, into: %{} do
-        {to_string(name), value}
-      end
+      inputs
+      |> Enum.reduce([], &standardize_inputs/2)
+      |> Enum.reverse()
 
     %{config | inputs: standardized_inputs}
   end
 
-  defp force_string_input_keys(config), do: config
+  defp standardize_inputs(config), do: config
+
+  defp standardize_inputs({name, value}, acc) do
+    normalized_name = to_string(name)
+
+    if List.keymember?(acc, normalized_name, 0) do
+      acc
+    else
+      [{normalized_name, value} | acc]
+    end
+  end
 
   defp merge_with_defaults(user_config) do
     DeepMerge.deep_merge(%Configuration{}, user_config)

lib/benchee/conversion.ex

@@ -6,7 +6,7 @@ defmodule Benchee.Conversion do
   """
 
   alias Benchee.Benchmark.Scenario
-  alias Benchee.Conversion.{Duration, Count, Memory}
+  alias Benchee.Conversion.{Count, Duration, Memory}
 
   @doc """
   Takes scenarios and a given scaling_strategy, returns the best units for the

lib/benchee/formatters/console.ex

@@ -70,14 +70,26 @@ defmodule Benchee.Formatters.Console do
       |> Map.merge(options)
 
     scenarios
-    |> Enum.group_by(fn scenario -> scenario.input_name end)
+    |> Enum.reduce([], &update_grouped_list/2)
+    |> Enum.reverse()
     |> Enum.map(fn {input, scenarios} ->
       scenarios
       |> Statistics.sort()
       |> generate_output(config, input)
     end)
   end
 
+  defp update_grouped_list(scenario, grouped_scenarios) do
+    case List.keyfind(grouped_scenarios, scenario.input_name, 0) do
+      {_, group} ->
+        new_tuple = {scenario.input_name, [scenario | group]}
+        List.keyreplace(grouped_scenarios, scenario.input_name, 0, new_tuple)
+
+      _ ->
+        [{scenario.input_name, [scenario]} | grouped_scenarios]
+    end
+  end
+
   def write(suite), do: write(suite, %{})
 
   @doc """

lib/benchee/formatters/console/memory.ex

@@ -9,9 +9,9 @@ defmodule Benchee.Formatters.Console.Memory do
   alias Benchee.{
     Benchmark.Scenario,
     Conversion,
-    Conversion.Unit,
     Conversion.Count,
     Conversion.Memory,
+    Conversion.Unit,
     Formatters.Console.Helpers,
     Statistics
   }

lib/benchee/formatters/console/run_time.ex

@@ -10,8 +10,8 @@ defmodule Benchee.Formatters.Console.RunTime do
     Benchmark.Scenario,
     Conversion,
     Conversion.Count,
-    Conversion.Unit,
     Conversion.Duration,
+    Conversion.Unit,
     Formatters.Console.Helpers,
     Statistics
   }

lib/benchee/output/benchmark_printer.ex

@@ -72,7 +72,7 @@ defmodule Benchee.Output.BenchmarkPrinter do
 
   defp inputs_out(inputs) do
     inputs
-    |> Map.keys()
+    |> Enum.map(fn {name, _} -> name end)
     |> Enum.join(", ")
   end
 

mix.exs

@@ -51,13 +51,12 @@ defmodule Benchee.Mixfile do
     [
       {:deep_merge, "~> 0.1"},
       {:ex_guard, "~> 1.3", only: :dev},
-      {:credo, "< 0.9.3", only: :dev},
-      # newer versions break in older elixirs
+      {:credo, "~> 1.0.0", only: :dev},
       {:ex_doc, "~> 0.18.0", only: :dev},
       {:earmark, "~> 1.0", only: :dev},
       {:excoveralls, "~> 0.7", only: :test},
       {:inch_ex, "~> 0.5", only: :docs},
-      {:dialyxir, "~> 0.5", only: :dev, runtime: false}
+      {:dialyxir, "~> 1.0.0-rc.4", only: :dev, runtime: false}
     ]
   end
 

mix.lock

@@ -1,13 +1,14 @@
 %{
   "bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm"},
   "certifi": {:hex, :certifi, "2.4.2", "75424ff0f3baaccfd34b1214184b6ef616d89e420b258bb0a5ea7d7bc628f7f0", [:rebar3], [{:parse_trans, "~>3.3", [hex: :parse_trans, repo: "hexpm", optional: false]}], "hexpm"},
-  "credo": {:hex, :credo, "0.9.2", "841d316612f568beb22ba310d816353dddf31c2d94aa488ae5a27bb53760d0bf", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:poison, ">= 0.0.0", [hex: :poison, repo: "hexpm", optional: false]}], "hexpm"},
+  "credo": {:hex, :credo, "1.0.0", "aaa40fdd0543a0cf8080e8c5949d8c25f0a24e4fc8c1d83d06c388f5e5e0ea42", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
   "deep_merge": {:hex, :deep_merge, "0.2.0", "c1050fa2edf4848b9f556fba1b75afc66608a4219659e3311d9c9427b5b680b3", [:mix], [], "hexpm"},
-  "dialyxir": {:hex, :dialyxir, "0.5.1", "b331b091720fd93e878137add264bac4f644e1ddae07a70bf7062c7862c4b952", [:mix], [], "hexpm"},
-  "earmark": {:hex, :earmark, "1.2.6", "b6da42b3831458d3ecc57314dff3051b080b9b2be88c2e5aa41cd642a5b044ed", [:mix], [], "hexpm"},
+  "dialyxir": {:hex, :dialyxir, "1.0.0-rc.4", "71b42f5ee1b7628f3e3a6565f4617dfb02d127a0499ab3e72750455e986df001", [:mix], [{:erlex, "~> 0.1", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm"},
+  "earmark": {:hex, :earmark, "1.3.0", "17f0c38eaafb4800f746b457313af4b2442a8c2405b49c645768680f900be603", [:mix], [], "hexpm"},
+  "erlex": {:hex, :erlex, "0.1.6", "c01c889363168d3fdd23f4211647d8a34c0f9a21ec726762312e08e083f3d47e", [:mix], [], "hexpm"},
   "ex_doc": {:hex, :ex_doc, "0.18.4", "4406b8891cecf1352f49975c6d554e62e4341ceb41b9338949077b0d4a97b949", [:mix], [{:earmark, "~> 1.1", [hex: :earmark, repo: "hexpm", optional: false]}], "hexpm"},
   "ex_guard": {:hex, :ex_guard, "1.3.3", "941acb390979a1ef40ed0ef10d07035ce7d98b9d0089cecca132f9c0744ac620", [:mix], [{:fs, "~> 0.9", [hex: :fs, repo: "hexpm", optional: false]}], "hexpm"},
-  "excoveralls": {:hex, :excoveralls, "0.10.2", "fb4abd5b8a1b9d52d35e1162e7e2ea8bfb84b47ae07c38d39aa8ce64be0b0794", [:mix], [{:hackney, "~> 1.13", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
+  "excoveralls": {:hex, :excoveralls, "0.10.3", "b090a3fbcb3cfa136f0427d038c92a9051f840953ec11b40ee74d9d4eac04d1e", [:mix], [{:hackney, "~> 1.13", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
   "fs": {:hex, :fs, "0.9.2", "ed17036c26c3f70ac49781ed9220a50c36775c6ca2cf8182d123b6566e49ec59", [:rebar], [], "hexpm"},
   "hackney": {:hex, :hackney, "1.14.3", "b5f6f5dcc4f1fba340762738759209e21914516df6be440d85772542d4a5e412", [:rebar3], [{:certifi, "2.4.2", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "6.0.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "1.0.1", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "1.0.2", [hex: :mimerl, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "1.1.4", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm"},
   "idna": {:hex, :idna, "6.0.0", "689c46cbcdf3524c44d5f3dde8001f364cd7608a99556d8fbd8239a5798d4c10", [:rebar3], [{:unicode_util_compat, "0.4.1", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm"},

test/benchee/benchmark/runner_test.exs

@@ -1,9 +1,10 @@
 defmodule Benchee.Benchmark.RunnerTest do
   use ExUnit.Case, async: true
+
   import Benchee.TestHelpers
   import ExUnit.CaptureIO
-  alias Benchee.{Suite, Benchmark, Configuration}
-  alias Benchee.Benchmark.Scenario
+
+  alias Benchee.{Benchmark, Benchmark.Scenario, Configuration, Suite}
   alias Benchee.Test.FakeBenchmarkPrinter, as: TestPrinter
 
   @config %Configuration{

test/benchee/benchmark_test.exs

@@ -1,11 +1,16 @@
 defmodule Benchee.BenchmarkTest do
   use ExUnit.Case, async: true
-  alias Benchee.Benchmark
-  alias Benchee.Configuration
-  alias Benchee.Benchmark.{Scenario, ScenarioContext}
+
+  alias Benchee.{
+    Benchmark,
+    Benchmark.Scenario,
+    Benchmark.ScenarioContext,
+    Configuration,
+    Suite
+  }
+
   alias Benchee.Test.FakeBenchmarkPrinter, as: TestPrinter
   alias Benchee.Test.FakeBenchmarkRunner, as: TestRunner
-  alias Benchee.Suite
 
   describe ".benchmark" do
     test "can add jobs with atom keys but converts them to string" do

test/benchee/configuration_test.exs

@@ -9,26 +9,29 @@ defmodule Benchee.ConfigurationTest do
 
   @default_config %Configuration{}
 
-  describe ".init/1" do
-    test "it crashes for values that are going to be ignored" do
+  describe "init/1" do
+    test "crashes for values that are going to be ignored" do
       assert_raise KeyError, fn ->
         init(runntime: 2)
       end
     end
 
-    test "it converts input keys to strings" do
+    test "converts inputs map to a list and input keys to strings" do
       suite = init(inputs: %{"map" => %{}, list: []})
 
-      assert %Suite{
-               configuration: %{inputs: %{"list" => [], "map" => %{}}}
-             } = suite
+      assert %Suite{configuration: %{inputs: [{"list", []}, {"map", %{}}]}} = suite
     end
 
-    test "it loses duplicated inputs keys after normalization" do
+    test "doesn't convert input lists to maps" do
+      suite = init(inputs: [{"map", %{}}, {:list, []}])
+      assert %Suite{configuration: %{inputs: [{"map", %{}}, {"list", []}]}} = suite
+    end
+
+    test "loses duplicated inputs keys after normalization" do
       suite = init(inputs: %{"map" => %{}, map: %{}})
 
       assert %Suite{configuration: %{inputs: inputs}} = suite
-      assert %{"map" => %{}} == inputs
+      assert [{"map", %{}}] == inputs
     end
 
     test "uses information from :save to setup the external term formattter" do

test/benchee/formatter_test.exs

@@ -1,6 +1,6 @@
 defmodule Benchee.FormatterTest do
   use ExUnit.Case, async: true
-  alias Benchee.{Suite, Formatter, Test.FakeFormatter}
+  alias Benchee.{Formatter, Suite, Test.FakeFormatter}
 
   describe "output/1" do
     test "calls `write/1` with the output of `format/1` on each module" do