Support ExUnit filters

[josevalim]

An ExUnit filter is a mechanism to pass information from the external environment to the test suite. I am outlining this proposal for anyone interested in implementing this feature and it breaks into 3 steps:

1. Supporting filters on ExUnit.configure
2. Automatically match filters against tags
3. Support filters in mix command line

Different pull requests can be sent for each step but it is better to describe them together.

ExUnit.configure

This step consists of:

1. A :filter option should be allowed in ExUnit.configure/1 as a keyword list;

2. In case :filter is set, the ExUnit formatter should print a message before running the test suite along the lines of:

   Using filters: [foo: true, bar: false]
   

Tags support

We have recently added support for tags in ExUnit and we want filters to be able to turn tags on and off. Consider this test:

defmodule PathTest do
  use ExUnit.Case, async: true

  @tag :win32
  test "paths on windows" do
    ...
  end
end

When I configure my test suite now I can do the following:

win32 = match?({ :win32, _ }, :os.type)
ExUnit.configure(filter: [win32: win32])

And now the test will only run if I am running the test suite on win32 systems. Basically, if a test has a tag :win32 and there is a filter for that particular tag and the value of the tag does not match the value of filter, the test is skipped.

To skip a test, all we need to do is set its state field to { :skip, "due to win32 filter" }. A skipped test is not going to be printed (i.e. we won't see a dot nor anything) in the output, so ExUnit.CLIFormatter needs to be patched accordingly.

Mix support

Finally, mix test should support passing those filters via the command line. The API will be:

mix test --filter win32 --filter unix=false --filter os=unix

Basically:

• --filter win32 - equivalent to [win32: true]
• --filter unix:false - equivalent to [unix: false]
• --filter os:unix: - equivalent to [os: "unix"]

Note filters given via the command line needs to have higher priority than the ones set in test_helper.exs, so they need to be passed to ExUnit.configure/1 after test_helper.exs is loaded.