mix test --only describe:"sometest" and mix test myfile_test.ex:42 behave differently

[turion]

While mix test --only describe:"sometest" compiles and includes all modules, mix test myfile_test.ex:42 does not.

Environment

• Elixir & Erlang/OTP versions (elixir --version): Erlang/OTP 20 [erts-9.3.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]
  Elixir 1.6.6 (compiled with OTP 20)
• Operating system: NixOS

Current behavior

1. Create a new project with mix new
2. Create a file test/module_test.exs with this content:

defmodule ModuleTest do
  def foo() do
    false
  end
end

3. Append these lines to the main test file:

  describe "footest" do
    test "bar" do
      assert foo()
    end
  end

4. This works:

$ mix test --only describe:"footest"
Including tags: [describe: "footest"]
Excluding tags: [:test]



  1) test footest bar (ExunitBugTest)
     test/exunit_bug_test.exs:12
     Expected truthy, got false
     code: assert foo()
     stacktrace:
       test/exunit_bug_test.exs:13: (test)



Finished in 0.04 seconds
1 doctest, 2 tests, 1 failure, 2 skipped

Randomized with seed 600055

5. This doesn't:

$ mix test test/exunit_bug_test.exs:12
Including tags: [line: "12"]
Excluding tags: [:test]


== Compilation error in file test/exunit_bug_test.exs ==
** (CompileError) test/exunit_bug_test.exs:5: module ModuleTest is not loaded and could not be found
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
    (elixir) lib/code.ex:677: Code.require_file/2
    (elixir) lib/kernel/parallel_compiler.ex:201: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6

Expected behavior

Both work and do the same.

Why am I doing this?

This MWE doesn't look like a reasonable elixir project because I don't define the function I'm testing in lib/ but in test/. But often we need some convenience functions (generators etc.) that need to be included from the individual test cases, and it feels to me like the right place is a module in test/.

[josevalim]

This is by design. You want file:LINE to only load that file and avoid the work of loading the whole suite.

Also, keep in mind that there is no guarantee test/module_test.exs will be loaded before other test files, so you actually have a race condition.

The correct thing is to put any support module in a directory like test/support/module.exs and then you do Code.require_file "../support/module.exs" from the top of your test files that require module.exs. OR you always load the support files in your test_helper.