mix: when running `make test` some tests fail when `MIX_ENV` != "test"

[NobbZ]

Environment

• Elixir & Erlang/OTP versions (elixir --version): compiling from master
• Operating system: observed on linux like machines, probably similar behaviour will be observable on windows as well

Current behavior

MIX_ENV=Foo make test will fail with the following tests failed (at least):

  1) test env config defaults to the tasks's preferred cli environment (Mix.CLITest)
     test/mix/cli_test.exs:119
     Assertion with =~ failed
     code:  assert output =~ ~s"{:prod, [\"a\", \"b\", \"c\"]}"
     left:  "{:Foo, [\"a\", \"b\", \"c\"]}\n"
     right: "{:prod, [\"a\", \"b\", \"c\"]}"
     stacktrace:
       (elixir) lib/file.ex:1506: File.cd!/2
       test/mix/cli_test.exs:120: (test)



  2) test new --sup with tests (Mix.CLITest)
     test/mix/cli_test.exs:159
     Expected truthy, got false
     code: assert File.regular?("_build/test/lib/sup_with_tests/ebin/Elixir.SupWithTests.beam")
     stacktrace:
       test/mix/cli_test.exs:165: anonymous fn/0 in Mix.CLITest."test new --sup with tests"/1
       (elixir) lib/file.ex:1506: File.cd!/2
       test/mix/cli_test.exs:160: (test)



  3) test default task (Mix.CLITest)
     test/mix/cli_test.exs:6
     Expected truthy, got false
     code: assert File.regular?("_build/dev/lib/p/ebin/Elixir.A.beam")
     stacktrace:
       test/mix/cli_test.exs:16: anonymous fn/0 in Mix.CLITest."test default task"/1
       (elixir) lib/file.ex:1506: File.cd!/2
       test/test_helper.exs:123: MixTest.Case.in_fixture/3
       test/mix/cli_test.exs:7: (test)

.

  4) test new with tests (Mix.CLITest)
     test/mix/cli_test.exs:146
     Expected truthy, got false
     code: assert File.regular?("_build/test/lib/new_with_tests/ebin/Elixir.NewWithTests.beam")
     stacktrace:
       test/mix/cli_test.exs:152: anonymous fn/0 in Mix.CLITest."test new with tests"/1
       (elixir) lib/file.ex:1506: File.cd!/2
       test/mix/cli_test.exs:147: (test)

The files mentioned in 3 of the 4 messages are created at a place where the tests expect them, but with test substituted by Foo.

Expected behavior

Tests pass regardless the parents MIX_ENV or fail with a proper error message, eg: tests require `MIX_ENV` to be unset.