Make a distinction between load errors in the test file being loaded vs. in other namespaces.

[plexus]

If an exception occurs while Kaocha loads a test namespace, then a warning is printed, the :kaocha.type/namespace test is marked as failed, and loading/running continues.

This is so that when you have a typo which prevents a test namespace from loading, this doesn't stop the world. Instead it just runs the rest of the tests, and reports a failure pointing at the syntax error.

The problem with this approach is that loading a namespace could fail because a dependent namespace has issues. In the degenerate case a source namespace that is used indirectly in every test fails to load, causing a failure to be reported for every test namespace. This is especially easy to do (and annoying to encounter) in --watch mode.

Instead when loading a test namespace we should first evaluate its ns form. If that causes issues then we hard abort the test run. Only after the ns form has loaded do we load the full namespace, and turn load errors into test failures.

[magnars]

I came here to report this, glad to see it's already on the drawing board. :)

The most annoying part of my problem was that the error was due to using a function before declaring it - which did not trigger an error in my repl, since it was already declared! So kaocha gave me an error I was unable to reproduce in the repl. Grr. 😄

[plexus]

This is fixed in --watch mode now, but not yet in a regular run. I'm starting to have second thoughts about the whole idea of reporting load errors as test failures and continuing. An error during the load stage should instead immediately fail. It should be reported as if it's a failing test run though, so that for instance junit.xml or the notifier will be able to report the failure.