Proposal: Test framework

[jenshaase]

Motivation

Phel should provide a default testing framework to test Phel code.

Concept

The code for the Phel's testing library is bundled in the namespace phel.test.

Assertions

The core of the library is the is macro to define assertions. For example

(is (= 4 (+ 2 2))
(is (true? (or true false))
(is (not (true? (and true false)))

The is macro supports following forms:

• (predicate expected actual) like (= 4 (+ 2 2))
• (predicate value) like (true? (or true false))
• (not (predicate expected actual)) like (not (= 4 (+ 2 3)))
• (not (predicate value)) like (not (true? false))
• (thrown? exception-symbol body) like (thrown? Exception (/ 1 0))
• (thrown-with-msg? exception-symbol exception-msg body) like (thrown? Exception "Division by zero" (/ 1 0))
• (output-string expected-string body) like (output-string "test" (php/echo "test"))
• anything else that evaluates to true or false

The is macro supports an additional second string argument to document the test case

(is (= 4 (+ 2 2)) "add operation")

An assertion can lead to the three states:

• passed: When the form evaluates to true
• failed: When the from evaluates to false
• error: When the form could not be executed successfully (throws Exceptions)

Defining tests

To define a test the deftest macro can be used

(deftest my-add-test
  (is (= 4 (+ 2 2)))

Internally deftest is a defn with zero arguments. As metadata the defn has a :test attribute. So deftest is equivalent to

(defn my-add-test @{:test true} []
  (is (= 4 (+ 2 2)))

Running tests

The function run-tests can be used to executes tests in one or more namespaces

(run-tests 
  my-namespace\a 
  my-namespace\b)

Reporting

When executing the tests a reporter should print the current status of the test suite. While running it should print a . for a successful test a F for a failed test and a E for an error test. Afterwards it prints a list with details about all failures and errors. At the end a summary is printed (tests executed, tests passed, tests failed, tests with error).

Failure examples

(is (= 4 (+ 2 3)) "my test")

Failure in my test (path/to/file:123)
          Test: (+ 2 3)
  evaluated to: 5
    but is not: = to 4

---

(is (true? false) "my test")

Failure in my test (path/to/file:123)
                 Test: false
         evaluated to: false
  but doesn't satisfy: true?

---

(is (not (= 4 (+ 2 2))) "my test")

Failure in my test (path/to/file:123)
          Test: (+ 2 2)
  evaluated to: 4
        but is: = to 4 (it shouldn't be)

---

(is (not (true? true)) "my test")

Failure in my test (path/to/file:123)
              Test: true
      evaluated to: true
  but does satisfy: true? (it shouldn't)

Error examples

Error in path/to/file:123:
<Exception output (Message + StackTrace)>

Additional information

To make the API look nice, the namespace macro needs to be extended to support direct imports. Currently, it works the following way

(ns my-test-ns
  (:require phel\test :as test))

(test/deftest ...)

But it would be nice if we can omit the test/ prefix. Therefore, I would propose following addition

(ns my-test-ns
  (:require phel\test :as test :refer [deftest is]))

(deftest ...)

[mabasic]

Looks very useful and simple, but I have some questions/suggestions.

Since (deftest) is the same as (defn) which is the same as (def) why can't we add (deftest) macro in the core and thus avoid adding :refer to the (ns)?

As an alternative, why not just use (defn) and tell people to add metadata @{:test true} for writing tests?

Could we add a command for vendor/bin/phel test namespace/file/test-name?

[jenshaase]

Since (deftest) is the same as (defn) which is the same as (def) why can't we add (deftest) macro in the core and thus avoid adding :refer to the (ns)?

I see the :refer not only useful for this case. It can also be quite useful for library developers to extend Phel with some nice APIs. For example an if anyone whats to define a routing library, he can define a defroute macro like this:

(defroute "/my/route/:id" [request]
  "hello world")

Could we add a command for vendor/bin/phel test namespace/file/test-name?

Sure!

[mabasic]

I see. Makes sense now. Thank you for explaining the use case a little bit more.

[paullaffitte]

I really like the idea of using ., F or E and to print the detailed output at the end. I think that most test libraries do like this and it produce a good and readable output.

I think it would be pretty useful to print the expected value on test failure. It would help developers to understand what is going on without having to manually print the test's parameters. Quasi-quotation could help here.

Here is an example of what I'm thinking about:

(defmacro assertt
  [test label]
  `(assert ,test (str ,label ", Expected: " ,(print-str test)))
)

(assertt (= 1 2) "Test case")

And the output of this would be e Test case, Expected: (= 1 2).

Then I have a question about run-tests, we would have to pass as parameters all tests that we want to execute right? Maybe auto-discovery of the tests cases (just like phpunit does) would be nice. Then you don't have to worry about forgetting to add one of your tests to the list. And then (again, just like phpunit), we could add a flag or something similar to run only tests that match a pattern, are in some namespace, or even eventually in a test suite (this would require to add something like deftestsuite btw).

[jenshaase]

@paullaffitte Thanks for your input. I will add more details in the report section, on how failure and error message should look like (with examples).

Regarding run-tests. This function is the base for running all tests. It's a good idea to add auto-discovery. That can be build on top of run-tests. I thinks this can be done with the vendor/bin/phel test command. If no namespace is provided all test should run.

[jenshaase]

I updated the Report section. What do you think?

[jenshaase]

Current Status:

• I implemented the framework as described above
• I updated the old tests to the new test framework
• I added the test framework documentation on the website

What is left:

• Autodiscovery of all tests
• ./vendor/bin/phel test command

[jenshaase]

This is now completed! All test of the core library can now be executed by ./vendor/bin/phel test

[mabasic]

Great work! Really loving how the language is turning out. Sorry I can't contribute more to the code. I have my hands full atm... but testing and experimenting is what I love to do in the morning, so I will give it a go and report back.