Move from Prove to Lisp-Unit2

[Ambrevar]

Prove has some issues:

• Reporting is limited, it does not tell you precisely what went wrong. It cannot be turned off.
• The debugger is not triggered when a test fails (unless a condition was raised and prove:*debug-on-error* set to T), which makes it very inconvenient to write tests.
• Slow.
• Fixtures not supported.

According to https://sabracrolleton.github.io/testing-framework

• Rove, the successor of Prove, is not production ready.
• Parachute seems to be great. Its author, Shinmera, has produced many good libraries which we already use.
• Other interesting suites according to the article author: cl-unit2, lisp-unit2, fiasco.

A benefit of Parachute is that it supports Prove syntax, so migration would be easy.

Thoughts?

[aadcg]

What an exhaustive article! I will try to digest it to have an informed opinion.

[aartaka]

More Shinmeraware, yay! On a related note, LASS can simplify the theme library a whole lot :D

I'm more than fine with Parachute, especially if it's more interactive than Prove and FiveAM (used in cl-webkit) :)

[jmercouris]

Eitaro always says 'beta' quality, I believe it is just a way of limiting liability. I suggest we try rove and if it gives us issues, we move to parachute. I personally really like prove and rove.

[Ambrevar]

The aforementioned review lists two blockers for rove: - issues with multi-threading - only works with infered package systems, which we don't use.

[jmercouris]

OK, well then it is clear!

[hendursaga]

Eitaro always says 'beta' quality, I believe it is just a way of limiting liability.

What more do you need to say to limit liability than a screaming "THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"..."? 😛

I cast my vote for Parachute.

[Ambrevar]

Since I was offline most of last week, I seized the opportunity to review a few testing framework.

I used

https://sabracrolleton.github.io/testing-framework

as a starting point, but I also noticed that the review was not always accurate, sometimes mentioning that a feature was missing while it's there (maybe due to some copy-pasting that went out of control).

Frameworks I've reviewed:

• clunit2
• fiasco
• fiveam
• lift
• lisp-unit2
• parachute
• prove

And the winner is... Lisp-Unit2!
Closely followed by Parachute.

Main criteria:

• Composable wrapping fixtures.

  Some fixture systems are not "wrapping" over the tests, they just offer
  "setup / teardown" options and thus we cannot pass local variables around.

  Composable means that a fixture can call another one. Without this, one would
  have to duplicate code in fixtures. Or create a defun just for this.

  Of course it's always possible to implement fixtures with macros. But that
  means lots of search-and-replace of the "deftest" definers. Besides it's not
  as composable.

• Well presented and exhaustive reports.

  "Well presented" means that the end of the output must tell everything
  important (the failures) to the tester. It must be as concise as possible.

  "Exhaustive" means all failures are reported, not just the first one.

Minor criteria:

• Vocabulary (assert-* and define-* are nice since they are colored in Emacs).
• No setup (e.g. no need to define a suite if I don't want to).
• Run test at point.
• Support "go to test definition".
• Extensible.

Review details:

clunit2

Only 1 fixture per suite.

Fixture composition is reversed: In Nfiles, the GPG suite must be the parent of the default suite, which means that then we must test the GPG suite to test everything (testing the default suite would skip the GPG suite).

What if there are multiple parents then?
Design seems very broken.

Fiasco

No fixtures at all?

FiveAM

Pretty good overall but

• Little documentation.
• Too macro based?
• Reports are overly verbose.
• Deal breaker: fixtures cannot be composed.

Lift

Deal breakers:

• Reports only the 1st failure among test cases.
• Fixtures are not wrapping.

Lisp-Unit 2

Has mostly everything.

Well documented, well designed.

Tests are functions and thus "go to test definition" works!

"Compile at point" does not work, but see below for a work-around.

Parachute

Well documented, well designed.

CLOS-extensible!

"Fixtures" unfortunately only "fix" symbol values.
To wrap fixtures around test, one must extend the test class.
It works well and it's elegant, but slightly more verbose then Lisp-Unit2.

"Compile at point" does not work, but see below for a work-around.

Prove

It's impossible to load the system without running the test suite, which is
quite cumbersome. The whole design ((plan nil) terminated by (finalize)) is
confusing, I can't see why we need all this setup code.

Similarly, one cannot just compile a test (and thus catch compilation errors)
without running it.

It seems that it won't return any result object, it can only produce reports.

Prove defaults to using colors... Which are broken even when ANSI color parsing
is enabled in SLY, since it fails to reset some color. This results in unreadable
standard-output text after Prove has run.

Prove is archived and unmaintained, which means the many issues won't ever be fixed.

Deal breakers:

• Reports are poor, it's not easy to see what failed without scrolling back the
  entire log. Changing the report style may help here.

• No fixture.

Universal "run test at point"

If the framework allows to programmatically run an individual test, then it's
possible to run the test at point by adding this little Emacs snippet (for Lisp-Unit2):

(defun ambrevar/sly-run-lisp-unit-test-at-point (&optional raw-prefix-arg)
  "See `sly-compile-defun' for RAW-PREFIX-ARG."
  (interactive "P")
   (call-interactively 'sly-compile-defun)
  (let ((name `(quote ,(intern (sly-qualify-cl-symbol-name (sly-parse-toplevel-form))))))
    (sly-eval-async
        `(cl:string-trim "

�"
                         (cl:with-output-to-string (s)
                                                   (cl:let ((lisp-unit2:*test-stream* s))
                                                           (lisp-unit2:run-tests :tests ,name :run-contexts 'lisp-unit2:with-summary-context))))
      (lambda (results)
        (switch-to-buffer-other-window  (get-buffer-create "*Test Results*"))
        (erase-buffer)
        (insert results)))))

(define-key lisp-mode-map (kbd "C-c C-v") 'ambrevar/sly-run-lisp-unit-test-at-point)

[aartaka]

#2465 seems to complete this.