- 1 The try ASDF System
- 2 Links
- 3 Tutorial
- 4 Emacs Integration
- 5 Events
- 6 The
IS
Macro - 7 Check Library
- 8 Tests
- 9 Implementation Notes
- 10 Glossary
- Version: 0.0.1
- Description: Try is an extensible test framework with equal support for interactive and non-interactive workflows.
- Long Description: Try stays as close to normal Lisp evaluation
rules as possible. Tests are functions that record the checks they
perform as events. These events provide the means of customization
of what to debug, print, rerun. There is a single fundamental check,
the extensible
IS
macro. Everything else is built on top. - Licence: MIT, see COPYING.
- Author: Gábor Melis
- Mailto: mega@retes.hu
- Homepage: http://melisgl.github.io/try
- Bug tracker: https://github.com/melisgl/try/issues
- Source control: GIT
Here is the official repository and the HTML documentation for the latest version.
Try is a library for unit testing with equal support for interactive and non-interactive workflows. Tests are functions, and almost everything else is a condition, whose types feature prominently in parameterization.
Try is is what we get if we make tests functions and build a test
framework on top of the condition system as
Stefil did
but also address the issue of rerunning and replaying, make the
IS
check more capable, use the types of the condition hierarchy
to parameterize what to debug, print, rerun, and finally document
the whole thing.
The IS
Macro is a replacement for CL:ASSERT
, that can capture values of
subforms to provide context to failures:
(is (= (1+ 5) 0))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS (= #1=(1+ 5) 0))
.. where
.. #1# = 6
This is a PAX transcript,
output is prefixed with ..
. Readable and unreadable return values
are prefixed with =>
and ==>
, respectively.
Note the #N#
syntax due to *PRINT-CIRCLE*
.
IS
automatically captures values of
arguments to functions like 1+
in the above example. Values of
other interesting subforms can be explicitly
captured. IS
supports capturing multiple
values and can be taught how to deal with macros. The combination of these
features allows MATCH-VALUES
to be implementable as tiny extension:
(is (match-values (values (1+ 5) "sdf")
(= * 0)
(string= * "sdf")))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS
.. (MATCH-VALUES #1=(VALUES (1+ 5) #2="sdf")
.. (= * 0)
.. (STRING= * "sdf")))
.. where
.. #1# == 6
.. #2#
In the body of MATCH-VALUES
, *
is bound to
successive return values of some form, here (VALUES (1+ 5) "sdf")
.
MATCH-VALUES
comes with an automatic rewrite rule that captures the
values of this form, which are printed above as #1# == 6 #2#
. IS
is flexible enough that all other checks (SIGNALS
, SIGNALS-NOT
,
INVOKES-DEBUGGER
, INVOKES-DEBUGGER-NOT
, FAILS
, and IN-TIME
are built
on top of it.
Beyond IS
, a fancy ASSERT
, Try provides tests, which are Lisp
functions that record their execution in TRIAL
objects. Let's define
a test and run it:
(deftest should-work ()
(is t))
(should-work)
.. SHOULD-WORK ; TRIAL-START
.. ⋅ (IS T) ; EXPECTED-RESULT-SUCCESS
.. ⋅ SHOULD-WORK ⋅1 ; EXPECTED-VERDICT-SUCCESS
..
==> #<TRIAL (SHOULD-WORK) EXPECTED-SUCCESS 0.000s ⋅1>
Try is driven by conditions, and the comments to the right give the
type of the condition that is printed on that line. The ⋅
character marks successes.
We could have run our test with (TRY 'SHOULD-WORK)
as well, which
does pretty much the same thing except it defaults to never entering
the debugger, whereas calling a test function directly enters the
debugger on events whose type matches the type in the variable
*DEBUG*
.
(try 'should-work)
.. SHOULD-WORK
.. ⋅ (IS T)
.. ⋅ SHOULD-WORK ⋅1
..
==> #<TRIAL (SHOULD-WORK) EXPECTED-SUCCESS 0.000s ⋅1>
Test suites are just tests that call other tests.
(deftest my-suite ()
(should-work)
(is (= (foo) 5)))
(defun foo ()
4)
(try 'my-suite)
.. MY-SUITE ; TRIAL-START
.. SHOULD-WORK ; TRIAL-START
.. ⋅ (IS T) ; EXPECTED-RESULT-SUCCESS
.. ⋅ SHOULD-WORK ⋅1 ; EXPECTED-VERDICT-SUCCESS
.. ⊠ (IS (= #1=(FOO) 5)) ; UNEXPECTED-RESULT-FAILURE
.. where
.. #1# = 4
.. ⊠ MY-SUITE ⊠1 ⋅1 ; UNEXPECTED-VERDICT-FAILURE
..
==> #<TRIAL (MY-SUITE) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅1>
⊠
marks UNEXPECTED-FAILURE
s. Note how the failure of (IS (= (FOO) 5))
caused MY-SUITE
to fail as well. Finally, the ⊠1
and the
⋅1
in the TRIAL
's printed representation are the event
counts.
To focus on the important bits, we can print only the UNEXPECTED
events:
(try 'my-suite :print 'unexpected)
.. MY-SUITE
.. ⊠ (IS (= #1=(FOO) 5))
.. where
.. #1# = 4
.. ⊠ MY-SUITE ⊠1 ⋅1
..
==> #<TRIAL (MY-SUITE) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅1>
Note that SHOULD-WORK
is still run, and its check's success is
counted as evidenced by⋅1
. The above effect can also be achieved
without running the tests again with REPLAY-EVENTS
.
Let's figure out what went wrong:
(my-suite)
;;; Here the debugger is invoked:
UNEXPECTED-FAILURE in check:
(IS (= #1=(FOO) 5))
where
#1# = 4
Restarts:
0: [RECORD-EVENT] Record the event and continue.
1: [FORCE-EXPECTED-SUCCESS] Change outcome to TRY:EXPECTED-RESULT-SUCCESS.
2: [FORCE-UNEXPECTED-SUCCESS] Change outcome to TRY:UNEXPECTED-RESULT-SUCCESS.
3: [FORCE-EXPECTED-FAILURE] Change outcome to TRY:EXPECTED-RESULT-FAILURE.
4: [ABORT-CHECK] Change outcome to TRY:RESULT-ABORT*.
5: [SKIP-CHECK] Change outcome to TRY:RESULT-SKIP.
6: [RETRY-CHECK] Retry check.
7: [ABORT-TRIAL] Record the event and abort trial TRY::MY-SUITE.
8: [SKIP-TRIAL] Record the event and skip trial TRY::MY-SUITE.
9: [RETRY-TRIAL] Record the event and retry trial TRY::MY-SUITE.
10: [SET-TRY-DEBUG] Supply a new value for :DEBUG of TRY:TRY.
11: [RETRY] Retry SLIME interactive evaluation request.
In the SLIME
debugger, we press v
on the frame of the call to MY-SUITE
to
navigate to its definition, realize what the problem is and fix
FOO
:
(defun foo ()
5)
Now, we select the RETRY-TRIAL
restart, and on the retry
MY-SUITE
passes. The full output is:
MY-SUITE
SHOULD-WORK
⋅ (IS T)
⋅ SHOULD-WORK ⋅1
WARNING: redefining TRY::FOO in DEFUN
⊠ (IS (= #1=(FOO) 5))
where
#1# = 4
MY-SUITE retry #1
SHOULD-WORK
⋅ (IS T)
⋅ SHOULD-WORK ⋅1
⋅ (IS (= (FOO) 5))
⋅ MY-SUITE ⋅2
Instead of working interactively, one can fix the failing test and
rerun it. Now, let's fix MY-SUITE
and rerun it:
(deftest my-suite ()
(should-work)
(is nil))
(try 'my-suite)
.. MY-SUITE
.. SHOULD-WORK
.. ⋅ (IS T)
.. ⋅ SHOULD-WORK ⋅1
.. ⊠ (IS NIL)
.. ⊠ MY-SUITE ⊠1 ⋅1
..
==> #<TRIAL (MY-SUITE) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅1>
(deftest my-suite ()
(should-work)
(is t))
(try !)
.. MY-SUITE
.. ⋅ (IS T)
.. ⋅ MY-SUITE ⋅1
..
==> #<TRIAL (MY-SUITE) EXPECTED-SUCCESS 0.004s ⋅1>
Here, !
refers to the most recent TRIAL
returned by TRY
. When a
trial is passed to TRY
or is FUNCALL
ed, trials in it that match
the type in TRY
's RERUN
argument are rerun (here, UNEXPECTED
by
default). SHOULD-WORK
and its check are EXPECTED-SUCCESS
es,
hence they don't match UNEXPECTED
and are not rerun.
Conditional execution can be achieved simply testing the TRIAL
object returned by Tests.
(deftest my-suite ()
(when (passedp (should-work))
(is t :msg "a test that depends on SHOULD-WORK")
(when (is nil)
(is nil :msg "never run"))))
Sometimes, we do not know up front that a test should not be
executed. Calling SKIP-TRIAL
unwinds from the CURRENT-TRIAL
and sets
it skipped.
(deftest my-suite ()
(is t)
(skip-trial)
(is nil))
(my-suite)
==> #<TRIAL (MY-SUITE) SKIP 0.000s ⋅1>
In the above, (IS T)
was executed, but (IS NIL)
was not.
(deftest known-broken ()
(with-failure-expected (t)
(is nil)))
(known-broken)
.. KNOWN-BROKEN
.. × (IS NIL)
.. ⋅ KNOWN-BROKEN ×1
..
==> #<TRIAL (KNOWN-BROKEN) EXPECTED-SUCCESS 0.000s ×1>
×
marks EXPECTED-FAILURE
s. (WITH-SKIP (T) ...)
makes all checks
successes and failures EXPECTED
, which are counted in their own
*CATEGORIES*
by default but don't make the enclosing tests to fail.
Also see WITH-EXPECTED-OUTCOME
.
With *RUN-DEFTEST-WHEN*
, tests on in various EVAL-WHEN
situations.
To run tests on evaluation, as in SLIME C-M-x
, slime-eval-defun
:
(setq *run-deftest-when* :execute)
(deftest some-test ()
(is t))
.. SOME-TEST
.. ⋅ (IS T)
.. ⋅ SOME-TEST ⋅1
..
=> SOME-TEST
(setq *run-deftest-when* nil)
There is no direct support for fixtures in Try. One can easily write macros like the following.
(defvar *server* nil)
(defmacro with-xxx (&body body)
`(flet ((,with-xxx-body ()
,@body))
(if *server*
(with-xxx-body)
(with-server (make-expensive-server)
(with-xxx-body)))))
Plus, with support for selectively Rerunning Trials, the need for fixtures is lessened.
The suggested way of writing tests is to call test functions explicitly:
(defpackage :some-test-package
(:use #:common-lisp #:try))
(in-package :some-test-package)
(deftest test-all ()
(test-this)
(test-that))
(deftest test-this ()
(test-this/more))
(deftest test-this/more ()
(is t))
(deftest test-that ()
(is t))
(deftest not-called ()
(is t))
(defun test ()
(warn-on-tests-not-run ((find-package :some-test-package))
(try 'test-all)))
(test)
.. TEST-ALL
.. TEST-THIS
.. TEST-THIS/MORE
.. ⋅ (IS T)
.. ⋅ TEST-THIS/MORE ⋅1
.. ⋅ TEST-THIS ⋅1
.. TEST-THAT
.. ⋅ (IS T)
.. ⋅ TEST-THAT ⋅1
.. ⋅ TEST-ALL ⋅2
.. WARNING: Test NOT-CALLED not run.
==> #<TRIAL (TEST-ALL) EXPECTED-SUCCESS 0.012s ⋅2>
Note how the TEST
function uses WARN-ON-TESTS-NOT-RUN
to catch any
tests defined in SOME-TEST-PACKAGE
that were not run. Tests can be
deleted by FMAKUNBOUND
, UNINTERN
, or by redefining the function with
DEFUN
. Tests defined in a given package can be listed with
LIST-PACKAGE-TESTS
.
This style allows higher level tests to establish the dynamic environment necessary for lower level tests.
The Elisp mgl-try
interactive command runs a Try test and
displays its output in a lisp-mode
buffer with minor modes
outline-mode
and mgl-try-mode
. It is assumed that the lisp is
running under Slime. In the
buffer,
-
use
M-.
to visit a test function; -
move between
UNEXPECTED
events with keysp
andn
; -
move between events which are not
EXPECTED-SUCCESS
es withP
andN
; -
rerun the most recent trial (
TRY:!
) withr
(subject to the filtering described Rerunning Trials); -
rerun the most recently finished test with
R
(and all tests it calls); -
run an arbitrary test with
t
(defaults to symbol under point); -
some low-level outline mode commands are also given convenient bindings:
<tab> outline-cycle C-p outline-previous-visible-heading C-n outline-next-visible-heading U outline-up-heading
Load src/mgl-try.el
in Emacs.
If you installed Try with Quicklisp, the location of mgl-try.el
may change with updates, and you may want to copy the current
version of mgl-try.el
to a stable location:
(try:install-try-elisp "~/quicklisp/")
Then, assuming the Elisp file is in the quicklisp directory, add
something like this to your .emacs
:
(load "~/quicklisp/mgl-try.el")
-
[function] INSTALL-TRY-ELISP TARGET-DIR
Copy
mgl-try.el
distributed with this package toTARGET-DIR
.
Try is built around events implemented as CONDITION
s.
Matching the types of events to *DEBUG*
, *COUNT*
, *COLLECT*
, *RERUN*
,
*PRINT*
, and *DESCRIBE*
is what gives Try its flexibility.
The event hierarchy is fairly involved, so let's start in the middle.
The condition EVENT
has 4 disjoint subclasses:
-
TRIAL-START
, which corresponds to the entry to a test (see Tests), -
ERROR*
, an unexpectedCL:ERROR
(0
1
) or unadorned non-local exit.
(let (;; We don't want to debug nor print a backtrace for the error below.
(*debug* nil)
(*describe* nil))
;; signals TRIAL-START / VERDICT-ABORT* on entry / exit
(with-test (demo)
;; signals EXPECTED-RESULT-SUCCESS
(is t)
;; signals UNHANDLED-ERROR with a nested CL:ERROR
(error "xxx")))
.. DEMO ; TRIAL-START
.. ⋅ (IS T) ; EXPECTED-RESULT-SUCCESS (⋅)
.. ⊟ "xxx" (SIMPLE-ERROR) ; UNHANDLED-ERROR (⊟)
.. ⊟ DEMO ⊟1 ⋅1 ; VERDICT-ABORT* (⊟)
..
==> #<TRIAL (WITH-TEST (DEMO)) ABORT* 0.004s ⊟1 ⋅1>
The non-abstract condition classes of events that are actually signalled are called concrete.
TRIAL-START
is a concrete event class. RESULT
s and VERDICT
s have six
concrete subclasses:
-
EXPECTED-RESULT-SUCCESS
,UNEXPECTED-RESULT-SUCCESS
,EXPECTED-RESULT-FAILURE
,UNEXPECTED-RESULT-FAILURE
,RESULT-SKIP
,RESULT-ABORT*
-
EXPECTED-VERDICT-SUCCESS
,UNEXPECTED-VERDICT-SUCCESS
,EXPECTED-VERDICT-FAILURE
,UNEXPECTED-VERDICT-FAILURE
,VERDICT-SKIP
,VERDICT-ABORT*
ERROR*
is an abstract class with two concrete subclasses:
-
UNHANDLED-ERROR
, signalled when aCL:ERROR
(0
1
) reaches the handler set up byDEFTEST
orWITH-TEST
, or when the debugger is invoked. -
NLX
, signalled when no error was detected by the handler, but the trial finishes with a non-local exit.
These are the 15 concrete event classes.
These condition classes group various bits of the Concrete Events and the Middle Layer of Events for ease of reference.
Concrete event classes except TRIAL-START
are subclasses of
hyphen-separated words in their name. For example,
UNEXPECTED-RESULT-FAILURE
inherits from UNEXPECTED
, RESULT
, and
FAILURE
, so it matches types such as UNEXPECTED
or (AND UNEXPECTED RESULT)
.
-
[condition] EVENT
Common abstract superclass of all events in Try.
-
[condition] EXPECTED EVENT
Concrete condition classes with
EXPECTED
in their name are subclasses ofEXPECTED
.SKIP
is also a subclass ofEXPECTED
.
-
[condition] UNEXPECTED EVENT
Concrete condition classes with
UNEXPECTED
in their name are subclasses ofUNEXPECTED
.ABORT*
is also a subclass ofUNEXPECTED
.
-
[condition] SUCCESS EVENT
See Checks and Trial Verdicts for how
SUCCESS
orFAILURE
is decided.
-
[condition] ABORT* UNEXPECTED
-
[type] EXPECTED-SUCCESS
A shorthand for
(AND EXPECTED SUCCESS)
.
-
[type] UNEXPECTED-SUCCESS
A shorthand for
(AND UNEXPECTED SUCCESS)
.
-
[type] EXPECTED-FAILURE
A shorthand for
(AND EXPECTED FAILURE)
.
-
[type] UNEXPECTED-FAILURE
A shorthand for
(AND UNEXPECTED FAILURE)
.
-
[type] PASS
An
OUTCOME
that's not anABORT*
or anUNEXPECTED
FAILURE
.
-
[type] FAIL
An
ABORT*
or anUNEXPECTED
FAILURE
.
-
[variable] *EVENT-PRINT-BINDINGS* ((*PRINT-CIRCLE* T))
EVENT
s are conditions signalled in code that may change printer variables such as*PRINT-CIRCLE*
,*PRINT-LENGTH*
, etc. To control how events are printed, the list of variable bindings in*EVENT-PRINT-BINDINGS*
is established whenever anEVENT
is printed as if with:(progv (mapcar #'first *event-print-bindings*) (mapcar #'second *event-print-bindings*) ...)
The default value ensures that shared structure is recognized (see Captures). If the
#N#
syntax feels cumbersome, then change this variable.
Only RECORD-EVENT
is applicable to all EVENT
s. See
Check Restarts, Trial Restarts for more.
-
[function] RECORD-EVENT &OPTIONAL CONDITION
This restart is always the first restart available when an
EVENT
is signalled running underTRY
(i.e. there is aCURRENT-TRIAL
).TRY
always invokesRECORD-EVENT
when handling events.
-
[condition] OUTCOME EVENT
An
OUTCOME
is the resolution of either aTRIAL
or a check (see Checks), corresponding to subclassesVERDICT
andRESULT
.
-
[macro] WITH-EXPECTED-OUTCOME (EXPECTED-TYPE) &BODY BODY
When an
OUTCOME
is to be signalled,EXPECTED-TYPE
determines whether it's going to beEXPECTED
. The concreteOUTCOME
classes are{EXPECTED,UNEXPECTED}-{RESULT,VERDICT}-{SUCCESS,FAILURE}
(see Events), of whichRESULT
orVERDICT
andSUCCESS
orFAILURE
are already known. If aRESULT
FAILURE
is to be signalled, then the moral equivalent of(SUBTYPEP '(AND RESULT FAILURE) EXPECTED-TYPE)
is evaluated and depending on whether it's true,EXPECTED-RESULT-FAILURE
orUNEXPECTED-RESULT-FAILURE
is signalled.By default,
SUCCESS
is expected. The following example shows how to expect bothSUCCESS
andFAILURE
forRESULT
s, while requiringVERDICT
s to succeed:(let ((*debug* nil)) (with-expected-outcome ('(or result (and verdict success))) (with-test (t1) (is nil)))) .. T1 .. × (IS NIL) .. ⋅ T1 ×1 .. ==> #<TRIAL (WITH-TEST (T1)) EXPECTED-SUCCESS 0.000s ×1>
This is equivalent to
(WITH-FAILURE-EXPECTED () ...)
. To make result failures expected but result successes unexpected:(let ((*debug* nil)) (with-expected-outcome ('(or (and result failure) (and verdict success))) (with-test (t1) (is t) (is nil)))) .. T1 .. ⊡ (IS T) .. × (IS NIL) .. ⋅ T1 ⊡1 ×1 .. ==> #<TRIAL (WITH-TEST (T1)) EXPECTED-SUCCESS 0.000s ⊡1 ×1>
This is equivalent to
(WITH-FAILURE-EXPECTED ('FAILURE) ...)
. The final example leaves result failures unexpected but makes both verdict successes and failures expected:(let ((*debug* nil)) (with-expected-outcome ('(or (and result success) verdict)) (with-test (t1) (is nil)))) .. T1 .. ⊠ (IS NIL) .. × T1 ⊠1 .. ==> #<TRIAL (WITH-TEST (T1)) EXPECTED-FAILURE 0.004s ⊠1>
-
[macro] WITH-FAILURE-EXPECTED (&OPTIONAL (RESULT-EXPECTED-TYPE T) (VERDICT-EXPECTED-TYPE ''SUCCESS)) &BODY BODY
A convenience macro on top of
WITH-EXPECTED-OUTCOME
,WITH-FAILURE-EXPECTED
expectsVERDICT
s to haveVERDICT-EXPECTED-TYPE
andRESULT
s to haveRESULT-EXPECTED-TYPE
. A simple(WITH-FAILURE-EXPECTED () ...)
makes allRESULT
SUCCESS
es andFAILURE
sEXPECTED
.(WITH-FAILURE-EXPECTED ('FAILURE) ..)
expectsFAILURE
s only, and anySUCCESS
es will beUNEXPECTED
.
-
[macro] WITH-SKIP (&OPTIONAL (SKIP T)) &BODY BODY
WITH-SKIP
skips checks and trials. It forces an immediateSKIP-TRIAL
whenever a trial is started (which turns into aVERDICT-SKIP
) and makes checks (without intervening trials, of course) evaluate normally but signalRESULT-SKIP
.SKIP
isNIL
cancels the effect of any enclosingWITH-SKIP
withSKIP
true.
-
[function] FORCE-EXPECTED-SUCCESS &OPTIONAL CONDITION
Change the type of the
OUTCOME
being signalled toEXPECTED
andSUCCESS
. If the original condition is aRESULT
, then this will beEXPECTED-RESULT-SUCCESS
, if it is aVERDICT
, thenEXPECTED-VERDICT-SUCCESS
.
-
[function] FORCE-UNEXPECTED-SUCCESS &OPTIONAL CONDITION
Change the type of
OUTCOME
being signalled toUNEXPECTED
andSUCCESS
.
-
[function] FORCE-EXPECTED-FAILURE &OPTIONAL CONDITION
Change the type of
OUTCOME
being signalled toEXPECTED
andFAILURE
.
-
[function] FORCE-UNEXPECTED-FAILURE &OPTIONAL CONDITION
Change the type of
OUTCOME
being signalled toUNEXPECTED
andFAILURE
.
Checks are like CL:ASSERT
s, they check whether some condition holds
and signal an OUTCOME
. The outcome signalled for checks is a
subclass of RESULT
.
Take, for example, (IS (= X 5))
. Depending on whether X
is
indeed 5, some kind of RESULT
SUCCESS
or FAILURE
will be signalled.
WITH-EXPECTED-OUTCOME
determines whether it's EXPECTED
or
UNEXPECTED
, and we have one of EXPECTED-RESULT-SUCCESS
,
UNEXPECTED-RESULT-SUCCESS
, EXPECTED-RESULT-FAILURE
,
UNEXPECTED-RESULT-FAILURE
to signal. Furthermore, if WITH-SKIP
is in
effect, then RESULT-SKIP
is signalled.
The result is signalled with #'SIGNAL
if it is a PASS
, else it's
signalled with #'ERROR
. This distinction matters only if the event
is not handled, which is never the case in a TRIAL
. Standalone
checks though - those that are not enclosed by a trial - invoke the
debugger on RESULT
s which are not of type PASS
.
The signalled RESULT
is not final until RECORD-EVENT
is invoked on
it, and it can be changed with the Outcome Restarts and the
Check Restarts.
- [condition] UNEXPECTED-RESULT-SUCCESS UNEXPECTED RESULT SUCCESS
- [condition] UNEXPECTED-RESULT-FAILURE UNEXPECTED RESULT FAILURE
-
[function] ABORT-CHECK &OPTIONAL CONDITION
Change the
OUTCOME
of the check being signalled toRESULT-ABORT*
.RESULT-ABORT*
, being(NOT PASS)
, will cause the check to returnNIL
ifRECORD-EVENT
is invoked on it.
-
[function] SKIP-CHECK &OPTIONAL CONDITION
Change the
OUTCOME
of the check being signalled toRESULT-SKIP
.RESULT-SKIP
, being aPASS
, will cause the check to returnT
ifCONTINUE
(0
1
) orRECORD-EVENT
is invoked on it.
-
[function] RETRY-CHECK &OPTIONAL CONDITION
Initiate a non-local exit to go reevaluate the forms wrapped by the check without signalling an
OUTCOME
.
-
[class] TRIAL SB-MOP:FUNCALLABLE-STANDARD-OBJECT
Trials are records of calls to tests (see Counting Events, Collecting Events). Their behaviour as funcallable instances is explained in Rerunning Trials.
There are three ways to acquire a
TRIAL
object: by callingCURRENT-TRIAL
, through the lexical binding of the symbol that names the test or through the return value of a test:(deftest xxx () (prin1 xxx)) (xxx) .. #<TRIAL (XXX) RUNNING> ==> #<TRIAL (XXX) EXPECTED-SUCCESS 0.000s>
WITH-TRIAL
can also provide access to itsTRIAL
:(with-test (t0) (prin1 t0)) .. #<TRIAL (WITH-TEST (T0)) RUNNING> ==> #<TRIAL (WITH-TEST (T0)) EXPECTED-SUCCESS 0.000s>
TRIAL
s are not to be instantiated by client code.
-
[function] CURRENT-TRIAL
TRIAL
s, like the calls to tests they stand for, nest.CURRENT-TRIAL
returns the innermost trial. If there is no currently running test, then an error is signalled. The returned trial isRUNNINGP
.
-
[condition] TRIAL-EVENT EVENT
A
TRIAL-EVENT
is either aTRIAL-START
or aVERDICT
.
- [reader] TRIAL TRIAL-EVENT (:TRIAL)
-
[condition] TRIAL-START TRIAL-EVENT
TRIAL-START
is signalled when a test function (see Tests) is entered and aTRIAL
is started, it is already theCURRENT-TRIAL
, and the Trial Restarts are available. It is also signalled when a trial is retried:(let ((*print* nil) (n 0)) (with-test () (handler-bind ((trial-start (lambda (c) (format t "TRIAL-START for ~S retry#~S~%" (test-name (trial c)) (n-retries (trial c)))))) (with-test (this) (incf n) (when (< n 3) (retry-trial)))))) .. TRIAL-START for THIS retry#0 .. TRIAL-START for THIS retry#1 .. TRIAL-START for THIS retry#2 ..
The matching of
TRIAL-START
events is less straightforward than that of otherEVENT
s.-
When a
TRIAL-START
event matches theCOLLECT
type (see Collecting Events), itsTRIAL
is collected. -
Similarly, when a
TRIAL-START
matches thePRINT
type (see Printing Events), it is printed immediately, and its trial'sVERDICT
will be printed too regardless of whether it matchesPRINT
. IfTRIAL-START
does not matchPRINT
, it may still be printed if for example*PRINT-PARENT*
requires it. -
When a
TRIAL-START
matches theRERUN
type (see Rerunning Trials), itsTRIAL
may be rerun. -
Also, see
WITH-SKIP
.
-
-
[condition] VERDICT TRIAL-EVENT OUTCOME
A
VERDICT
is theOUTCOME
of aTRIAL
. It is one of{EXPECTED,UNEXPECTED}-VERDICT-{SUCCESS,FAILURE}
,VERDICT-SKIP
andVERDICT-ABORT*
. Regarding how the verdict type is determined, see Trial Verdicts.Verdicts are signalled while their
TRIAL
is still theCURRENT-TRIAL
, and Trial Restarts are still available.(try (lambda () (handler-bind (((and verdict failure) #'retry-trial)) (with-test (this) (is (zerop (random 2))))))) .. (TRY #<FUNCTION (LAMBDA ()) {53038ADB}>) .. THIS .. ⊠ (IS (ZEROP #1=(RANDOM 2))) .. where .. #1# = 1 .. THIS retry #1 .. ⋅ (IS (ZEROP (RANDOM 2))) .. ⋅ THIS ⋅1 .. ⋅ (TRY #<FUNCTION (LAMBDA ()) {53038ADB}>) ⋅1 .. ==> #<TRIAL (TRY #<FUNCTION (LAMBDA ()) {53038ADB}>) EXPECTED-SUCCESS 0.000s ⋅1>
- [condition] UNEXPECTED-VERDICT-SUCCESS UNEXPECTED VERDICT SUCCESS
- [condition] UNEXPECTED-VERDICT-FAILURE UNEXPECTED VERDICT FAILURE
When a trial finished, a VERDICT
is signalled. The verdict's type
is determined as follows.
-
It is a
VERDICT-SKIP
if-
SKIP-TRIAL
was called on the trial, or -
ABORT-TRIAL
,SKIP-TRIAL
, orRETRY-TRIAL
was called on an enclosing trial, and -
these were not overruled by a later
ABORT-TRIAL
orRETRY-TRIAL
on the trial.
-
-
It is a
VERDICT-ABORT*
ifABORT-TRIAL
was called on the trial, and it wasn't overruled by a laterSKIP-TRIAL
orRETRY-TRIAL
. -
If all children (including those not collected in
CHILDREN
) of the trialPASS
, then the verdict will be aSUCCESS
, else it will be aFAILURE
. -
Subject to the
WITH-EXPECTED-OUTCOME
in effect,{EXPECTED,UNEXPECTED}-VERDICT-{SUCCESS,FAILURE}
is the type of the verdict which will be signalled.
The verdict of this type is signalled, but its type can be changed
by the Outcome Restarts or the Trial Restarts before
RECORD-EVENT
is invoked on it.
-
[reader] VERDICT TRIAL (= NIL)
The
VERDICT
EVENT
signalled when thisTRIAL
finished orNIL
if it has not finished yet.
-
[function] RUNNINGP TRIAL
See if the function call associated with
TRIAL
has not returned yet. Trials that are not running have aVERDICT
and are said to be finished.
There are three restarts available for manipulating running
trials: ABORT-TRIAL
, SKIP-TRIAL
, and RETRY-TRIAL
. They may be
invoked programatically or from the debugger. ABORT-TRIAL
is also
invoked by TRY
when encountering UNHANDLED-ERROR
.
The functions below invoke one of these restarts associated with a
TRIAL
. It is an error to call them on trials that are not RUNNINGP
,
but they may be called on trials other than the CURRENT-TRIAL
. In
that case, any intervening trials are skipped.
;; Skipped trials are marked with '-' in the output.
(with-test (outer)
(with-test (inner)
(is t)
(skip-trial nil outer)))
.. OUTER
.. INNER
.. ⋅ (IS T)
.. - INNER ⋅1
.. - OUTER ⋅1
..
==> #<TRIAL (WITH-TEST (OUTER)) SKIP 0.000s ⋅1>
Furthermore, all three restarts initiate a non-local exit to
return from the trial. If during the unwinding of the stack, the
non-local-exit is cancelled (see cancelled non-local exit), the appropriate
restart will be invoked upon returning from the trial. In the
following example, the non-local exit from a skip is cancelled by a
THROW
.
(with-test (some-test)
(catch 'foo
(unwind-protect
(skip-trial)
(throw 'foo nil)))
(is t :msg "check after skip"))
.. SOME-TEST
.. ⋅ check after skip
.. - SOME-TEST ⋅1
..
==> #<TRIAL (WITH-TEST (SOME-TEST)) SKIP 0.000s ⋅1>
In the next example, the non-local exit from a skip is cancelled by
an ERROR
(0
1
), which triggers an ABORT-TRIAL
.
(let ((*debug* nil)
(*describe* nil))
(with-test (foo)
(unwind-protect
(skip-trial)
(error "xxx"))))
.. FOO
.. ⊟ "xxx" (SIMPLE-ERROR)
.. ⊟ FOO ⊟1
..
==> #<TRIAL (WITH-TEST (FOO)) ABORT* 0.000s ⊟1>
All three restarts may be invoked on any EVENT
, including the
trial's own TRIAL-START
and VERDICT
. If their CONDITION
argument is an EVENT
(RETRY-TRIAL
has a special case here), they
also record it (as in RECORD-EVENT
) to ensure that when they handle
an EVENT
in the debugger or programatically that event is not
dropped.
-
[function] ABORT-TRIAL &OPTIONAL CONDITION (TRIAL (CURRENT-TRIAL))
Invoke the
ABORT-TRIAL
restart of aRUNNINGP
TRIAL
.When
CONDITION
is aVERDICT
forTRIAL
,ABORT-TRIAL
signals a new verdict of typeVERDICT-ABORT*
. This behavior is similar to that ofABORT-CHECK
. Else, theABORT-TRIAL
restart may recordCONDITION
, then it initiates a non-local exit to return from the test function withVERDICT-ABORT*
. If during the unwindingSKIP-TRIAL
orRETRY-TRIAL
is called, then the abort is cancelled.Since
ABORT*
is anUNEXPECTED
EVENT
,ABORT-TRIAL
is rarely used programatically. Signalling any error in a trial that's not caught before the trial's handler catches it will get turned into anUNHANDLED-ERROR
, andTRY
will invokeABORT-TRIAL
with it. Thus, instead of invokingABORT-TRIAL
directly, signalling an error will often suffice.
-
[function] SKIP-TRIAL &OPTIONAL CONDITION (TRIAL (CURRENT-TRIAL))
Invoke the
SKIP-TRIAL
restart of aRUNNINGP
TRIAL
.When
CONDITION
is aVERDICT
forTRIAL
,SKIP-TRIAL
signals a new verdict of typeVERDICT-SKIP
. This behavior is similar to that ofSKIP-CHECK
. Else, theSKIP-TRIAL
restart may recordCONDITION
, then it initiates a non-local exit to return from the test function withVERDICT-SKIP
. If during the unwindingABORT-TRIAL
orRETRY-TRIAL
is called, then the skip is cancelled.(with-test (skipped) (handler-bind ((unexpected-result-failure #'skip-trial)) (is nil))) .. SKIPPED .. ⊠ (IS NIL) .. - SKIPPED ⊠1 .. ==> #<TRIAL (WITH-TEST (SKIPPED)) SKIP 0.000s ⊠1>
Invoking
SKIP-TRIAL
on theTRIAL
's ownTRIAL-START
skips the trial being started.(let ((*print* '(or outcome leaf))) (with-test (parent) (handler-bind ((trial-start #'skip-trial)) (with-test (child) (is nil))))) .. PARENT .. - CHILD .. ⋅ PARENT ..
-
[function] RETRY-TRIAL &OPTIONAL CONDITION (TRIAL (CURRENT-TRIAL))
Invoke the
RETRY-TRIAL
restart ofRUNNINGP
TRIAL
. TheRETRY-TRIAL
restart may recordCONDITION
, then it initiates a non-local exit to go back to the beginning of the test function. If the non-local exit completes, then-
(
N-RETRIES
TRIAL
) is incremented, -
collected results and trials are cleared (see Collecting Events),
-
counts are zeroed (see Counting Events), and
-
TRIAL-START
is signalled again.
If during the unwinding
ABORT-TRIAL
orSKIP-TRIAL
is called, then the retry is cancelled.CONDITION
(which may beNIL
) is recorded if it is anEVENT
but not theVERDICT
ofTRIAL
, and theRECORD-EVENT
restart is available. -
-
[reader] N-RETRIES TRIAL (:N-RETRIES = 0)
The number of times this
TRIAL
has been retried. SeeRETRY-TRIAL
.
-
[condition] ERROR* ABORT* TRIAL-EVENT LEAF
Either
UNHANDLED-ERROR
orNLX
,ERROR*
causes or represents abnormal termination of aTRIAL
.ABORT-TRIAL
can be called withERROR*
s, but there is little need for explicitly doing so asRECORD-EVENT
, whichTRY
invokes, takes care of this.
- [reader] TEST-NAME ERROR* (:TEST-NAME)
-
[condition] UNHANDLED-ERROR ERROR*
Signalled when an
CL:ERROR
condition reaches the handlers set upDEFTEST
orWITH-TEST
, or when their*DEBUGGER-HOOK*
is invoked with a condition that's not anEVENT
.
- [reader] NESTED-CONDITION UNHANDLED-ERROR (:CONDITION = 'NIL)
- [reader] BACKTRACE-OF UNHANDLED-ERROR (:BACKTRACE = 'NIL)
- [reader] DEBUGGER-INVOKED-P UNHANDLED-ERROR (:DEBUGGER-INVOKED-P = 'NIL)
-
[variable] *GATHER-BACKTRACE* T
Capturing the backtrace can be expensive.
*GATHER-BACKTRACE*
controls whetherUNHANDLED-ERROR
s shall have theirBACKTRACE-OF
populated.
-
[condition] NLX ERROR*
Representing a non-local exit of unknown origin, this is signalled if a
TRIAL
does not return normally although it should have because it was not dismissed (seeDISMISSAL
,SKIP-TRIAL
,ABORT-TRIAL
). In this case, there is noCL:ERROR
(0
1
) associated with the event.
Categories determine how event types are printed and events of what types are counted together.
The default value of *CATEGORIES*
is
((abort* :marker "⊟")
(unexpected-failure :marker "⊠")
(unexpected-success :marker "⊡")
(skip :marker "-")
(expected-failure :marker "×")
(expected-success :marker "⋅"))
which says that all concrete EVENT
s that are of type ABORT*
(i.e.
RESULT-ABORT*
, VERDICT-ABORT*
, UNHANDLED-ERROR
, and NLX
) are to
be marked with "⊟"
when printed (see Printing Events). Also, the six
types define six counters for Counting Events. Note that UNEXPECTED
events have the same marker but squared as their EXPECTED
counterpart.
-
[variable] *CATEGORIES* "- see above -"
A list of of elements like
(TYPE &KEY MARKER)
. When Printing Events, Concrete Events are printed with the marker of the first matching type. When Counting Events, the counts associated with all matching types are incremented.
-
[function] FANCY-STD-CATEGORIES
Returns the default value of
*CATEGORIES*
(see Categories), which contains some fancy Unicode characters.
-
[function] ASCII-STD-CATEGORIES
Returns a value suitable for
*CATEGORIES*
, which uses only ASCII characters for the markers.'((abort* :marker "!") (unexpected-failure :marker "F") (unexpected-success :marker ":") (skip :marker "-") (expected-failure :marker "f") (expected-success :marker "."))
IS
is the fundamental one among Checks, on which all
the others are built, and it is a replacement for CL:ASSERT
that can
capture values of subforms to provide context to failures:
(is (= (1+ 5) 0))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS (= #1=(1+ 5) 0))
.. where
.. #1# = 6
IS
automatically captures values of arguments to functions like 1+
in the above example. Values of other interesting subforms can be
explicitly requested to be captured. IS
supports capturing multiple
values and can be taught how to deal with macros. The combination of
these features allows MATCH-VALUES
to be implementable as tiny
extension:
(is (match-values (values (1+ 5) "sdf")
(= * 0)
(string= * "sdf")))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS
.. (MATCH-VALUES #1=(VALUES (1+ 5) #2="sdf")
.. (= * 0)
.. (STRING= * "sdf")))
.. where
.. #1# == 6
.. #2#
IS
is flexible enough that all other checks (SIGNALS
, SIGNALS-NOT
,
INVOKES-DEBUGGER
, INVOKES-DEBUGGER-NOT
, FAILS
, and IN-TIME
are built
on top of it.
-
[macro] IS FORM &KEY MSG CTX (CAPTURE T) (PRINT-CAPTURES T) (RETRY T)
Evaluate
FORM
and signal aRESULT
SUCCESS
if its first return value is notNIL
, else signal aRESULT
FAILURE
(see Outcomes).IS
returns normally if-
the
RECORD-EVENT
restart is invoked (available when running in a trial), or -
the
CONTINUE
restart is invoked (available when not running in a trial), or -
the signalled
RESULT
condition is not handled (possible only when not running in a trial, and the result is aPASS
).
The return value of
IS
isT
if the last condition signalled is aSUCCESS
, andNIL
otherwise.MSG
andCTX
are Format Specifier Forms.MSG
prints a description of the check being made, which is by default the wholeIS
form. Due to how conditions are printed,MSG
says what the desired outcome is, andCTX
provides information about the evaluation.(is (equal (prin1-to-string 'hello) "hello") :msg "Symbols are replacements for strings." :ctx ("*PACKAGE* is ~S and *PRINT-CASE* is ~S~%" *package* *print-case*)) .. debugger invoked on UNEXPECTED-RESULT-FAILURE: .. UNEXPECTED-FAILURE in check: .. Symbols are replacements for strings. .. where .. (PRIN1-TO-STRING 'HELLO) = "HELLO" .. *PACKAGE* is #<PACKAGE "TRY"> and *PRINT-CASE* is :UPCASE ..
If
CAPTURE
is true, the value(s) of some subforms ofFORM
may be automatically recorded in the condition and also made available forCTX
via*IS-CAPTURES*
. See Captures for more.If
PRINT-CAPTURES
is true, the captures made are printed when theRESULT
condition is displayed in the debugger or*DESCRIBE*
d (see Printing Events). This is thewhere (PRIN1-TO-STRING 'HELLO) ="HELLO"
part above. IfPRINT-CAPTURES
isNIL
, the captures are still available in*IS-CAPTURES*
for writing customCTX
messages.If
RETRY
is true, then theRETRY-CHECK
restart evaluatesFORM
again and signals a newRESULT
. IfRETRY
isNIL
, then theRETRY-CHECK
restart returns:RETRY
, which allows complex checks such asSIGNALS
to implement their own retry mechanism. -
-
[variable] *IS-FORM*
IS
binds this to itsFORM
argument forCTX
andMSG
.
-
[variable] *IS-CAPTURES*
Captures made during an
IS
evaluation are made available forCTX
via*IS-CAPTURES*
.
A format specifier form is a Lisp form, typically an argument to
macro, standing for the FORMAT-CONTROL
and FORMAT-ARGS
arguments to
the FORMAT
function.
It may be a constant string:
(is nil :msg "FORMAT-CONTROL~%with no args.")
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. FORMAT-CONTROL
.. with no args.
It may be a list whose first element is a constant string, and the rest are the format arguments to be evaluated:
(is nil :msg ("Implicit LIST ~A." "form"))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. Implicit LIST form.
Or it may be a form that evaluates to a list like (FORMAT-CONTROL &REST FORMAT-ARGS)
:
(is nil :msg (list "Full ~A." "form"))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. Full form.
Finally, it may evaluate to NIL
, in which case some context specific
default is implied.
-
[function] CANONICALIZE-FORMAT-SPECIFIER-FORM FORM
Ensure that the format specifier form
FORM
is in its full form.
During the evaluation of the FORM
argument of IS
, evaluation of any
form (e.g. a subform of FORM
) may be recorded, which are called
captures.
IS
automatically captures some subforms of FORM
that are likely
to be informative. In particular, if FORM
is a function call, then
non-constant arguments are automatically captured:
(is (= 3 (1+ 2) (- 4 3)))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS (= 3 #1=(1+ 2) #2=(- 4 3)))
.. where
.. #1# = 3
.. #2# = 1
By default, automatic captures are not made for subforms deeper in
FORM
, except for when FORM
is a call to NULL
,
ENDP
and NOT
:
(is (null (find (1+ 1) '(1 2 3))))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS (NULL #1=(FIND #2=(1+ 1) '(1 2 3))))
.. where
.. #2# = 2
.. #1# = 2
(is (endp (member (1+ 1) '(1 2 3))))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS (ENDP #1=(MEMBER #2=(1+ 1) '(1 2 3))))
.. where
.. #2# = 2
.. #1# = (2 3)
Note that the argument of NOT
is not captured as it is
assumed to be NIL
or T
. If that's not true, use NULL
.
(is (not (equal (1+ 5) 6)))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS (NOT (EQUAL #1=(1+ 5) 6)))
.. where
.. #1# = 6
Other automatic captures are discussed with the relevant
functionality such as MATCH-VALUES
.
-
[class] SUB STRUCTURE-OBJECT
A
SUB
(short for substitution) says that in the original formIS
is checking, aSUBFORM
was substituted (bySUBSTITUTE-IS-FORM
) withVAR
(ifVALUESP
isNIL
) or with (VALUES-LIST
VAR
) ifVALUESP
is true. Conversely,VAR
is to be bound to the evaluatedNEW-FORM
ifVALUESP
isNIL
, and to (MULTIPLE-VALUE-LIST
FORM
) ifVALUESP
.NEW-FORM
is oftenEQ
toSUBFORM
, but it may be different, which is the case when further substitutions are made within a substitution.
- [function] MAKE-SUB VAR SUBFORM NEW-FORM VALUESP
- [structure-accessor] SUB-VAR SUB
- [structure-accessor] SUB-SUBFORM SUB
- [structure-accessor] SUB-NEW-FORM SUB
- [structure-accessor] SUB-VALUESP SUB
-
[generic-function] SUBSTITUTE-IS-LIST-FORM FIRST FORM ENV
In the list
FORM
, whoseCAR
isFIRST
, substitute subexpressions of interest with aGENSYM
and return the new form. As the second value, return a list ofSUB
s.For example, consider
(IS (FIND (FOO) LIST))
. WhenSUBSTITUTE-IS-LIST-FORM
is invoked on(FIND (FOO) LIST)
, it substitutes each argument ofFIND
with a variable, returning the new form(FIND TEMP1 TEMP2)
and the list of two substitutions((TEMP2 (FOO) (FOO) NIL) (TEMP3 LIST LIST NIL))
. This allows the original form to be rewritten as(let* ((temp1 (foo)) (temp2 list)) (find temp1 temp2))
TEMP1 and TEMP2 may then be reported in the
OUTCOME
condition signalled byIS
like this:The following check failed: (is (find #1=(foo) #2=list)) where #1# = <return-value-of-foo> #2# = <value-of-variable-list>
In addition to automatic captures, which are prescribed by rewriting rules (see Writing Automatic Capture Rules), explicit, ad-hoc captures can also be made.
(is (let ((x 1))
(= (capture x) 2)))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS
.. (LET ((X 1))
.. (= (CAPTURE X) 2)))
.. where
.. X = 1
If CAPTURE
showing up in the form that IS
prints is undesirable,
then %
may be used instead:
(is (let ((x 1))
(= (% x) 2)))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS
.. (LET ((X 1))
.. (= X 2)))
.. where
.. X = 1
Multiple values may be captured with CAPTURE-VALUES
and its
secretive counterpart %%
:
(is (= (%% (values 1 2)) 2))
.. debugger invoked on UNEXPECTED-RESULT-FAILURE:
.. UNEXPECTED-FAILURE in check:
.. (IS (= #1=(VALUES 1 2) 2))
.. where
.. #1# == 1
.. 2
where printing ==
instead of = indicates that this
is a multiple value capture.
-
[macro] CAPTURE FORM
Evaluate
FORM
, record its primary return value if within the dynamic extent of anIS
evaluation, and finally return that value. IfCAPTURE
is used within the lexical scope ofIS
, thenCAPTURE
itself will show up in the form that the defaultMSG
prints. Thus it is recommended to use the equivalentMACROLET
%
in the lexical scope as%
is removed before printing.
-
[macro] CAPTURE-VALUES FORM
Like
CAPTURE-VALUES
, but record and return all values returned byFORM
. It is recommended to use the equivalentMACROLET
%%
in the lexical scope as%%
is removed before printing.
-
[macrolet] % FORM
An alias for
CAPTURE
in the lexical scope ofIS
. Removed from theIS
form when printed.
-
[macrolet] %% FORM
An alias for
CAPTURE-VALUES
in the lexical scope ofIS
. Removed from theIS
form when printed.
In the following, various checks built on top of IS
are described.
Many of them share a number of arguments, which are described here.
-
ON-RETURN
is a boolean that determines whether the check in a macro that wrapsBODY
is made whenBODY
returns normally. -
ON-NLX
is a boolean that determines whether the check in a macro that wrapsBODY
is made whenBODY
performs a non-local exit. -
MSG
andCTX
are Format Specifier Forms as inIS
. -
NAME
may be provided so that it is printed (withPRIN1
) instead ofBODY
inMSG
.
The macros SIGNALS
, SIGNALS-NOT
, INVOKES-DEBUGGER
, and
INVOKES-DEBUGGER-NOT
all check whether a condition of a given type,
possibly also matching a predicate, was signalled. In addition to
those already described in Check Library, these macros share a
number of arguments.
Matching conditions are those that are of type CONDITION-TYPE
(not
evaluated) and satisfy the predicate PRED
.
When PRED
is NIL
, it always matches. When it is a string, then it
matches if it is a substring of the printed representation of the
condition being handled (by PRINC
under WITH-STANDARD-IO-SYNTAX
).
When it is a function, it matches if it returns true when called
with the condition as its argument.
The check is performed in the cleanup form of an UNWIND-PROTECT
around BODY
.
HANDLER
is called when a matching condition is found. It can be a
function, T
, or NIL
. When it is a function, it is called from the
condition handler (SIGNALS
and SIGNALS-NOT
) or the debugger
hook (invokes-debugger and INVOKES-DEBUGGER-NOT
) with the matching
condition. HANDLER
may perform a non-local exit. When HANDLER
is T
,
the matching condition is handled by performing a non-local exit to
just outside BODY
. If the exit completes, BODY
is treated as if it
had returned normally, and ON-RETURN
is consulted. When HANDLER
is
NIL
, no addition action is performed when a matching condition is
found.
The default CTX
describes the result of the matching process in
terms of *CONDITION-MATCHED-P*
and *BEST-MATCHING-CONDITION*
.
-
[variable] *CONDITION-MATCHED-P*
When a check described in Checking Conditions signals its
OUTCOME
, this variable is bound to a boolean value to indicate whether a condition that matchedCONDITION-TYPE
andPRED
was found.
-
[variable] *BEST-MATCHING-CONDITION*
Bound when a check described in Checking Conditions signals its
OUTCOME
. If*CONDITION-MATCHED-P*
, then it is the most recent condition that matched bothCONDITION-TYPE
andPRED
. Else, it is the most recent condition that matchedCONDITION-TYPE
orNIL
if no such conditions were detected.
-
[macro] SIGNALS (CONDITION-TYPE &KEY PRED (HANDLER T) (ON-RETURN T) (ON-NLX T) NAME MSG CTX) &BODY BODY
Check that
BODY
signals aCONDITION
ofCONDITION-TYPE
(not evaluated) that matchesPRED
. To detect matching conditions,SIGNALS
sets up aHANDLER-BIND
. Thus it can only see whatBODY
does not handle. The arguments are described in Checking Conditions.(signals (error) (error "xxx")) => NIL
The following example shows a failure where
CONDITION-TYPE
matches butPRED
does not.(signals (error :pred "non-matching") (error "xxx")) .. debugger invoked on UNEXPECTED-RESULT-FAILURE: .. UNEXPECTED-FAILURE in check: .. (ERROR "xxx") signals a condition of type ERROR that matches .. "non-matching". .. The predicate did not match "xxx".
-
[macro] SIGNALS-NOT (CONDITION-TYPE &KEY PRED (HANDLER T) (ON-RETURN T) (ON-NLX T) NAME MSG CTX) &BODY BODY
Check that
BODY
does not signal aCONDITION
ofCONDITION-TYPE
(not evaluated) that matchesPRED
. To detect matching conditions,SIGNALS-NOT
sets up aHANDLER-BIND
. Thus, it can only see whatBODY
does not handle. The arguments are described in Checking Conditions.
-
[macro] INVOKES-DEBUGGER (CONDITION-TYPE &KEY PRED (HANDLER T) (ON-RETURN T) (ON-NLX T) NAME MSG CTX) &BODY BODY
Check that
BODY
enters the debugger with aCONDITION
ofCONDITION-TYPE
(not evaluated) that matchesPRED
. To detect matching conditions,INVOKES-DEBUGGER
sets up a*DEBUGGER-HOOK*
. Thus, if*DEBUGGER-HOOK*
is changed byBODY
, it may not detect the condition. The arguments are described in Checking Conditions.Note that in a trial (see
CURRENT-TRIAL
), allERROR
(0
1
)s are handled, and a*DEBUGGER-HOOK*
is set up (seeUNHANDLED-ERROR
). Thus, invoking debugger would normally cause the trial to abort.(invokes-debugger (error :pred "xxx") (handler-bind ((error #'invoke-debugger)) (error "xxx"))) => NIL
-
[macro] INVOKES-DEBUGGER-NOT (CONDITION-TYPE &KEY PRED (HANDLER T) (ON-RETURN T) (ON-NLX T) NAME MSG CTX) &BODY BODY
Check that
BODY
does not enter the debugger with aCONDITION
ofCONDITION-TYPE
(not evaluated) that matchesPRED
. To detect matching conditions,INVOKES-DEBUGGER-NOT
sets up a*DEBUGGER-HOOK*
. Thus, if*DEBUGGER-HOOK*
is changed byBODY
, it may not detect the condition. The arguments are described in Checking Conditions.
-
[macro] FAILS (&KEY NAME MSG CTX) &BODY BODY
Check that
BODY
performs a non-local exit but do not cancel it (see cancelled non-local exit). See Check Library for the descriptions of the other arguments.In the following example,
FAILS
signals aSUCCESS
.(catch 'foo (fails () (throw 'foo 7))) => 7
Next,
FAILS
signals anUNEXPECTED-FAILURE
becauseBODY
returns normally.(fails () (print 'hey)) .. .. HEY .. debugger invoked on UNEXPECTED-RESULT-FAILURE: .. UNEXPECTED-FAILURE in check: .. (PRINT 'HEY) does not return normally.
Note that there is no
FAILS-NOT
asWITH-TEST
fills that role.
-
[macro] IN-TIME (SECONDS &KEY (ON-RETURN T) (ON-NLX T) NAME MSG CTX) &BODY BODY
Check that
BODY
finishes inSECONDS
. See Check Library for the descriptions of the other arguments.(in-time (1) (sleep 2)) .. debugger invoked on UNEXPECTED-RESULT-FAILURE: .. UNEXPECTED-FAILURE in check: .. (SLEEP 2) finishes within 1s. .. Took 2.000s.
RETRY-CHECK
restarts timing.
-
[variable] *IN-TIME-ELAPSED-SECONDS*
Bound to the number of seconds passed during the evaluation of
BODY
whenIN-TIME
signals itsOUTCOME
.
These utilities are not checks (which signal OUTCOME
s) but simple
functions and macros that may be useful for writing IS
checks.
-
[macro] ON-VALUES FORM &BODY BODY
ON-VALUES
evaluatesFORM
and transforms its return values one by one based on forms inBODY
. The Nth value is replaced by the return value of the Nth form ofBODY
evaluated with*
bound to the Nth value. If the number of values exceeds the number of transformation forms inBODY
then the excess values are returned as is.(on-values (values 1 "abc" 7) (1+ *) (length *)) => 2 => 3 => 7
If the number of values is less than the number of transformation forms, then in later transformation forms
*
is bound toNIL
.(on-values (values) * *) => NIL => NIL
The first forms in
BODY
may be options. Options must precede transformation forms. With:TRUNCATE
T
, the excess values are discarded.(on-values (values 1 "abc" 7) (:truncate t) (1+ *) (length *)) => 2 => 3
The
:ON-LENGTH-MISMATCH
option may beNIL
or a function of a single argument. If the number of values and the number of transformation forms are different, then this function is called to transform the list of values.:TRUNCATE
is handled before:ON-LENGTH-MISMATCH
.(on-values 1 (:on-length-mismatch (lambda (values) (if (= (length values) 1) (append values '("abc")) values))) (1+ *) *) => 2 => "abc"
If the same option is specified multiple times, the first one is in effect.
-
[macro] MATCH-VALUES FORM &BODY BODY
MATCH-VALUES
returns true iff all return values ofFORM
satisfy the predicates given byBODY
, which are described inON-VALUES
. The:TRUNCATE
option ofON-VALUES
is supported, but:ON-LENGTH-MISMATCH
always returnsNIL
.;; no values (is (match-values (values))) ;; single value success (is (match-values 1 (= * 1))) ;; success with different types (is (match-values (values 1 "sdf") (= * 1) (string= * "sdf"))) ;; too few values (is (not (match-values 1 (= * 1) (string= * "sdf")))) ;; too many values (is (not (match-values (values 1 "sdf" 3) (= * 1) (string= * "sdf")))) ;; too many values, but truncated (is (match-values (values 1 "sdf" 3) (:truncate t) (= * 1) (string= * "sdf")))
-
[function] MISMATCH% SEQUENCE1 SEQUENCE2 &KEY FROM-END (TEST #'EQL) (START1 0) END1 (START2 0) END2 KEY MAX-PREFIX-LENGTH MAX-SUFFIX-LENGTH
Like
CL:MISMATCH
butCAPTURE
s and returns the common prefix and the mismatched suffixes. TheTEST-NOT
argument is deprecated by theCLHS
and is not supported. In addition, ifMAX-PREFIX-LENGTH
andMAX-SUFFIX-LENGTH
are non-NIL
, they must be non-negative integers, and they limit the number of elements in the prefix and the suffixes.(is (null (mismatch% '(1 2 3) '(1 2 4 5)))) .. debugger invoked on UNEXPECTED-RESULT-FAILURE: .. UNEXPECTED-FAILURE in check: .. (IS (NULL #1=(MISMATCH% '(1 2 3) '(1 2 4 5)))) .. where .. COMMON-PREFIX = (1 2) .. MISMATCHED-SUFFIX-1 = (3) .. MISMATCHED-SUFFIX-2 = (4 5) .. #1# = 2
(is (null (mismatch% "Hello, World!" "Hello, world!"))) .. debugger invoked on UNEXPECTED-RESULT-FAILURE: .. UNEXPECTED-FAILURE in check: .. (IS (NULL #1=(MISMATCH% "Hello, World!" "Hello, world!"))) .. where .. COMMON-PREFIX = "Hello, " .. MISMATCHED-SUFFIX-1 = "World!" .. MISMATCHED-SUFFIX-2 = "world!" .. #1# = 7
-
[function] DIFFERENT-ELEMENTS SEQUENCE1 SEQUENCE2 &KEY (PRED #'EQL) (MISSING :MISSING)
Return the different elements under
PRED
in the given sequences as a list of(:INDEX <INDEX> <E1> <E2>)
elements, whereE1
andE2
are elements ofSEQUENCE1
andSEQUENCE2
at<INDEX>
, respectively, and they may beMISSING
if the corresponding sequence is too short.(is (endp (different-elements '(1 2 3) '(1 b 3 d)))) .. debugger invoked on UNEXPECTED-RESULT-FAILURE: .. UNEXPECTED-FAILURE in check: .. (IS (ENDP #1=(DIFFERENT-ELEMENTS '(1 2 3) '(1 B 3 D)))) .. where .. #1# = ((:INDEX 1 2 B) (:INDEX 3 :MISSING D))
-
[function] SAME-SET-P LIST1 LIST2 &KEY KEY (TEST #'EQL)
See if
LIST1
andLIST2
represent the same set. SeeCL:SET-DIFFERENCE
for a description of theKEY
andTEST
arguments.(try:is (try:same-set-p '(1) '(2))) .. debugger invoked on UNEXPECTED-RESULT-FAILURE: .. UNEXPECTED-FAILURE in check: .. (IS (SAME-SET-P '(1) '(2))) .. where .. ONLY-IN-1 = (1) .. ONLY-IN-2 = (2)
-
[macro] WITH-SHUFFLING NIL &BODY BODY
Execute the forms that make up the list of forms
BODY
in random order and returnNIL
. This may be useful to prevent writing tests that accidentally depend on the order in which subtests are called.(loop repeat 3 do (with-shuffling () (prin1 1) (prin1 2))) .. 122112 => NIL
Float comparisons following https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/.
-
[function] FLOAT-~= X Y &KEY (MAX-DIFF-IN-VALUE *MAX-DIFF-IN-VALUE*) (MAX-DIFF-IN-ULP *MAX-DIFF-IN-ULP*)
Return whether two numbers,
X
andY
, are approximately equal either according toMAX-DIFF-IN-VALUE
orMAX-DIFF-IN-ULP
.If the absolute value of the difference of two floats is not greater than
MAX-DIFF-IN-VALUE
, then they are considered equal.If two floats are of the same sign and the number of representable floats (ULP, unit in the last place) between them is less than
MAX-DIFF-IN-ULP
, then they are considered equal.If neither
X
norY
are floats, then the comparison is done with=
. If one of them is aDOUBLE-FLOAT
, then the other is converted to a double float, and the comparison takes place in double float space. Else, both are converted toSINGLE-FLOAT
and the comparison takes place in single float space.
-
[variable] *MAX-DIFF-IN-VALUE* 1.0e-16
The default value of the
MAX-DIFF-IN-VALUE
argument ofFLOAT-~=
.
-
[variable] *MAX-DIFF-IN-ULP* 2
The default value of the
MAX-DIFF-IN-ULP
argument ofFLOAT-~=
.
-
[function] FLOAT-~< X Y &KEY (MAX-DIFF-IN-VALUE *MAX-DIFF-IN-VALUE*) (MAX-DIFF-IN-ULP *MAX-DIFF-IN-ULP*)
Return whether
X
is approximately less thanY
. Equivalent to<
, but it also allows for approximate equality according toFLOAT-~=
.
-
[function] FLOAT-~> X Y &KEY (MAX-DIFF-IN-VALUE *MAX-DIFF-IN-VALUE*) (MAX-DIFF-IN-ULP *MAX-DIFF-IN-ULP*)
Return whether
X
is approximately greater thanY
. Equivalent to>
, but it also allows for approximate equality according toFLOAT-~=
.
In Try, tests are Lisp functions that record their execution in
TRIAL
objects. TRIAL
s are to tests what function call traces are to
functions. In more detail, tests
-
create a
TRIAL
object and signal aTRIAL-START
event upon entry to the function, -
signal a
VERDICT
condition before returning normally or via a non-local exit, -
return the
TRIAL
object as the first value, -
return explicitly returned values as the second, third, and so on values.
See DEFTEST
and WITH-TEST
for more precise descriptions.
-
[macro] DEFTEST NAME LAMBDA-LIST &BODY BODY
DEFTEST
is a wrapper aroundDEFUN
to define global test functions. SeeDEFUN
for a description ofNAME
,LAMBDA-LIST
, andBODY
. The behaviour common withWITH-TEST
is described in Tests.(deftest my-test () (write-string "hey")) => MY-TEST (test-bound-p 'my-test) => T (my-test) .. hey ==> #<TRIAL (MY-TEST) EXPECTED-SUCCESS 0.000s>
Although the common case is for tests to have no arguments,
DEFTEST
supports general function lambda lists. Within a global test,-
NAME
is bound to theTRIAL
object -
the first return value is the trial
-
values are not returned implicitly
-
values returned with an explicit
RETURN-FROM
are returned as values after the trial
(deftest my-test () (prin1 my-test) (return-from my-test (values 2 3))) (my-test) .. #<TRIAL (MY-TEST) RUNNING> ==> #<TRIAL (MY-TEST) EXPECTED-SUCCESS 0.000s> => 2 => 3
-
-
[variable] *RUN-DEFTEST-WHEN* NIL
This may be any of
:COMPILE-TOPLEVEL
,:LOAD-TOPLEVEL
,:EXECUTE
, or a list thereof. The value of*RUN-DEFTEST-WHEN*
determines in whatEVAL-WHEN
situation to call the test function immediately after it has been defined withDEFTEST
.For interactive development, it may be convenient to set it to
:EXECUTE
and have the test run when theDEFTEST
is evaluated (maybe with SlimeC-M-x
,slime-eval-defun
). Or set it to:COMPILE-TOPLEVEL
, and have it rerun on SlimeC-c C-c
,slime-compile-defun
.If the test has required arguments, an argument list is prompted for and read from
*QUERY-IO*
.
-
[function] TEST-BOUND-P SYMBOL
See if
SYMBOL
names a global test (i.e. a test defined withDEFTEST
). If since the execution ofDEFTEST
, the symbol has beenUNINTERN
ed,FMAKUNBOUND
ed, or redefined withDEFUN
, then it no longer names a global test.
-
[macro] WITH-TEST (&OPTIONAL TRIAL-VAR &KEY NAME) &BODY BODY
Define a so-called lambda test to group together
CHECK
s and other tests it executes.WITH-TEST
executesBODY
in its lexical environment even on a rerun (see Rerunning Trials).If
TRIAL-VAR
is a non-NIL
symbol, bind it to the trial object.NAME
may be any type, it is purely for presentation purposes. IfNAME
isNIL
, then it defaults toTRIAL-VAR
.To facilitate returning values, a
BLOCK
is wrapped aroundBODY
. The name of the block isTRIAL-VAR
if it is a symbol, else it'sNIL
.When both
TRIAL-VAR
andNAME
are specified:(with-test (some-feature :name "obscure feature") (prin1 some-feature) (is t) (return-from some-feature (values 1 2))) .. #<TRIAL (WITH-TEST ("obscure feature")) RUNNING> .. obscure feature .. ⋅ (IS T) .. ⋅ obscure feature ⋅1 .. ==> #<TRIAL (WITH-TEST ("obscure feature")) EXPECTED-SUCCESS 0.002s ⋅1> => 1 => 2
If only
TRIAL-VAR
is specified:(with-test (some-feature) (prin1 some-feature) (is t) (return-from some-feature (values 1 2))) .. #<TRIAL (WITH-TEST (SOME-FEATURE)) RUNNING> .. SOME-FEATURE .. ⋅ (IS T) .. ⋅ SOME-FEATURE ⋅1 .. ==> #<TRIAL (WITH-TEST (SOME-FEATURE)) EXPECTED-SUCCESS 0.000s ⋅1> => 1 => 2
If neither is specified:
(with-test () (prin1 (current-trial)) (is t) (return (values 1 2))) .. #<TRIAL (WITH-TEST (NIL)) RUNNING> .. NIL .. ⋅ (IS T) .. ⋅ NIL ⋅1 .. ==> #<TRIAL (WITH-TEST (NIL)) EXPECTED-SUCCESS 0.000s ⋅1> => 1 => 2
Finally, using that
NAME
defaults toTRIAL-VAR
and that it is valid to specify non-symbols forTRIAL-VAR
, one can also write:(with-test ("Some feature") (prin1 (current-trial)) (is t) (return (values 1 2))) .. #<TRIAL (WITH-TEST ("Some feature")) RUNNING> .. Some feature .. ⋅ (IS T) .. ⋅ Some feature ⋅1 .. ==> #<TRIAL (WITH-TEST ("Some feature")) EXPECTED-SUCCESS 0.000s ⋅1> => 1 => 2
In summary and in contrast to global tests (those defined with
DEFTEST
), lambda tests-
have no arguments,
-
are defined and called at the same time,
-
may not bind their trial object to any variable,
-
may have a
BLOCK
namedNIL
, -
have a
NAME
purely for presentation purposes.
Lambda tests can be thought of as analogous to
(FUNCALL (LAMBDA () BODY))
. The presence of theLAMBDA
(0
1
) is important because it is stored in theTRIAL
object to support Rerunning Trials. -
-
[function] LIST-PACKAGE-TESTS &OPTIONAL (PACKAGE *PACKAGE*)
List all symbols in
PACKAGE
that name global tests in the sense ofTEST-BOUND-P
.
-
[macro] WITH-TESTS-RUN (TESTS-RUN) &BODY BODY
Bind the symbol
TESTS-RUN
to an emptyEQ
hash table and executeBODY
. The has table reflects call counts to global tests. Keys are symbols naming global tests, and the values are the number of times the keys have been called.
-
[macro] WARN-ON-TESTS-NOT-RUN (&OPTIONAL (PACKAGE *PACKAGE*)) &BODY BODY
A convenience utility to that records the global tests run by
BODY
withWITH-TESTS-RUN
and, whenBODY
finishes, signals a warning for each global tests inPACKAGE
not run.This is how Try runs its own tests:
(defun test () ;; Bind *PACKAGE* so that names of tests printed have package names, ;; and M-. works on them in Slime. (let ((*package* (find-package :common-lisp))) (warn-on-tests-not-run ((find-package :try)) (print (try 'test-all :print 'unexpected :describe 'unexpected)))))
Tests can be run explicitly by invoking the TRY
function or
implicitly by calling a test function:
(deftest my-test ()
(is t))
(my-test)
.. MY-TEST
.. ⋅ (IS T)
.. ⋅ MY-TEST ⋅1
..
==> #<TRIAL (MY-TEST) EXPECTED-SUCCESS 0.004s ⋅1>
The situation is similar with a WITH-TEST
:
(with-test (my-test)
(is t))
.. MY-TEST
.. ⋅ (IS T)
.. ⋅ MY-TEST ⋅1
..
==> #<TRIAL (WITH-TEST (MY-TEST)) EXPECTED-SUCCESS 0.000s ⋅1>
Behind the scenes, the outermost test function calls TRY
with
(try trial :debug *debug* :collect *collect* :rerun *rerun*
:print *print* :describe *describe*
:stream *stream* :printer *printer*)
TRY
then calls the test function belonging to TRIAL
.
The rest of the behaviour is described in Explicit TRY
.
-
[variable] *DEBUG* (AND UNEXPECTED (NOT NLX) (NOT VERDICT))
The default value makes
TRY
invoke the debugger onUNHANDLED-ERROR
,RESULT-ABORT*
,UNEXPECTED-RESULT-FAILURE
, andUNEXPECTED-RESULT-SUCCESS
.NLX
is excluded because it is caught as the test function is being exited, but by that time the dynamic environment of the actual cause is likely gone.VERDICT
is excluded because it is a consequence of its child outcomes.
-
[variable] *COUNT* LEAF
Although the default value of
*CATEGORIES*
lumpsRESULT
s andVERDICT
s together, with the default ofLEAF
,VERDICT
s are not counted. See Counting Events.
-
[variable] *COLLECT* UNEXPECTED
To save memory, only the
UNEXPECTED
are collected by default. See Collecting Events.
-
[variable] *RERUN* UNEXPECTED
The default matches that of
*COLLECT*
. See Rerunning Trials.
-
[variable] *PRINT* LEAF
With the default of
LEAF
combined with the default*PRINT-PARENT*
T
, onlyTRIAL
s with checks orERROR*
in them are printed. IfUNEXPECTED
, only the interesting things are printed. See Printing Events.
-
[variable] *DESCRIBE* (OR UNEXPECTED FAILURE)
By default, the context (e.g. Captures, and the
CTX
argument of is and other checks) ofUNEXPECTED
events is described. See Printing Events.
- [variable] *STREAM* (MAKE-SYNONYM-STREAM '*DEBUG-IO*)
- [variable] *PRINTER* TREE-PRINTER
Instead of invoking the test function directly, tests can also be
run by invoking the TRY
function.
(deftest my-test ()
(is t))
(try 'my-test)
.. MY-TEST
.. ⋅ (IS T)
.. ⋅ MY-TEST ⋅1
..
==> #<TRIAL (MY-TEST) EXPECTED-SUCCESS 0.000s ⋅1>
The situation is similar with a WITH-TEST
, only that TRY
wraps an
extra TRIAL
around the execution of the LAMBDA
(0
1
) to ensure that all
EVENT
s are signalled within a trial.
(try (lambda ()
(with-test (my-test)
(is t))))
.. (TRY #<FUNCTION (LAMBDA ()) {531FE50B}>)
.. MY-TEST
.. ⋅ (IS T)
.. ⋅ MY-TEST ⋅1
.. ⋅ (TRY #<FUNCTION (LAMBDA ()) {531FE50B}>) ⋅1
..
==> #<TRIAL (TRY #<FUNCTION (LAMBDA ()) {531FE50B}>) EXPECTED-SUCCESS 0.000s ⋅1>
Invoking tests with an explicit TRY
is very similar to just calling
the test functions directly (see Calling Test Functions). The differences
are that TRY
Those arguments default to *TRY-DEBUG*
, *TRY-COLLECT*
, etc, which
parallel and default to *DEBUG*
, *COLLECT*
, etc if set to
:UNSPECIFIED
. *TRY-DEBUG*
is NIL
, the rest of them are :UNSPECIFIED
.
These defaults encourage the use of an explicit TRY
call in the
non-interactive case and calling the test functions directly in the
interactive one, but this is not enforced in any way.
-
[function] TRY TESTABLE &KEY (DEBUG *TRY-DEBUG*) (COUNT *TRY-COUNT*) (COLLECT *TRY-COLLECT*) (RERUN *TRY-RERUN*) (PRINT *TRY-PRINT*) (DESCRIBE *TRY-DESCRIBE*) (STREAM *TRY-STREAM*) (PRINTER *TRY-PRINTER*)
TRY
runsTESTABLE
and handles theEVENT
s to collect, debug, print the results of checks and trials, and to decide what tests to skip and what to rerun.DEBUG
,COUNT
,COLLECT
,RERUN
,PRINT
, andDESCRIBE
must all be valid specifiers for types that are eitherNIL
(the empty type) or have a non-empty intersection with the typeEVENT
(e.g.T
,OUTCOME
,UNEXPECTED
,VERDICT
).TRY
sets up aHANDLER-BIND
handler forEVENT
s and runsTESTABLE
(see Testables). When anEVENT
is signalled, the handler matches its type to the value of theDEBUG
argument (in the sense of(TYPEP EVENT DEBUG)
). If it matches, then the debugger is invoked with the event. In the debugger, the user has a number of restarts available to change (see Event Restarts, Outcome Restarts, Check Restarts, Trial Restarts, andSET-TRY-DEBUG
.If the debugger is not invoked,
TRY
invokes the very first restart available, which is alwaysRECORD-EVENT
.Recording the event is performed as follows.
-
Outcome counts are updated (see Counting Events).
-
The event is passed to the collector (see Collecting Events).
-
The event is passed to the printer (see Printing Events).
-
Finally, when rerunning a trial (i.e. when
TESTABLE
is a trial), on aTRIAL-START
event, the trial may be skipped (see Rerunning Trials).
TRY
returns the values returned by the outermost trial (see Tests). -
-
[function] SET-TRY-DEBUG DEBUG
Invoke the
SET-TRY-DEBUG
restart to override theDEBUG
argument of the currently runningTRY
.DEBUG
must thus be a suitable type. When theSET-TRY-DEBUG
restart is invoked interactively,DEBUG
is read as a non-evaluated form from*QUERY-IO*
.
-
[variable] *TRY-DEBUG* NIL
The default value for
TRY
's:DEBUG
argument. If:UNSPECIFIED
, then the value of*DEBUG*
is used instead.
-
[variable] *TRY-COUNT* :UNSPECIFIED
The default value for
TRY
's:COUNT
argument. If:UNSPECIFIED
, then the value of*COUNT*
is used instead.
-
[variable] *TRY-COLLECT* :UNSPECIFIED
The default value for
TRY
's:COLLECT
argument. If:UNSPECIFIED
, then the value of*COLLECT*
is used instead.
-
[variable] *TRY-RERUN* :UNSPECIFIED
The default value for
TRY
's:RERUN
argument. If:UNSPECIFIED
, then the value of*RERUN*
is used instead.
-
[variable] *TRY-PRINT* :UNSPECIFIED
The default value for
TRY
's:PRINT
argument. If:UNSPECIFIED
, then the value of*PRINT*
is used instead.
-
[variable] *TRY-DESCRIBE* :UNSPECIFIED
The default value for
TRY
's:DESCRIBE
argument. If:UNSPECIFIED
, then the value of*DESCRIBE*
is used instead.
-
[variable] *TRY-STREAM* :UNSPECIFIED
The default value for
TRY
's:STREAM
argument. If:UNSPECIFIED
, then the value of*STREAM*
is used instead.
-
[variable] *TRY-PRINTER* :UNSPECIFIED
The default value for
TRY
's:PRINTER
argument. If:UNSPECIFIED
, then the value of*PRINTER*
is used instead.
-
[variable] *N-RECENT-TRIALS* 3
See
*RECENT-TRIALS*
.
-
[function] RECENT-TRIAL &OPTIONAL (N 0)
Returns the
N
th most recent trial orNIL
if there are not enough trials recorded. EveryTRIAL
returned byTRY
gets pushed onto a list of trials, but only*N-RECENT-TRIALS*
are kept.
-
[variable] ! NIL
The most recent trial. Equivalent to
(RECENT-TRIAL 0)
.
-
[variable] !! NIL
Equivalent to
(RECENT-TRIAL 1)
.
-
[variable] !!! NIL
Equivalent to
(RECENT-TRIAL 2)
.
Valid first arguments to TRY
are called testables. A testable may
be:
-
-
the name of a global test
-
the name of a global function
-
a function object
-
a trial
-
-
a list of testables
-
a
PACKAGE
In the function designator cases, TRY
calls the designated function.
TRIAL
s, being funcallable instances, designate themselves. If the
trial is not RUNNINGP
, then it will be rerun (see Rerunning Trials). Don't
invoke TRY
with RUNNINGP
trials (but see
Implementation of Implicit TRY
for discussion).
When given a list of testables, TRY
calls each testable one by one.
Finally, a PACKAGE
stands for the result of calling
LIST-PACKAGE-TESTS
on that package.
What's happening in the implementation is that a test function,
when it is called, checks whether it is running under the TRY
function. If it isn't, then it invokes TRY
with its TRIAL
. TRY
realizes the trial cannot be rerun yet (see Rerunning Trials) because it
is RUNNINGP
, sets up its event handlers for debugging, collecting,
printing, and invokes the trial as if it were rerun but without
skipping anything based on the RERUN
argument. Thus the following
are infinite recursions:
(with-test (recurse)
(try recurse))
(with-test (recurse)
(funcall recurse))
TRY
instantiates a printer of the type given by its PRINTER
argument. All EVENT
s recorded by TRY
are sent to this printer. The
printer then prints events that match the type given by the PRINT
argument of TRY
. Events that also match the DESCRIBE
argument of TRY
are printed with context information (see IS
) and backtraces (see
UNHANDLED-ERROR
).
Although the printing is primarily customized with global special
variables, changing the value of those variables after the printer
object is instantiated by TRY
has no effect. This is to ensure
consistent output with nested TRY
calls of differing printer
setups.
-
[class] TREE-PRINTER
TREE-PRINTER
prints events in an indented tree-like structure, with each internal node corresponding to aTRIAL
. This is the default printer (according to*PRINTER*
and*TRY-PRINTER*
) and currently the only one.The following example prints all Concrete Events.
(let ((*debug* nil) (*print* '(not trial-start)) (*describe* nil)) (with-test (verdict-abort*) (with-test (expected-verdict-success)) (with-expected-outcome ('failure) (with-test (unexpected-verdict-success))) (handler-bind (((and verdict success) #'force-expected-failure)) (with-test (expected-verdict-failure))) (handler-bind (((and verdict success) #'force-unexpected-failure)) (with-test (unexpected-verdict-failure))) (with-test (verdict-skip) (skip-trial)) (is t :msg "EXPECTED-RESULT-SUCCESS") (with-failure-expected ('failure) (is t :msg "UNEXPECTED-RESULT-SUCCESS") (is nil :msg "EXPECTED-RESULT-FAILURE")) (is nil :msg "UNEXPECTED-RESULT-FAILURE") (with-skip () (is nil :msg "RESULT-SKIP")) (handler-bind (((and result success) #'abort-check)) (is t :msg "RESULT-ABORT*")) (catch 'foo (with-test (nlx-test) (throw 'foo nil))) (error "UNHANDLED-ERROR"))) .. VERDICT-ABORT* ; TRIAL-START .. ⋅ EXPECTED-VERDICT-SUCCESS .. ⊡ UNEXPECTED-VERDICT-SUCCESS .. × EXPECTED-VERDICT-FAILURE .. ⊠ UNEXPECTED-VERDICT-FAILURE .. - VERDICT-SKIP .. ⋅ EXPECTED-RESULT-SUCCESS .. ⊡ UNEXPECTED-RESULT-SUCCESS .. × EXPECTED-RESULT-FAILURE .. ⊠ UNEXPECTED-RESULT-FAILURE .. - RESULT-SKIP .. ⊟ RESULT-ABORT* .. NLX-TEST ; TRIAL-START .. ⊟ non-local exit ; NLX .. ⊟ NLX-TEST ⊟1 ; VERDICT-ABORT* .. ⊟ "UNHANDLED-ERROR" (SIMPLE-ERROR) .. ⊟ VERDICT-ABORT* ⊟3 ⊠1 ⊡1 -1 ×1 ⋅1 .. ==> #<TRIAL (WITH-TEST (VERDICT-ABORT*)) ABORT* 0.004s ⊟3 ⊠1 ⊡1 -1 ×1 ⋅1>
The
⊟3 ⊠1 ⊡1 -1 ×1 ⋅1
part is the counts for*CATEGORIES*
printed with their markers.
-
[variable] *PRINT-PARENT* T
When an
EVENT
is signalled and its parentTRIAL
's type matches*PRINT-PARENT*
, the trial is printed as if itsTRIAL-START
matched thePRINT
argument ofTRY
.(let ((*print* 'leaf) (*print-parent* t)) (with-test (t0) (is t) (is t))) .. T0 .. ⋅ (IS T) .. ⋅ (IS T) .. ⋅ T0 ⋅2 .. ==> #<TRIAL (WITH-TEST (T0)) EXPECTED-SUCCESS 0.000s ⋅2>
(let ((*print* 'leaf) (*print-parent* nil)) (with-test (t0) (is t) (is t))) .. ⋅ (IS T) .. ⋅ (IS T) .. ==> #<TRIAL (WITH-TEST (T0)) EXPECTED-SUCCESS 0.000s ⋅2>
*PRINT-PARENT*
NIL
combined with printingVERDICT
s results in a flat output:(let ((*print* '(or leaf verdict)) (*print-parent* nil)) (with-test (outer) (with-test (inner) (is t :msg "inner-t")) (is t :msg "outer-t"))) .. ⋅ inner-t .. ⋅ INNER ⋅1 .. ⋅ outer-t .. ⋅ OUTER ⋅2 .. ==> #<TRIAL (WITH-TEST (OUTER)) EXPECTED-SUCCESS 0.000s ⋅2>
-
[variable] *PRINT-INDENTATION* 2
The number of spaces each printed
TRIAL
increases the indentation of its children.
-
[variable] *PRINT-DURATION* NIL
If true, the number of seconds spent during execution is printed.
(let ((*print-duration* t) (*debug* nil) (*describe* nil)) (with-test (timed) (is (progn (sleep 0.3) t)) (is (progn (sleep 0.2) t)) (error "xxx"))) .. TIMED .. 0.300 ⋅ (IS (PROGN (SLEEP 0.3) T)) .. 0.200 ⋅ (IS (PROGN (SLEEP 0.2) T)) .. ⊟ ""xxx (SIMPLE-ERROR) .. 0.504 ⊟ TIMED ⊟1 ⋅2 .. ==> #<TRIAL (WITH-TEST (TIMED)) ABORT* 0.504s ⊟1 ⋅2>
Timing is available for all
OUTCOME
s (i.e. for Checks andTRIAL
s). Checks generally measure the time spent during evaluation the form they are wrapping. Trials measure the time betweenTRIAL-START
and theVERDICT
.Timing information is not available for
TRIAL-START
andERROR*
events.
-
[variable] *PRINT-COMPACTLY* NIL
EVENT
s whose type matches*PRINT-COMPACTLY*
are printed less verbosely.LEAF
events are printed only with their marker, andVERDICT
s of trials without printed child trials are printed with=> <MARKER>
(see*CATEGORIES*
).(let ((*print-compactly* t) (*debug* nil) (*describe* nil)) (with-test (outer) (loop repeat 10 do (is t)) (with-test (inner) (is t) (is nil) (error "xxx")) (loop repeat 10 do (is t)))) .. OUTER ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ .. INNER ⋅⊠⊟ => ⊟ .. ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅ .. ⊠ OUTER ⊟1 ⊠1 ⋅21 .. ==> #<TRIAL (WITH-TEST (OUTER)) UNEXPECTED-FAILURE 0.000s ⊟1 ⊠1 ⋅21>
*PRINT-COMPACTLY*
has no effect on events beingDESCRIBE
d.
-
[variable] *DEFER-DESCRIBE* NIL
When an
EVENT
is to be*DESCRIBE*
d and its type matches*DEFER-DESCRIBE*
, then instead of printing the often longish context information in the tree of events, it is deferred until afterTRY
has finished. The following example only printsLEAF
events (due to*PRINT*
and*PRINT-PARENT*
) and in compact form (see*PRINT-COMPACTLY*
), deferring description of events matching*DESCRIBE*
until the end.(let ((*print* 'leaf) (*print-parent* nil) (*print-compactly* t) (*defer-describe* t) (*debug* nil)) (with-test (outer) (loop repeat 10 do (is t)) (with-test (inner) (is (= (1+ 5) 7))))) .. ⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⊠ .. .. ;; UNEXPECTED-RESULT-FAILURE (⊠) in OUTER INNER: .. (IS (= #1=(1+ 5) 7)) .. where .. #1# = 6 .. ==> #<TRIAL (WITH-TEST (OUTER)) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅10>
TRIAL
s have a counter for each category in *CATEGORIES*
. When an
EVENT
is recorded by TRY
and its type matches *COUNT*
, the counters
of all categories matching the event type are incremented in the
CURRENT-TRIAL
. When a trial finishes and a VERDICT
is recorded, the
trial's event counters are added to that of its parent's (if any).
The counts are printed with VERDICT
s (see Printing Events).
If both *COUNT*
and *CATEGORIES*
are unchanged from the their
default values, then only LEAF
events are counted, and we get
separate counters for ABORT*
, UNEXPECTED-FAILURE
,
UNEXPECTED-SUCCESS
, SKIP
, EXPECTED-FAILURE
, and EXPECTED-SUCCESS
.
(let ((*debug* nil))
(with-test (outer)
(with-test (inner)
(is t))
(is t)
(is nil)))
.. OUTER
.. INNER
.. ⋅ (IS T)
.. ⋅ INNER ⋅1
.. ⋅ (IS T)
.. ⊠ (IS NIL)
.. ⊠ OUTER ⊠1 ⋅2
..
==> #<TRIAL (WITH-TEST (OUTER)) UNEXPECTED-FAILURE 0.000s ⊠1 ⋅2>
As the above example shows, EXPECTED-VERDICT-SUCCESS
and
EXPECTED-RESULT-SUCCESS
are both marked with "⋅"
, but only
EXPECTED-RESULT-SUCCESS
is counted due to *COUNT*
being LEAF
.
When an EVENT
is recorded and the type of the EVENT
matches the
COLLECT
type argument of TRY
, then a corresponding object is pushed
onto CHILDREN
of the CURRENT-TRIAL
for subsequent Rerunning Trials or
Reprocessing Trials.
In particular, if the matching event is a LEAF
, then the event
itself is collected. If the matching event is a TRIAL-EVENT
, then
its TRIAL
is collected. Furthermore, trials
which collected anything are always collected by their parent.
By default, both implicit and explicit calls to TRY
collect the
UNEXPECTED
(see *COLLECT*
and *TRY-COLLECT*
), and consequently all
the enclosing trials.
-
[reader] CHILDREN TRIAL (:CHILDREN = NIL)
A list of immediate child
VERDICT
s,RESULT
s, andERROR*
s collected in reverse chronological order (see Collecting Events). TheVERDICT
of thisTRIAL
is not amongCHILDREN
, but theVERDICT
s of child trials' are.
When a TRIAL
is FUNCALL
ed or passed to TRY
, the test that
created the trial is invoked, and it may be run again in its
entirety or in part. As the test runs, it may invoke other tests.
Any test (including the top-level one) is skipped if it does not
correspond to a collected trial or its TRIAL-START
event and VERDICT
do not match the RERUN
argument of TRY
. When that
happens, the corresponding function call immediately returns the
TRIAL
object.
-
A new trial is skipped (as if with
SKIP-TRIAL
) ifRERUN
is notT
and<