/
report.clj
412 lines (334 loc) · 15 KB
/
report.clj
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
(ns kaocha.report
"Reporters generate textual output during a test run, providing real-time
information on test progress, failures, errors, and so forth. They are in
nature imperative and side-effectful, they generate output on an output
stream (typically stdout), based on test events. Some reporters are also used
to track state. This is unfortunate as it goes against Kaocha's functional
design, but since we want test runs to be interruptible it is somewhat
inevitable.
The concept of reporters is directly taken from clojure.test, but is used in
Kaocha also when running other types of tests.
A reporter is a function which takes a single argument, a map. The map will
have a `:type` key indicating the type of event, e.g. `:begin-test-var`,
`:fail`, `:pass`, or `:summary`.
Reporters as imagined in `clojure.test` are a flawed design, we try to make
the best of it. See also the monkeypatching of `clojure.test/do-test` in
`kaocha.monkey-patch`, which is necessary to be able to intercept failures
quickly in case the users runs with `--fail-fast` enabled. The patch also
ensures that the current testable is always available in the event map under
`:kaocha/testable`,
Kaocha differs from stock `clojure.test` in that multiple reporters can be
active at the same time. On the command line you can specify `--reporter`
multiple times, in `tests.edn` you can pass a vector to `:kaocha/reporter`,
and/or point at a var which itself defines a vector of functions. Each of the
given functions will be called in turn for each event generated.
This has allowed Kaocha to split the functionality of reporters up, making
them more modular. E.g. `kaocha.report/report-counters` only keeps the
fail/error/pass/test counters, without concerning itself with output, making
it reusable.
This namespace implements the reporters provided by Kaocha out of the box that
don't need extra dependencies. Others like e.g. the progress bar are in their
own namespace to prevent loading files we don't need, and thus slowing down
startup.
### Issues with clojure.test reporters
`clojure.test` provides reporters as a way to extend the library. By default
`clojure.test/report` is a multimethod which dispatches on `:type`, and so
libraries can extend this multimethod to add support for their own event
types. A good example is the `:mismatch` event generated by
matcher-combinators.
Tools can also rebind `clojure.test/report`, and use it as an interface for
capturing test run information.
The problem is that these two approaches don't mesh. When tools (like Kaocha,
CIDER, Cursive, etc.) rebind `clojure.test/report`, then any custom extensions
to the multimethod disappear.
This can also cause troubles when a library which extends
`clojure.test/report` gets loaded after it has been rebound. This was an issue
for a while in test.check, which assumed `report` would always be a
multimethod (this has been rectified). For this reasons Kaocha only rebinds
`report` *after* the \"load\" step.
Kaocha tries to work around these issues to some extent by forwarding any keys
it does not know about to the original `clojure.test/report` multimethod. This
isn't ideal, as these extensions are not aware of Kaocha's formatting and
output handling, but it does provide some level of compatiblity with third
party libraries.
For popular libraries we will include reporter implementations that handle
these events in a way that makes sense within Kaocha, see e.g.
`kaocha.matcher-combinators`. Alternatively library authors can
themselves strive for Kaocha compatiblity, we try to give them the tools to
enable this, through keyword derivation and custom multimethods.
### Custom event types
`kaocha.report` makes use of Clojure's keyword hierarchy feature to determine
the type of test events. To make Kaocha aware of your custom event, first add
a derivation from `:kaocha/known-type`, this will stop the event from being
propagated to the original `clojure.test/report`
``` clojure
(kaocha.hierarchy/derive! :mismatch :kaocha/known-key)
```
If the event signals an error or failure which causes the test to fail, then
derive from `:kaocha/fail-type`. This will make Kaocha's existing reporters
compatible with your custom event.
``` clojure
(kaocha.hierarchy/derive! :mismatch :kaocha/fail-type)
```
"
(:require [kaocha.core-ext :refer :all]
[kaocha.output :as output]
[kaocha.plugin.capture-output :as capture]
[kaocha.stacktrace :as stacktrace]
[kaocha.testable :as testable]
[clojure.test :as t]
[slingshot.slingshot :refer [throw+]]
[clojure.string :as str]
[kaocha.history :as history]
[kaocha.testable :as testable]
[kaocha.hierarchy :as hierarchy]
[kaocha.jit :refer [jit]]))
(defonce clojure-test-report t/report)
(defn dispatch-extra-keys
"Call the original clojure.test/report multimethod when dispatching an unknown
key. This is to support libraries like nubank/matcher-combinators that extend
clojure.test/assert-expr, as well as clojure.test/report, to signal special
conditions."
[m]
(when (and (not (hierarchy/known-key? m))
(not= (get-method clojure-test-report :default)
(get-method clojure-test-report (:type m))))
(clojure-test-report m)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defmulti dots* :type :hierarchy #'hierarchy/hierarchy)
(defmethod dots* :default [_])
(defmethod dots* :pass [_]
(t/with-test-out
(print ".")
(flush)))
(defmethod dots* :kaocha/fail-type [_]
(t/with-test-out
(print (output/colored :red "F"))
(flush)))
(defmethod dots* :error [_]
(t/with-test-out
(print (output/colored :red "E"))
(flush)))
(defmethod dots* :kaocha/pending [_]
(t/with-test-out
(print (output/colored :yellow "P"))
(flush)))
(defmethod dots* :kaocha/begin-group [_]
(t/with-test-out
(print "(")
(flush)))
(defmethod dots* :kaocha/end-group [_]
(t/with-test-out
(print ")")
(flush)))
(defmethod dots* :begin-test-suite [_]
(t/with-test-out
(print "[")
(flush)))
(defmethod dots* :end-test-suite [_]
(t/with-test-out
(print "]")
(flush)))
(defmethod dots* :summary [_]
(t/with-test-out
(println)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defmulti report-counters :type :hierarchy #'hierarchy/hierarchy)
(defmethod report-counters :default [_])
(defmethod report-counters :pass [m]
(t/inc-report-counter :pass))
(defmethod report-counters :kaocha/fail-type [m]
(t/inc-report-counter :fail))
(defmethod report-counters :error [m]
(t/inc-report-counter :error))
(defmethod report-counters :kaocha/pending [m]
(t/inc-report-counter :pending))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defmulti result :type :hierarchy #'hierarchy/hierarchy)
(defmethod result :default [_])
(defn testing-vars-str
"Returns a string representation of the current test. Renders names
in :testing-vars as a list, then the source file and line of current
assertion."
[{:keys [file line testing-vars kaocha/testable] :as m}]
(let [file' (or file (some-> testable ::testable/meta :file))
line' (or line (some-> testable ::testable/meta :line))]
(str
;; Uncomment to include namespace in failure report:
;;(ns-name (:ns (meta (first *testing-vars*)))) "/ "
(or (some-> (:kaocha.testable/id testable) str (subs 1))
(and (seq testing-vars)
(reverse (map #(:name (meta %)) testing-vars))))
" (" file ":" line ")")))
(defn print-output [m]
(let [output (get-in m [:kaocha/testable ::capture/output])
buffer (get-in m [:kaocha/testable ::capture/buffer])
out (or output (and buffer (capture/read-buffer buffer)))]
(when (seq out)
(println "╭───── Test output ───────────────────────────────────────────────────────")
(println (str/replace (str/trim-newline out)
#"(?m)^" "│ "))
(println "╰─────────────────────────────────────────────────────────────────────────"))))
(defn assertion-type
"Given a clojure.test event, return the first symbol in the expression inside (is)."
[m]
(if-let [s (and (seq? (:expected m)) (seq (:expected m)))]
(first s)
:default))
(defmulti print-expr
assertion-type
:hierarchy #'hierarchy/hierarchy)
(defmethod print-expr :default [m]
(when (contains? m :expected)
(println "expected:" (pr-str (:expected m))))
(when (contains? m :actual)
(println " actual:" (pr-str (:actual m)))))
(defmethod t/assert-expr '= [msg form]
(if (= 2 (count form))
`(t/do-report {:type ::one-arg-eql
:message "Equality assertion expects 2 or more values to compare, but only 1 arguments given."
:expected '~(concat form '(arg2))
:actual '~form})
(t/assert-predicate msg form)))
(hierarchy/derive! ::one-arg-eql :kaocha/fail-type)
(defmethod print-expr '= [m]
(let [printer (output/printer)]
;; :actual is of the form (not (= ...))
(if (and (not= (:type m) ::one-arg-eql)
(seq? (second (:actual m)))
(> (count (second (:actual m))) 2))
(let [[_ expected & actuals] (-> m :actual second)]
(output/print-doc
[:span
"Expected:" :line
[:nest (output/format-doc expected printer)]
:break
"Actual:" :line
(into [:nest]
(interpose :break)
(for [actual actuals]
(output/format-doc ((jit lambdaisland.deep-diff/diff) expected actual)
printer)))]))
(output/print-doc
[:span
"Expected:" :line
[:nest (output/format-doc (:expected m) printer)]
:break
"Actual:" :line
[:nest (output/format-doc (:actual m) printer)]]))))
(defmulti fail-summary :type :hierarchy #'hierarchy/hierarchy)
(defmethod fail-summary :kaocha/fail-type [{:keys [testing-contexts testing-vars] :as m}]
(println (str "\n" (output/colored :red "FAIL") " in") (testing-vars-str m))
(when (seq testing-contexts)
(println (str/join " " (reverse testing-contexts))))
(when-let [message (:message m)]
(println message))
(print-expr m)
(print-output m))
(defmethod fail-summary :error [{:keys [testing-contexts testing-vars] :as m}]
(println (str "\n" (output/colored :red "ERROR") " in") (testing-vars-str m))
(when (seq testing-contexts)
(println (str/join " " (reverse testing-contexts))))
(when-let [message (:message m)]
(println message))
(print-output m)
(print "Exception: ")
(let [actual (:actual m)]
(if (throwable? actual)
(stacktrace/print-cause-trace actual t/*stack-trace-depth*)
(prn actual))))
(defmethod result :summary [m]
(t/with-test-out
(let [failures (filter hierarchy/fail-type? @history/*history*)]
(doseq [{:keys [testing-contexts testing-vars] :as m} failures]
(binding [t/*testing-contexts* testing-contexts
t/*testing-vars* testing-vars]
(fail-summary m))))
(doseq [deferred (filter hierarchy/deferred? @history/*history*)]
(clojure-test-report deferred))
(let [{:keys [test pass fail error pending] :or {pass 0 fail 0 error 0 pending 0}} m
failed? (pos-int? (+ fail error))
pending? (pos-int? pending)]
(println (output/colored (if failed? :red (if pending? :yellow :green))
(str test " tests, "
(+ pass fail error) " assertions, "
(when (pos-int? error)
(str error " errors, "))
(when pending?
(str pending " pending, "))
fail " failures."))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn fail-fast
"Fail fast reporter, add this as a final reporter to interrupt testing as soon
as a failure or error is encountered."
[m]
(when (and testable/*fail-fast?*
(hierarchy/fail-type? m)
(not (:kaocha.result/exception m))) ;; prevent handled exceptions from being re-thrown
(throw+ {:kaocha/fail-fast true})))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def doc-printed-contexts (atom nil))
(defn doc-print-contexts [contexts & [suffix]]
(let [printed-contexts @doc-printed-contexts]
(let [contexts (reverse contexts)
printed (reverse printed-contexts)
pairwise (map vector (concat printed (repeat nil)) contexts)
nesting (->> pairwise (take-while (fn [[x y]] (= x y))) count)
new-contexts (->> pairwise (drop-while (fn [[x y]] (= x y))) (map last))]
(when (seq new-contexts)
(doseq [[ctx idx] (map vector new-contexts (range))
:let [nesting (+ nesting idx)]]
(print (str "\n"
" "
(apply str (repeat nesting " "))
ctx))
(flush))))
(reset! doc-printed-contexts contexts)))
(defmulti doc :type :hierarchy #'hierarchy/hierarchy)
(defmethod doc :default [_])
(defmethod doc :begin-test-suite [m]
(t/with-test-out
(reset! doc-printed-contexts (list))
(print "---" (-> m :kaocha/testable :kaocha.testable/desc) "---------------------------")
(flush)))
(defmethod doc :kaocha/begin-group [m]
(t/with-test-out
(reset! doc-printed-contexts (list))
(print (str "\n" (-> m
:kaocha/testable
:kaocha.testable/desc)))
(flush)))
(defmethod doc :kaocha/end-group [m]
(t/with-test-out
(println)))
(defmethod doc :kaocha/begin-test [m]
(t/with-test-out
(let [desc (or (some-> m :kaocha/testable :kaocha.testable/desc)
(some-> m :var meta :name))]
(print (str "\n " desc))
(flush))))
(defmethod doc :pass [m]
(t/with-test-out
(doc-print-contexts t/*testing-contexts*)))
(defmethod doc :error [m]
(t/with-test-out
(doc-print-contexts t/*testing-contexts*)
(print (output/colored :red " ERROR"))))
(defmethod doc :kaocha/fail-type [m]
(t/with-test-out
(doc-print-contexts t/*testing-contexts*)
(print (output/colored :red " FAIL"))))
(defmethod doc :summary [m]
(t/with-test-out
(println)))
(defn debug [m]
(t/with-test-out
(prn (cond-> (select-keys m [:type :file :line :var :ns :expected :actual :message :kaocha/testable :debug])
(:kaocha/testable m)
(update :kaocha/testable select-keys [:kaocha.testable/id :kaocha.testable/type])))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def dots
"Reporter that prints progress as a sequence of dots and letters."
[dots* result])
(def documentation
[doc result])