Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 1403 lines (1203 sloc) 56.377 kb
5be7639 Anton Vodonosov first commit
avodonosov authored
1 (defpackage #:test-grid (:use :cl))
2
3 (in-package #:test-grid)
4
5 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
6
7 #|
8
9 TODO:
10 - information about test run:
11 + lisp-version-string,
12 + lib-world,
8d47120 d35h fixed text formatting
d35h authored
13 + author contact (get it from some settings file),
5be7639 Anton Vodonosov first commit
avodonosov authored
14 + date,
15 + run-duration
16 + organize database file format
17 + better decision for library name representation.
18 This representation is used in the:
19 - libtest method parameter (eql specialized)
20 - in the database and testrun datastructure.
21 Possible alternatives:
22 a keyword
23 good for READ (package independent),
24 good for EQL specializer
25 good for GETF when working with the database
26 a symbol from the test-grid package
27 - good for EQL specializer
28 - package dependent in READ
29 - good for GETF when working with the database
30 - adds one more (unnecessary) way to represent a library
31 in addition the specified for ASDF and Quicklisp
32 or a downcased string
33 - needs special handling in libtest eql specialization
34 - good ro READ (package independent)
35 - needs care to work with GETF when working with the database
06cb006 Anton Vodonosov fixed some typos in the TODO comment
avodonosov authored
36 + Generate a sufficiently large database so that we can
5be7639 Anton Vodonosov first commit
avodonosov authored
37 evaluate our reporting solution. We may generate
06cb006 Anton Vodonosov fixed some typos in the TODO comment
avodonosov authored
38 fake test results programmatically to make this task
5be7639 Anton Vodonosov first commit
avodonosov authored
39 easier.
40 - simpliest reporting to allow overview of library test statuses
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
41 - Test Runs report: every test run as a row in a table
5be7639 Anton Vodonosov first commit
avodonosov authored
42 + legend or a tooltip in the report for test statuses
43 + color for statuses
ec59872 Anton Vodonosov TODO comment about finalizing the terminology about our main data object...
avodonosov authored
44 + use the online blob URL in the report
fa1c4e1 Anton Vodonosov Pivot reports. First working version; no comments, data cell is rendered...
avodonosov authored
45 - A pivot -like table report of library test results, allowing
95e993b Anton Vodonosov updated the main TODO comment.
avodonosov authored
46 rows/columns to be any of quicklisp distro, lisp version
47 library name. With grouping and sorging.
3a160d3 Anton Vodonosov Better (most likely final) formatting for the test-reports-overview.html
avodonosov authored
48 - CSV export of the database to use it then with spreadsheets,
95e993b Anton Vodonosov updated the main TODO comment.
avodonosov authored
49 google fusion tables, etc. Initial intent
50 was to format it as a pivot for various projections
51 (by quicklisp releases, by platform, etc).
52 But neither google docs spreadsheet, nor google fusion
53 table allow as to format results as we want
54 (the main problem, it is impossible to use
55 a custom aggregation function for pivot
56 cells, because standard aggregation functions
57 are numeric, but we want a report cell
58 to represent test result(s) for a particular
59 library, i.e. :ok, :fail, :no-resource).
5be7639 Anton Vodonosov first commit
avodonosov authored
60 5h
3a160d3 Anton Vodonosov Better (most likely final) formatting for the test-reports-overview.html
avodonosov authored
61 - Test that the test-duration field value
62 (Common Lisp rational) can be read
63 by spreadsheet software (MS/Open Offices,
64 Google Spreadsheets).
06cb006 Anton Vodonosov fixed some typos in the TODO comment
avodonosov authored
65 - an informer which may be embedded into a library
66 project page, with reports about the test statuses
5be7639 Anton Vodonosov first commit
avodonosov authored
67 for this single library on various platforms with
68 various quicklisp versions
3a160d3 Anton Vodonosov Better (most likely final) formatting for the test-reports-overview.html
avodonosov authored
69 - an overview page with brief explanation of all
70 the above reports and links to the reports.
71 - change
72 "represents every test run as a separate row"
73 to
74 "represents every <tt>test-grid:run-tests</tt> as a separate row"
75 (after user will know this command from the main project description)
76 ?
77 - Description of CSV report may link to an example
78 of the CSV report imported to a Google Spreadsheet
79 with pivot calculating avearage duration of
80 tests for every library.
81 - spell check
5be7639 Anton Vodonosov first commit
avodonosov authored
82 - simple UI (command line) with guiding messages
83 for the user who runs the tests. Spend as little
84 efforts as possible on this task, to release quickly.
85 4h
86 - readme with explanation of the project goal and
87 how to use it
88 5h
8d47120 d35h fixed text formatting
d35h authored
89 + change db format
c023819 Anton Vodonosov test run as plist (:descr <descr> :run-results <run-results>) instead of...
avodonosov authored
90 + test run as plist (:descr <descr> :run-results <run-results>)
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
91 instead of just (<descr> <run-results>)
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
92 + run-results as a list instead of plist; libname
93 which was a plist key is now a property of the lib-result
94 object. It is more convenient for standard mapping functions,
95 instead of current do-lib-results.
88734ae Anton Vodonosov updated the TODO comment
avodonosov authored
96 + add more libraries: total number of 20 libraries
97 is enough for the beginning.
98 Result: we have 23 libraries.
8d47120 d35h fixed text formatting
d35h authored
99 + when loading of a library or library test system
5be7639 Anton Vodonosov first commit
avodonosov authored
100 fails, ensure we have the error description in the output
101 0.5h
8d47120 d35h fixed text formatting
d35h authored
102 + The "thank you" message: where exactly to submit test results?
5be7639 Anton Vodonosov first commit
avodonosov authored
103 Specify an email or issue tracker of the cl-test-grid project.
8d47120 d35h fixed text formatting
d35h authored
104 + how to store public (central) database and failed library
5be7639 Anton Vodonosov first commit
avodonosov authored
105 outputs (files).
106 An appealing way is to store it in the same git repository
107 on github, but with the std-out files the repository will
108 quickly grow to an unconvenient size (for new people the
109 checkout procedure will be too long to be considered
110 convenient)
111 5h
8a2e2a0 Anton Vodonosov update the TODO comment.
avodonosov authored
112 Solution: files are stored in Google App Engine blob store.
5f523a0 Anton Vodonosov added metatilities-base cl-cont
avodonosov authored
113 - More detailed output for libraries using the RT test
114 framework. Ensure the libs with other test framework
115 are all have sufficiently detailed output too.
88734ae Anton Vodonosov updated the TODO comment
avodonosov authored
116 - Some libraries stil print messages to console,
117 meaning their output is not only *standard-output*
118 and *standard-error*. Fix that, all the output
119 should be in the log files, but not on console.
120 Examples of such libraries: trivieal-backtrace,
121 trival-timeout, metatilities (i.e. the ones
122 using the Lift test framework).
123 - On Windows, SBCL and CCL seem to have different
124 results of (user-homedir-pathname); in result
125 CCL doesn't see the settings file created by SBCL.
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
126 - run the tests on all the implementations available for us.
4055e27 Anton Vodonosov Added cl-base64 to the tests.
avodonosov authored
127 - usocket test suite might need manual configuration,
128 see their README. Distinguish the case
129 when the manual configuration hasn't been
130 performed and return :no-resource status.
131 - For all the libraries which need manual configuration
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
132 (cffi, usocket) provide guiding message to the
4055e27 Anton Vodonosov Added cl-base64 to the tests.
avodonosov authored
133 user how to configure them.
3a160d3 Anton Vodonosov Better (most likely final) formatting for the test-reports-overview.html
avodonosov authored
134 - finalize the decision what command user runs
135 to performs the tests. Describe this main command
136 in the README (in the first paragraph).
4055e27 Anton Vodonosov Added cl-base64 to the tests.
avodonosov authored
137
5be7639 Anton Vodonosov first commit
avodonosov authored
138 ==================================================
139 ========== Milestone: release 0 ============
140 ==================================================
ec59872 Anton Vodonosov TODO comment about finalizing the terminology about our main data object...
avodonosov authored
141 - finalize the terminology we use in the code
142 to refer our main data:
143 - test status for a particular library
144 - library test result object (includes the status
145 as well as log length, the key of the log
146 in the online blob store, probably the
147 library test duration)
148 - list of library test results in a particular test
149 run
150 - test run description, consists of lisp name,
151 libraries set (think quicklisp distro),
152 the user contacts, total test run duration,
153 etc.
7ab792d Anton Vodonosov Added trivial-backtrace to the tests. (Always fails; it tries to create ...
avodonosov authored
154 - watchdog for hanging tests
5be7639 Anton Vodonosov first commit
avodonosov authored
155 + more abstract accessor to parts of DB info instead of
156 getf by properties: run-descr, run-results.
157 1h
158 + safe-read database
159 + create a project with asdf system
160 0.5h
161 + DB file path based on the asdf system location
162 0.5h
163 + accumulate failed library output
164 1h
165 - DB file formatting should be equal in all lisps,
166 so that diff shows only new records.
167 (use pprint ?)
168 4h
169 - a way to specify lib-wold as a quicklisp version with some
170 library versions overriden (checkout this particular
171 libraries from the scm), so that library author can quickly
172 get test result for his changes (fixes) in scm.
173 An implementation idea to consider: almost every scm allows
174 to download asnapshot via http, so the quicklisp http machinery may
175 be reused here, whithout running a shell command for
176 checkout.
177 24h
178 - should we save library log to a file only if the tests failed,
179 or always? (now we save log in any case)
180 - During run-libtests, probably we should redirect the library
181 output to file directly, without caching it in memory
182 - it is more convenient when you are watching the testing
183 process, you can observe the file being populated with
184 logs (because some libraries, like flexi-streams, take
185 time about minute to finish, and if during this minute
186 nithing happens it is not user-friendly)
187 |#
188
189 (defgeneric libtest (library-name)
190 (:documentation "Define a method for this function
b279089 Anton Vodonosov Fixed return value of the run-testlibs - if logs upload to the blobstore...
avodonosov authored
191 with LIBRARY-NAME eql-specialized for for every library added
5be7639 Anton Vodonosov first commit
avodonosov authored
192 to the test grid.
193
194 The method should run test suite and return the resulting
195 status. Status is one of three values:
196 :OK - all tests passed,
197 :FAIL - some test failed,
198 :NO-RESOURCE - test suite can not be run because some required
199 resource is absent in the environment. For example, CFFI library
200 test suite needs a small C library compiled to DLL. User must
201 do it manually. In case the DLL is absent, the LIBTEST method
202 for CFFI returns :NO-RESOURCE.
203
204 For convenience, T may be returned instead of :OK and NIL instead of :FAIL."))
205
206 (defun normalize-status (status)
207 "Normilzies test resul status - converts T to :OK and NIL to :FAIL."
208 (case status
209 ((t :ok) :ok)
210 ((nil :fail) :fail)
211 (otherwise status)))
212
4055e27 Anton Vodonosov Added cl-base64 to the tests.
avodonosov authored
213 (defparameter *all-libs* '(:alexandria :babel :trivial-features :cffi
214 :cl-ppcre :usocket :flexi-streams :bordeaux-threads
44b1c75 Anton Vodonosov Added parenscript
avodonosov authored
215 :cl-base64 :trivial-backtrace :puri :anaphora
7b69c8d Anton Vodonosov cl-json added to *all-libs*
avodonosov authored
216 :parenscript :trivial-garbage :iterate :metabang-bind
efd325f Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
217 :cl-json :cl-containers :metatilities-base :cl-cont
218 :moptilities :trivial-timeout :metatilities)
5be7639 Anton Vodonosov first commit
avodonosov authored
219 "All the libraries currently supported by the test-grid.")
220
221 (defun clean-rt ()
222 "Helper function to assist running test suites created using the RT
223 test framework. The problem is that RT uses global storage for all
224 the tests; in result if we previously loaded any test system,
225 after loading another test system the global test RT test suite
226 contains the tests of _both_ libraries."
227 (let ((rem-all-tests (and (find-package '#:rt)
228 (find-symbol (symbol-name '#:rem-all-tests) '#:rt))))
229 (when rem-all-tests (funcall rem-all-tests))))
230
231 (defmethod libtest ((library-name (eql :alexandria)))
232
233 ;; The test framework used: rt.
234 (clean-rt)
235 (asdf:clear-system :alexandria-tests)
236
237 (quicklisp:quickload :alexandria-tests)
238
239 (flet (
240 ;; the run-tests local function is copy/pasted
241 ;; from alexandria-tests.asd
242 (run-tests (&rest args)
243 (apply (intern (string '#:run-tests) '#:alexandria-tests) args)))
244
245 (let ((a (run-tests :compiled nil))
246 (b (run-tests :compiled t)))
247 (and a b))))
248
249 (defmethod libtest ((library-name (eql :babel)))
250
251 ;; The test framework used: stefil.
252
253 (quicklisp:quickload :babel-tests)
254
255 (let ((result (funcall (intern (string '#:run) '#:babel-tests))))
256 (zerop
257 (length (funcall (intern (string '#:failure-descriptions-of) '#:hu.dwim.stefil)
258 result)))))
259
260 (defmethod libtest ((library-name (eql :trivial-features)))
261
262 ;; The test framework used: rt.
263 (clean-rt)
264 (asdf:clear-system :trivial-features-tests)
265
266 (quicklisp:quickload :trivial-features-tests)
267
268 ;; copy/past from trivial-features-tests.asd
269 (let ((*package* (find-package 'trivial-features-tests)))
270 (funcall (find-symbol (symbol-name '#:do-tests)))))
271
272 (defmethod libtest ((library-name (eql :cffi)))
273
274 ;; The test framework used: rt.
275 (clean-rt)
276 (asdf:clear-system :cffi-tests)
277
278 (handler-case (quicklisp:quickload :cffi-tests)
279 ;; CFFI tests work with a small test C
280 ;; library. The user is expected to compile
281 ;; the library. If the library is not available,
282 ;; CFFI tests signal cffi:load-foreign-library-error.
283 (t (e)
284 (when (eq (type-of e)
285 (find-symbol (symbol-name '#:load-foreign-library-error) '#:cffi))
286 (return-from libtest :no-resource))))
287
288 (flet (
289 ;; copy/paste from cffi-tests.asd
290 (run-tests (&rest args)
291 (apply (intern (string '#:run-cffi-tests) '#:cffi-tests) args)))
292
293 (let ((a (run-tests :compiled nil))
294 (b (run-tests :compiled t)))
295 (and a b))))
296
297 (defmethod libtest ((library-name (eql :cl-ppcre)))
298
299 ;; The test framework used: custom.
300
301 ;; Workaround the quicklisp issue #225 -
302 ;; https://github.com/quicklisp/quicklisp-projects/issues/225 -
303 ;; first load cl-ppcre-unicode, because otherwise
304 ;; current quicklisp can not find cl-ppcre-unicode-test
305 (quicklisp:quickload :cl-ppcre-unicode)
306 (quicklisp:quickload :cl-ppcre-unicode-test)
307
308 ;; copy/paste from cl-ppcre-unicode.asd
309 (funcall (intern (symbol-name :run-all-tests) (find-package :cl-ppcre-test))
310 :more-tests (intern (symbol-name :unicode-test) (find-package :cl-ppcre-test))))
311
312 (defmethod libtest ((library-name (eql :usocket)))
313
314 ;; The test framework used: rt.
315 (clean-rt)
316 (asdf:clear-system :usocket-test)
317
318 ; (asdf:operate 'asdf:load-op :usocket-test :force t)
319
320 (quicklisp:quickload :usocket-test)
321
322 ;; TODO: usocket test suite might need manual configuration,
323 ;; see their README. Distinguish the case
324 ;; when the manual configuration hasn't been
325 ;; performed and return :no-resource status.
4055e27 Anton Vodonosov Added cl-base64 to the tests.
avodonosov authored
326 ;;
5be7639 Anton Vodonosov first commit
avodonosov authored
327 ;; (setf usocket-test::*common-lisp-net*
328 ;; (or usocket-test::*common-lisp-net*
329 ;; "74.115.254.14"))
330
331 ;; copy/paste from usocket-test.asd
332 (funcall (intern "DO-TESTS" "USOCKET-TEST")))
333
334
335 (defmethod libtest ((library-name (eql :flexi-streams)))
336
337 ;; The test framework used: custom.
338
339 (quicklisp:quickload :flexi-streams-test)
340
341 ;; copy/paste from flexi-streams.asd
342 (funcall (intern (symbol-name :run-all-tests)
343 (find-package :flexi-streams-test))))
344
d4fc132 Anton Vodonosov added cl-json
avodonosov authored
345 (defun run-fiveam-suite (fiveam-test-spec)
346 "Runs the specified test suite created with the FiveAM
347 test framework. Returns T if all the tests succeeded and
348 NIL otherwise. The FIVEAM-TEST-SPEC specifies the tests
349 suite according to the FiveAM convention."
350 (let ((run (intern (string '#:run) :fiveam))
351 (explain (intern (string '#:explain) :fiveam))
352 (detailed-text-explainer (intern (string '#:detailed-text-explainer) :fiveam))
353 (test-failure-type (intern (string '#:test-failure) :fiveam)))
354
355 (let ((results (funcall run fiveam-test-spec)))
356 (funcall explain (make-instance detailed-text-explainer) results *standard-output*)
357 (zerop (count-if (lambda (res)
358 (typep res test-failure-type))
359 results)))))
360
5be7639 Anton Vodonosov first commit
avodonosov authored
361 (defmethod libtest ((library-name (eql :bordeaux-threads)))
362
363 ;; The test framework used: fiveam.
364
365 (quicklisp:quickload :bordeaux-threads-test)
d4fc132 Anton Vodonosov added cl-json
avodonosov authored
366
367 (run-fiveam-suite :bordeaux-threads))
5be7639 Anton Vodonosov first commit
avodonosov authored
368
4055e27 Anton Vodonosov Added cl-base64 to the tests.
avodonosov authored
369 (defmethod libtest ((library-name (eql :cl-base64)))
370
371 ;; The test framework used: ptester.
5be7639 Anton Vodonosov first commit
avodonosov authored
372
4055e27 Anton Vodonosov Added cl-base64 to the tests.
avodonosov authored
373 (quicklisp:quickload :cl-base64-tests)
374
375 (funcall (intern (symbol-name '#:do-tests)
376 (find-package '#:cl-base64-tests))))
377
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
378 (defun lift-tests-ok-p (lift-tests-result)
379 "Helper function to work with Lift test framework.
380 Examines the tests result object and retuns T is all
381 the tests are successull and NIL otherwise."
382 (let ((errors (intern (symbol-name '#:errors) :lift))
383 (expected-errors (intern (symbol-name '#:expected-errors) :lift))
384 (failures (intern (symbol-name '#:failures) :lift))
385 (expected-failures (intern (symbol-name '#:expected-failures) :lift)))
386 (zerop
387 (+ (length (set-difference (funcall errors lift-tests-result)
388 (funcall expected-errors lift-tests-result)))
389 (length (set-difference (funcall failures lift-tests-result)
390 (funcall expected-failures lift-tests-result)))))))
391
392 (defun run-lift-tests (suite-name)
393 "Helper function to work with the Lift test framework.
394 Runs the specified Lift test suite and returns T
395 if all the tests succeeded and NIL othersize."
396 (let ((result (funcall (intern (symbol-name '#:run-tests) :lift)
397 :suite suite-name)))
398 (describe result *standard-output*)
399 (lift-tests-ok-p result)))
400
7ab792d Anton Vodonosov Added trivial-backtrace to the tests. (Always fails; it tries to create ...
avodonosov authored
401 (defmethod libtest ((library-name (eql :trivial-backtrace)))
402
403 ;; The test framework used: lift.
404
405 (quicklisp:quickload :trivial-backtrace-test)
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
406
407 (run-lift-tests :trivial-backtrace-test))
7ab792d Anton Vodonosov Added trivial-backtrace to the tests. (Always fails; it tries to create ...
avodonosov authored
408
1639b53 Anton Vodonosov Added puri. A typo fixed in the LIBTEST for trivial-backtrace. Decided t...
avodonosov authored
409 (defmethod libtest ((library-name (eql :puri)))
410
411 ;; The test framework used: ptester.
412
413 (quicklisp:quickload :puri-tests)
414
415 ;; copy/paste from puri.asd
416 (funcall (intern (symbol-name '#:do-tests)
417 (find-package :puri-tests))))
418
1d6be7b Anton Vodonosov Added anaphora
avodonosov authored
419 (defmethod libtest ((library-name (eql :anaphora)))
420
421 ;; The test framework used: rt.
422 (clean-rt)
423 (asdf:clear-system :anaphora-test)
424
425 (quicklisp:quickload :anaphora-test)
426
427 ;; copy/paste from anaphora.asd
428 (funcall (intern "DO-TESTS" :rt)))
429
44b1c75 Anton Vodonosov Added parenscript
avodonosov authored
430 (defmethod libtest ((library-name (eql :parenscript)))
431 ;; The test framework used: eos (similar to FiveAM).
432
433 ;; asdf:test-op is not provided for parenscript,
434 ;; only a separate package ps-test with public
435 ;; function run-tests.
436
437 ;; The test suites to run determined by looking
438 ;; into the function run-tests in the file test.lisp.
439 (let* ((run (intern (string '#:run) :eos))
440 (test-failure-type (intern (string '#:test-failure) :eos))
441 (results (append (funcall run (intern (string '#:output-tests) :ps-test))
442 (funcall run (intern (string '#:package-system-tests) :ps-test)))))
5be7639 Anton Vodonosov first commit
avodonosov authored
443 (zerop (count-if (lambda (res)
444 (typep res test-failure-type))
445 results))))
446
d7c17f3 Anton Vodonosov Added trivial-garbage.
avodonosov authored
447 (defmethod libtest ((library-name (eql :trivial-garbage)))
448
449 ;; The test framework used: rt.
450 (clean-rt)
451 (asdf:clear-system :trivial-garbage) ; yes, trivial-garbage but not trivial-garbage-tests,
452 ; because the trivial-garbage-tests system is defined
453 ; in the same trivial-garbage.asd and neither
454 ; asdf nor quicklisp can't find trivial-garbage-tests.
455
456 (quicklisp:quickload :trivial-garbage); trivial-garbage but not trivial-garbage-tests,
457 ; for the same reasons as explained above.
458 (asdf:operate 'asdf:load-op :trivial-garbage-tests)
459
460 (funcall (find-symbol (string '#:do-tests) '#:rtest)))
1d6be7b Anton Vodonosov Added anaphora
avodonosov authored
461
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
462 (defmethod libtest ((library-name (eql :iterate)))
463
464 ;; The test framework used: rt.
465 (clean-rt)
466 (asdf:clear-system :iterate-tests)
467 (asdf:clear-system :iterate)
468
469 (quicklisp:quickload :iterate-tests)
470
471 (funcall (intern "DO-TESTS" (find-package #+sbcl "SB-RT"
472 #-sbcl "REGRESSION-TEST"))))
473
474 (defmethod libtest ((library-name (eql :metabang-bind)))
475
476 ;; The test framework used: lift.
477
d4fc132 Anton Vodonosov added cl-json
avodonosov authored
478 ;; metabang-bind-test includes binding syntax
479 ;; for regular-expression and corresponding
480 ;; tests; but this functionality is only
481 ;; loaded if cl-ppcre is loaded first.
482 ;; (this conditional loading is achieaved
483 ;; with asdf-system-connections).
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
484 (quicklisp:quickload :cl-ppcre)
485 (quicklisp:quickload :metabang-bind-test)
486
487 (run-lift-tests :metabang-bind-test))
488
d4fc132 Anton Vodonosov added cl-json
avodonosov authored
489 (defmethod libtest ((library-name (eql :cl-json)))
490 ;; The test framework used: fiveam.
491 (quicklisp:quickload :cl-json.test)
492 (run-fiveam-suite (intern (symbol-name '#:json) :json-test)))
493
0fe5b33 Anton Vodonosov added cl-containers
avodonosov authored
494 (defmethod libtest ((library-name (eql :cl-containers)))
495 ;; The test framework used: lift.
496 (quicklisp:quickload :cl-containers-test)
497 (run-lift-tests :cl-containers-test))
d4fc132 Anton Vodonosov added cl-json
avodonosov authored
498
5f523a0 Anton Vodonosov added metatilities-base cl-cont
avodonosov authored
499 (defmethod libtest ((library-name (eql :metatilities-base)))
500 ;; The test framework used: lift.
501 (quicklisp:quickload :metatilities-base-test)
502 (run-lift-tests :metatilities-base-test))
503
504 (defmethod libtest ((library-name (eql :cl-cont)))
505 ;; The test framework used: rt.
506 (clean-rt)
507 (asdf:clear-system :cl-cont-test)
508
509 (quicklisp:quickload :cl-cont-test)
510 (funcall (intern (string '#:test-cont) :cl-cont-test)))
511
efd325f Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
512 (defmethod libtest ((library-name (eql :moptilities)))
513 ;; The test framework used: lift.
514 (quicklisp:quickload :moptilities-test)
c3e14f2 Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
515 (run-lift-tests :moptilities-test))
efd325f Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
516
517 (defmethod libtest ((library-name (eql :trivial-timeout)))
518 ;; The test framework used: lift.
519 (quicklisp:quickload :trivial-timeout-test)
c3e14f2 Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
520 (run-lift-tests :trivial-timeout-test))
efd325f Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
521
522 (defmethod libtest ((library-name (eql :metatilities)))
523 ;; The test framework used: lift.
524 (quicklisp:quickload :metatilities-test)
b38c686 Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
525 (run-lift-tests :metatilities-test))
efd325f Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
526
5be7639 Anton Vodonosov first commit
avodonosov authored
527 (defun run-libtest (lib)
528 (let* ((orig-std-out *standard-output*)
529 (buf (make-string-output-stream))
530 (*standard-output* buf)
d88a518 d35h finally finished function get-user-email, added new property in result l...
d35h authored
531 (*error-output* buf)
532 (start-time (get-internal-real-time)))
5be7639 Anton Vodonosov first commit
avodonosov authored
533
b279089 Anton Vodonosov Fixed return value of the run-testlibs - if logs upload to the blobstore...
avodonosov authored
534 (format orig-std-out
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
535 "Running tests for ~A. *STANDARD-OUTPUT* and *ERROR-OUTPUT* are redirected.~%"
5be7639 Anton Vodonosov first commit
avodonosov authored
536 lib)
537 (finish-output orig-std-out)
538
539 (let ((status (handler-case (normalize-status (libtest lib))
540 (t () :fail))))
541 (when (eq :fail status)
542 (format t "~A tests failed." lib))
543 (let ((output (get-output-stream-string buf)))
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
544 (list :libname lib
545 :status status :output output
d88a518 d35h finally finished function get-user-email, added new property in result l...
d35h authored
546 :log-char-length (length output)
547 :test-duration (/ (- (get-internal-real-time) start-time)
548 internal-time-units-per-second))))))
5be7639 Anton Vodonosov first commit
avodonosov authored
549
4cea853 Anton Vodonosov Use ql:quickload instead of asdf:operate when loading the GAE blobstore ...
avodonosov authored
550 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
551 ;; Utils
552 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
553 (defun do-plist-impl (plist handler)
554 (do* ((cur-pos plist (cddr cur-pos))
555 (prop (first cur-pos) (first cur-pos))
556 (val (second cur-pos) (second cur-pos)))
557 ((null prop))
558 (funcall handler prop val)))
559
560 (defmacro do-plist ((key val plist &optional result) &body body)
561 `(block nil
562 (do-plist-impl ,plist (lambda (,key ,val) ,@body))
563 ,result))
564
565 (defun plist-comparator (&rest props-and-preds)
566 (lambda (plist-a plist-b)
567 (do-plist (prop pred props-and-preds)
568 ;; iterate over all the property/predicate pairs
569 ;; "compare" the values of the current property
570 ;; in both plists
571 (let ((val-a (getf plist-a prop))
572 (val-b (getf plist-b prop)))
573 (if (funcall pred val-a val-b)
574 (return t))
575 ;; Ok, val-a is not less than val-b (as defined by our predicate).
576 ;; Lets check if they are equal. If the reverse comparation [val-b less val-a]
577 ;; is also false, then they are equal, and we proceed to the next
578 ;; property/predicate pair.
579 (when (funcall pred val-b val-a)
580 (return nil))))))
581
582 ;; examples:
583 #|
584 (let ((less (plist-comparator :a '< :b 'string<)))
585 (and (funcall less '(:a 1 :b "x") '(:a 2 :b "y"))
586 (funcall less '(:a 2 :b "x") '(:a 2 :b "y"))
587 (not (funcall less '(:a 3 :b "x") '(:a 2 :b "y")))))
588
589 (equalp
590 (sort '((:a 1 :b "x")
591 (:a 2 :b "y")
592 (:a 2 :b "y")
593 (:a 3 :b "z"))
594 (plist-comparator :a '< :b 'string<))
595 '((:A 1 :B "x") (:A 2 :B "y") (:A 2 :B "y") (:A 3 :B "z")))
596 |#
597
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
598 (defun getter (prop)
599 #'(lambda (plist)
600 (getf plist prop)))
4cea853 Anton Vodonosov Use ql:quickload instead of asdf:operate when loading the GAE blobstore ...
avodonosov authored
601
fa1c4e1 Anton Vodonosov Pivot reports. First working version; no comments, data cell is rendered...
avodonosov authored
602 (defun list< (predicates l1 l2)
603 "Compares two lists L1 and L2 of equal lenght,
604 using for every pair of elements a corresponding predicate
605 from the PREDICATES list (of the same length). Returns
606 T if L1 is less than (according the PREDICATES) L2.
607 Othersise returns NIL."
608 (if (null predicates)
609 nil
610 (let ((pred (car predicates))
611 (elem1 (car l1))
612 (elem2 (car l2)))
613 (if (funcall pred elem1 elem2)
614 t
615 ;; Ok, elem1 is not less than elem2 (as defined by our predicate).
616 ;; Lets check if they are equal. If the reverse comparation [elem2 less elem1]
617 ;; is also false, then they are equal, and we proceed to the next
618 ;; property/predicate pair.
619 (if (funcall pred elem2 elem1)
620 nil
621 (list< (cdr predicates)
622 (cdr l1)
623 (cdr l2)))))))
624
625 #|
626 Examples:
627
628 (and
629 (list< '(< <) '(1 2) '(2 2))
630 (not (list< '(< <) '(1 2) '(1 2)))
631 (list< '(< <) '(1 2) '(1 3))
632 (not (list< '(string< string<)
633 '("quicklisp-fake-2011-00-02" "ccl-fake-1")
634 '("quicklisp-fake-2011-00-01" "clisp-fake-1"))))
635 |#
636
637 (defun hash-table-keys (hash-table)
638 (let (keys)
639 (maphash #'(lambda (key val)
640 (declare (ignore val))
641 (push key keys))
642 hash-table)
643 keys))
644
4cea853 Anton Vodonosov Use ql:quickload instead of asdf:operate when loading the GAE blobstore ...
avodonosov authored
645 ;; copy/paste from
646 ;; http://www.gigamonkeys.com/book/practical-an-mp3-browser.html
647 (defmacro with-safe-io-syntax (&body body)
648 `(with-standard-io-syntax
649 (let ((*read-eval* nil))
650 ,@body)))
651
652 (defun safe-read (&rest args)
653 (with-safe-io-syntax (apply #'read args)))
654
655 (defun safe-read-file (file)
656 (with-open-file (in file
657 :direction :input
658 :element-type 'character ;'(unsigned-byte 8) + flexi-stream
659 )
660 (safe-read in)))
661
662 ;; copy/paste from
663 ;; http://cl-user.net/asp/-1MB/sdataQ0mpnsnLt7msDQ3YNypX8yBX8yBXnMq=/sdataQu3F$sSHnB==
664 (defun file-string (path)
665 "Sucks up an entire file from PATH into a freshly-allocated string,
666 returning two values: the string and the number of bytes read."
667 (with-open-file (s path)
668 (let* ((len (file-length s))
669 (data (make-string len)))
670 (values data (read-sequence data s)))))
671
672 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
673 ;; Settings
674 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
0f16035 d35h fixed return value of function get-user-email
d35h authored
675
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
676 (defun get-settings-file()
677 (merge-pathnames (user-homedir-pathname) "cl-test-grid-settings.lisp"))
678
679 (defun prompt-for-email ()
8d47120 d35h fixed text formatting
d35h authored
680 (format *query-io* "~a: " "Please enter your email for questions about this, test, your environment, etc.")
681 (force-output *query-io*)
682 (string-trim " " (read-line *query-io*)))
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
683
e2fbbf3 d35h fixed function name get-user-email and now it returns user email
d35h authored
684 (defun get-user-email ()
8d47120 d35h fixed text formatting
d35h authored
685 (let ((user-email nil))
0f16035 d35h fixed return value of function get-user-email
d35h authored
686 (handler-case
8d47120 d35h fixed text formatting
d35h authored
687 (if (string= "" (setf user-email(getf (safe-read-file (get-settings-file)
688 ) :user-email)))
689 (format t "Warning! Empty email is specified in the settings file ~a~%" (get-settings-file)))
690 (t ()
7b70188 d35h fixed text formatting
d35h authored
691 (progn
8d47120 d35h fixed text formatting
d35h authored
692 (write-to-file (list :user-email (setf user-email (prompt-for-email)))
693 (get-settings-file)))))
694 user-email))
0f16035 d35h fixed return value of function get-user-email
d35h authored
695
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
696 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
4cea853 Anton Vodonosov Use ql:quickload instead of asdf:operate when loading the GAE blobstore ...
avodonosov authored
697 ;; Test Runs
698 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
5be7639 Anton Vodonosov first commit
avodonosov authored
699
700 (defun run-descr (run)
701 "The description part of the test run."
c023819 Anton Vodonosov test run as plist (:descr <descr> :run-results <run-results>) instead of...
avodonosov authored
702 (getf run :descr))
5be7639 Anton Vodonosov first commit
avodonosov authored
703
704 (defun run-results (run)
705 "The list of test suite statuses for every library in the specified test run."
c023819 Anton Vodonosov test run as plist (:descr <descr> :run-results <run-results>) instead of...
avodonosov authored
706 (getf run :results))
5be7639 Anton Vodonosov first commit
avodonosov authored
707
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
708 (defun (setf run-results) (new-run-results test-run)
c023819 Anton Vodonosov test run as plist (:descr <descr> :run-results <run-results>) instead of...
avodonosov authored
709 (setf (getf test-run :results) new-run-results))
710
711 (defun make-run (description lib-results)
712 (list :descr description :results lib-results))
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
713
5be7639 Anton Vodonosov first commit
avodonosov authored
714 (defun fmt-time (universal-time &optional destination)
715 "The preferred time format used in the cl-test-grid project."
716 (multiple-value-bind (sec min hour date month year)
717 (decode-universal-time universal-time 0)
718 (funcall #'format
719 destination
720 "~2,'0D~2,'0D~2,'0D~2,'0D~2,'0D~2,'0D"
721 year month date hour min sec)))
722
0a04998 Anton Vodonosov Clarified a little bit more the comment about why hunchentoot is not inc...
avodonosov authored
723 (defun pretty-fmt-time (universal-time &optional destination)
724 "The human-readable time format, used in reports."
725 (multiple-value-bind (sec min hour date month year)
726 (decode-universal-time universal-time 0)
727 (funcall #'format
728 destination
729 "~2,'0D-~2,'0D-~2,'0D ~2,'0D:~2,'0D:~2,'0D"
730 year month date hour min sec)))
731
5be7639 Anton Vodonosov first commit
avodonosov authored
732 (defun make-run-descr ()
733 "Generate a description for a test run which might be
734 performed in the current lisp system."
735 (list :lisp (asdf::implementation-identifier)
736 :lib-world (format nil "quicklisp ~A"
737 (ql-dist:version (ql-dist:dist "quicklisp")))
738 :time (get-universal-time)
739 :run-duration :unknown
0f16035 d35h fixed return value of function get-user-email
d35h authored
740 :contact (list :email (get-user-email))))
5be7639 Anton Vodonosov first commit
avodonosov authored
741
742 (defun name-run-directory (run-descr)
743 "Generate name for the directory where test run
744 data (libraries test suites output and the run results) will be saved."
745 (format nil
746 "~A-~A"
747 (fmt-time (getf run-descr :time))
748 (getf run-descr :lisp)))
749
750 (defun test-output-base-dir ()
751 (merge-pathnames "test-runs/"
752 test-grid-config:*src-base-dir*))
753
754 (defun run-directory (run-descr)
755 (merge-pathnames (make-pathname
756 :directory (list :relative (name-run-directory run-descr))
757 :name nil
758 :type nil)
759 (test-output-base-dir)))
760
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
761 (defun lib-log-file (test-run-directory lib-name)
762 (merge-pathnames (string-downcase lib-name)
763 test-run-directory))
764
5be7639 Anton Vodonosov first commit
avodonosov authored
765 (defun save-lib-log (lib-name log test-run-directory)
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
766 (let ((lib-log-file (lib-log-file test-run-directory lib-name)))
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
767 (with-open-file (out lib-log-file
768 :direction :output
769 :if-exists :overwrite
770 :if-does-not-exist :create)
771 (write-sequence log out))))
5be7639 Anton Vodonosov first commit
avodonosov authored
772
773 (defun write-to-file (obj file)
774 "Write to file the lisp object OBJ in a format acceptable to READ."
775 (with-open-file (out file
776 :direction :output
777 :if-exists :supersede
778 :if-does-not-exist :create)
779 (pprint obj out))
780 obj)
781
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
782 (defun run-info-file (test-run-directory)
783 (merge-pathnames "test-run-info.lisp"
784 test-run-directory))
785
5be7639 Anton Vodonosov first commit
avodonosov authored
786 (defun save-run-info (test-run directory)
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
787 (let ((run-file (run-info-file directory)))
788 (write-to-file test-run run-file)))
789
790 (defun gae-blobstore-dir ()
791 (merge-pathnames "gae-blobstore/lisp-client/" test-grid-config:*src-base-dir*))
792
793 (defparameter *gae-blobstore-base-url* "http://cl-test-grid.appspot.com")
794
795 (defun get-blobstore ()
796 (pushnew (truename (gae-blobstore-dir)) asdf:*central-registry* :test #'equal)
4cea853 Anton Vodonosov Use ql:quickload instead of asdf:operate when loading the GAE blobstore ...
avodonosov authored
797 (ql:quickload '#:test-grid-gae-blobstore)
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
798 (funcall (intern (string '#:make-blob-store) '#:test-grid-gae-blobstore)
799 :base-url *gae-blobstore-base-url*))
800
801 (defun submit-logs (test-run-dir)
802 (let* ((blobstore (get-blobstore))
803 (run-info (safe-read-file (run-info-file test-run-dir)))
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
804 ;; prepare parameters for the SUBMIT-FILES blobstore function
805 (submit-params (mapcar #'(lambda (lib-result)
806 (let ((libname (getf lib-result :libname)))
807 (cons libname
808 (lib-log-file test-run-dir libname))))
809 (run-results run-info))))
810 ;; submit files to the blobstore and receive
811 ;; their blobkeys in response
812 (let ((libname-to-blobkey-alist
813 (test-grid-blobstore:submit-files blobstore
814 submit-params)))
a8e8a13 Anton Vodonosov Implemented online storage for libraries test logs, hosted on Google App...
avodonosov authored
815 ;; Now store the blobkeys for every library in the run-info.
816 ;; Note, we destructively modify parts of the previously
817 ;; read run-info.
818 (flet ((get-blob-key (lib)
819 (or (cdr (assoc lib libname-to-blobkey-alist))
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
820 (error "blobstore didn't returned blob key for the log of the ~A libary" lib))))
821 (setf (run-results run-info)
822 (mapcar #'(lambda (lib-result)
823 (setf (getf lib-result :log-blob-key)
824 (get-blob-key (getf lib-result :libname)))
825 lib-result)
826 (run-results run-info))))
827 ;; finally, save the updated run-info with blobkeys
828 ;; to the file. Returns the run-info.
829 (save-run-info run-info test-run-dir))))
830
5be7639 Anton Vodonosov first commit
avodonosov authored
831 (defun run-libtests (&optional (libs *all-libs*))
832 (let* ((run-descr (make-run-descr))
833 (run-dir (run-directory run-descr))
834 (lib-results))
835 (ensure-directories-exist run-dir)
836 (dolist (lib libs)
837 (let ((lib-result (run-libtest lib)))
838 (save-lib-log lib (getf lib-result :output) run-dir)
839 (remf lib-result :output)
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
840 (push lib-result lib-results)))
5be7639 Anton Vodonosov first commit
avodonosov authored
841 (setf (getf run-descr :run-duration)
842 (- (get-universal-time)
843 (getf run-descr :time)))
c023819 Anton Vodonosov test run as plist (:descr <descr> :run-results <run-results>) instead of...
avodonosov authored
844 (let ((run (make-run run-descr lib-results)))
5be7639 Anton Vodonosov first commit
avodonosov authored
845 (save-run-info run run-dir)
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
846 (format t "The test results were saved to this directory: ~%~A.~%"
847 (truename run-dir))
5bffbe5 Anton Vodonosov fixed regresson introduced when implementing GAE blobstore: the function...
avodonosov authored
848 (format t "~%Submitting libraries test logs to the online blobstore...~%")
849 (handler-case
b279089 Anton Vodonosov Fixed return value of the run-testlibs - if logs upload to the blobstore...
avodonosov authored
850 (progn
851 (setf run (submit-logs run-dir))
5bffbe5 Anton Vodonosov fixed regresson introduced when implementing GAE blobstore: the function...
avodonosov authored
852 (format t "The log files are successfully uploaded to the online blobstore.
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
853
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
854 Please submit the test run results file
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
855 ~A
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
856 to the cl-test-grid issue tracker:
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
857 https://github.com/cl-test-grid/cl-test-grid/issues
858
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
859 (we are working on automating the test results upload).~%"
b279089 Anton Vodonosov Fixed return value of the run-testlibs - if logs upload to the blobstore...
avodonosov authored
860 (truename (run-info-file run-dir))))
5bffbe5 Anton Vodonosov fixed regresson introduced when implementing GAE blobstore: the function...
avodonosov authored
861 (t (e) (format t "Error occured while uploading the libraries test logs to the online store: ~A: ~A.
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
862 Please submit manually the full content of the results directory
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
863 ~A
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
864 to the cl-test-grid issue tracker:
1195d22 Anton Vodonosov Fix the last merge
avodonosov authored
865 https://github.com/cl-test-grid/cl-test-grid/issues~%"
5bffbe5 Anton Vodonosov fixed regresson introduced when implementing GAE blobstore: the function...
avodonosov authored
866 (type-of e)
867 e
868 (truename run-dir))))
869 (format t "~%Thank you for the participation!~%")
870 run)))
efe7ab8 d35h added function get-user-email. reads from file if file is absent then as...
d35h authored
871
5be7639 Anton Vodonosov first commit
avodonosov authored
872 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
873 ;; Database
874 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
875 (defparameter *db* '(:version 0 :runs ()))
876
877 (defvar *standard-db-file*
878 (merge-pathnames "db.lisp"
879 test-grid-config:*src-base-dir*))
880
881 (defun add-run (run-info &optional (db *db*))
882 (push run-info (getf db :runs)))
883
85e3774 Anton Vodonosov Store the reports generated in the repository, in the directory reports-...
avodonosov authored
884 (defun save-db (&optional (db *db*) (stream-or-path *standard-db-file*))
5be7639 Anton Vodonosov first commit
avodonosov authored
885 (with-open-file (out stream-or-path
886 :direction :output
887 :element-type 'character ;'(unsigned-byte 8) + flexi-stream
888 :if-exists :overwrite
889 :if-does-not-exist :create)
890 (write db :stream out)))
891
892 (defun read-db (&optional (stream-or-path *standard-db-file*))
893 (with-open-file (in stream-or-path
894 :direction :input
895 :element-type 'character ;'(unsigned-byte 8) + flexi-stream
896 )
897 (safe-read in)))
898
899 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
900 ;; Reports
901 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
902
903 (defun generate-fake-run-results ()
904 "Generate fake test run result enought to test our reports."
905 (flet ((random-status ()
906 (let ((r (random 1.0)))
907 (cond ((< r 0.43)
908 :ok)
909 ((< r 0.86)
910 :fail)
911 (t :no-resource)))))
912 (let ((runs '()))
913 (dolist (lisp '("sbcl-fake-1" "sbcl-fake-2" "clisp-fake-1" "ccl-fake-1" "abcl-fake-2"))
914 (dolist (lib-world '("quicklisp-fake-2011-00-01" "quicklisp-fake-2011-00-02" "quicklisp-fake-2011-00-03"))
915 (let ((run-descr (list :lisp lisp
916 :lib-world lib-world
917 :time (get-universal-time)
918 :run-duration (+ 100 (random 90))
919 :contact (list :email
920 (nth (random 3) '("avodonosov@yandex.ru"
921 "other-user@gmail.com"
922 "foo@gmail.com")))))
923 (lib-results '()))
924 (dolist (lib *all-libs*)
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
925 (push (list :libname lib :status (random-status) :log-char-length 50)
926 lib-results))
c023819 Anton Vodonosov test run as plist (:descr <descr> :run-results <run-results>) instead of...
avodonosov authored
927 (push (make-run run-descr lib-results) runs))))
5be7639 Anton Vodonosov first commit
avodonosov authored
928 runs)))
929
00e5da5 Anton Vodonosov more correct name for the summary-table report: test-runs-report
avodonosov authored
930 (defvar *test-runs-report-template*
931 (merge-pathnames "test-runs-report-template.html"
5be7639 Anton Vodonosov first commit
avodonosov authored
932 test-grid-config:*src-base-dir*))
00e5da5 Anton Vodonosov more correct name for the summary-table report: test-runs-report
avodonosov authored
933
5be7639 Anton Vodonosov first commit
avodonosov authored
934 (defun vertical-html (libname)
935 (let ((maybeBr "")
936 (libname (string libname)))
937 (with-output-to-string (out)
938 (loop for char across libname
939 do (princ maybeBr out)
940 (princ (if (char= char #\-) #\| char) out)
941 (setf maybeBr "<br/>")))))
942
943 ;; example:
944 #|
945 (string= (vertical-html "cl-abc")
946 "c<br/>l<br/>|<br/>a<br/>b<br/>c")
947 |#
948
ec59872 Anton Vodonosov TODO comment about finalizing the terminology about our main data object...
avodonosov authored
949 ;; todo: this should be a blobstore method, but
950 ;; until we move the reporting to a separate
951 ;; asdf system, we don't want the dependency
952 ;; on blobstore here.
953 (defun blob-uri (blob-key)
954 (format nil "~A/blob?key=~A"
955 *gae-blobstore-base-url* blob-key))
956
957 (defun lib-log-local-uri (test-run lib-result)
958 (format nil "file://~A~A"
5be7639 Anton Vodonosov first commit
avodonosov authored
959 (run-directory (run-descr test-run))
ec59872 Anton Vodonosov TODO comment about finalizing the terminology about our main data object...
avodonosov authored
960 (string-downcase (getf lib-result :libname))))
961
962 (defun lib-log-uri (test-run lib-result)
e75f5e9 Anton Vodonosov Avoid compile warning about unused variable.
avodonosov authored
963 (declare (ignore test-run))
ec59872 Anton Vodonosov TODO comment about finalizing the terminology about our main data object...
avodonosov authored
964 (let ((blob-key (getf lib-result :log-blob-key)))
965 (if blob-key
966 (blob-uri blob-key)
967 "javascript:alert('The blobstore key is not specified, seems like the library log was not submitted to the online storage')")))
5be7639 Anton Vodonosov first commit
avodonosov authored
968
969 (defun single-letter-status (normalized-status)
970 (case normalized-status
971 (:ok "O")
972 (:fail "F")
973 (:no-resource "R")
974 (otherwise normalized-status)))
975
976 (defun status-css-class (normalized-status)
977 (case normalized-status
978 (:ok "ok-status")
979 (:fail "fail-status")
980 (:no-resource "no-resource-status")
981 (otherwise "")))
49f4d4c d35h add function export-to-csv
d35h authored
982
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
983 (defun render-single-letter-status (test-run lib-test-result)
cd2a46d Anton Vodonosov test runs report: if a library wasn't run in particulare test run, rende...
avodonosov authored
984 (if (null lib-test-result)
985 "&nbsp;"
986 (let ((status (normalize-status (getf lib-test-result :status))))
987 (format nil "<a class=\"test-status ~A\" href=\"~A\">~A</a>"
988 (status-css-class status)
989 (lib-log-uri test-run lib-test-result)
990 (single-letter-status status)))))
5be7639 Anton Vodonosov first commit
avodonosov authored
991
5f37895 Anton Vodonosov more correct name for the summary-table report: test-runs-report
avodonosov authored
992 (defun test-runs-table-html (&optional
993 (db *db*)
994 (status-renderer 'render-single-letter-status))
5be7639 Anton Vodonosov first commit
avodonosov authored
995 (with-output-to-string (out)
996 (write-line "<table cellspacing=\"1\" class=\"tablesorter\">" out)
997
b94ab6b Anton Vodonosov test runs report: added Start Time column.
avodonosov authored
998 (princ "<thead><tr style=\"vertical-align: bottom;\"><th>Start Time</th><th>Lib World</th><th>Lisp</th><th>Runner</th>" out)
5be7639 Anton Vodonosov first commit
avodonosov authored
999 (dolist (lib *all-libs*)
1000 (format out "<th>~A</th>" (vertical-html lib)))
1001 (write-line "</tr></thead>" out)
1002
1003 (write-line "<tbody>" out)
1004 (dolist (run (getf db :runs))
1005 (let ((run-descr (run-descr run))
1006 (lib-statuses (run-results run)))
b94ab6b Anton Vodonosov test runs report: added Start Time column.
avodonosov authored
1007 (format out "<tr><td>~A</td><td>~A</td><td>~A</td><td>~A</td>"
0a04998 Anton Vodonosov Clarified a little bit more the comment about why hunchentoot is not inc...
avodonosov authored
1008 (pretty-fmt-time (getf run-descr :time))
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
1009 (getf run-descr :lib-world)
1010 (getf run-descr :lisp)
1011 (getf (getf run-descr :contact) :email))
1012 (dolist (lib *all-libs*)
1013 (format out "<td>~A</td>"
1014 (funcall status-renderer run (find lib lib-statuses
1015 :key (getter :libname)))))
1016 (write-line "</tr>" out)))
5be7639 Anton Vodonosov first commit
avodonosov authored
1017 (write-line "</tbody>" out)
1018 (write-line "</table>" out)))
1019
5f37895 Anton Vodonosov more correct name for the summary-table report: test-runs-report
avodonosov authored
1020 (defun fmt-test-runs-report (html-table)
00e5da5 Anton Vodonosov more correct name for the summary-table report: test-runs-report
avodonosov authored
1021 (let* ((template (file-string *test-runs-report-template*))
5be7639 Anton Vodonosov first commit
avodonosov authored
1022 (placeholder "{THE-TABLE}")
1023 (pos (or (search placeholder template)
ccf71e3 Anton Vodonosov Fixed one forgotten *report-template* variable after it was renamed to *...
avodonosov authored
1024 (error "Can't find the placeholder ~A in the report template file ~A" placeholder *test-runs-report-template*))))
5be7639 Anton Vodonosov first commit
avodonosov authored
1025 (concatenate 'string
1026 (subseq template 0 pos)
1027 html-table
1028 (subseq template (+ pos (length placeholder))))))
1029
5f37895 Anton Vodonosov more correct name for the summary-table report: test-runs-report
avodonosov authored
1030 (defun test-runs-report (&optional (db *db*))
1031 (fmt-test-runs-report (test-runs-table-html db)))
1032
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
1033 (defun export-to-csv (out &optional (db *db*))
d88a518 d35h finally finished function get-user-email, added new property in result l...
d35h authored
1034 (format out "Lib World,Lisp,Runner,LibName,Status,TestDuration~%")
49f4d4c d35h add function export-to-csv
d35h authored
1035 (dolist (run (getf db :runs))
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
1036 (let ((run-descr (run-descr run)))
1037 (dolist (lib-result (run-results run))
d88a518 d35h finally finished function get-user-email, added new property in result l...
d35h authored
1038 (format out "~a,~a,~a,~a,~a,~a~%"
9dd75c2 Anton Vodonosov data format: instead of storing librari name as a key for the library re...
avodonosov authored
1039 (getf run-descr :lib-world)
1040 (getf run-descr :lisp)
1041 (getf (getf run-descr :contact) :email)
1042 (string-downcase (getf lib-result :libname))
d88a518 d35h finally finished function get-user-email, added new property in result l...
d35h authored
1043 (getf lib-result :status)
1044 (getf lib-result :test-duration))))))
49f4d4c d35h add function export-to-csv
d35h authored
1045
fa1c4e1 Anton Vodonosov Pivot reports. First working version; no comments, data cell is rendered...
avodonosov authored
1046 ;; ========= Pivot Reports ==================
1047
1048 (defun build-joined-index (db)
1049 (let ((all-results (make-hash-table :test 'equal)))
1050 (dolist (run (getf db :runs))
1051 (let* ((run-descr (run-descr run))
1052 (lisp (getf run-descr :lisp))
1053 (lib-world (getf run-descr :lib-world)))
1054 (dolist (lib-result (run-results run))
1055 (let ((libname (getf lib-result :libname)))
1056 (push lib-result
1057 (gethash (list lisp lib-world libname) all-results))))))
1058 all-results))
1059
1060
1061 #|
1062 HTML table properties:
1063 - rows and cols are not equal: html tables are row-first - TR includes TDs
1064
1065 Only the deepest level row and col fields will have corresponding
1066 TR and TD cells in the table data (we do not consider the table
1067 column headers now).
1068
1069 And the TD cells are included in a TR.
1070
1071 Algorithm sketch:
1072 Iterate over all row properties and their values in rows.
1073 For every deepest level combination create a TR (if
1074 any values exist for this combination).
1075
1076 Iterate over all the col properties and their values.
1077 For every deepest level combination create a TD.
1078
1079 partitioning structure
1080 -----------------------------
1081 field1 | field2
1082 -----------------------------
1083 field1 val1 (count)
1084 field2 val1 (count)
1085 field2 val2 (count)
1086 field1 val2 (count)
1087 field2 val1 (count)
1088 field2 val3 (count)
1089 field1 val3 (count)
1090 field2 val2 (count)
1091
1092
1093 <tr> <td> <td> <td>
1094 <tr> <td> <td>
1095 <tr> <td>
1096 <tr> <td> <td>
1097 <tr> <td> <td>
1098 <tr> <td>
1099 <tr> <td> <td> <td>
1100 <tr> <td> <td>
1101 <tr> <td> <td> <td>
1102 <tr> <td> <td>
1103 <tr> <td>
1104
1105 |#
1106
1107 (defun make-fields-values-setter (fields)
1108 "Creates a function which destructively modifies
1109 the specified fields in the index key passed to it
1110 as a parameter"
1111 (let ((index-key-setters (list :lisp #'(lambda (index-key lisp)
1112 (setf (first index-key) lisp))
1113 :lib-world #'(lambda (index-key lib-world)
1114 (setf (second index-key) lib-world))
1115 :libname #'(lambda (index-key libname)
1116 (setf (third index-key) libname)))))
1117 (flet ((field-setter (field)
1118 (or (getf index-key-setters field)
1119 (error "field ~A is unknown" field))))
1120 (let ((setters (mapcar #'field-setter fields)))
1121 #'(lambda (index-key field-vals)
1122 (mapc #'(lambda (setter field-val)
1123 (funcall setter index-key field-val))
1124 setters
1125 field-vals)
1126 index-key)))))
1127
1128 (defun make-fields-values-getter (fields)
1129 (let ((index-key-getters (list :lisp #'first
1130 :lib-world #'second
1131 :libname #'third)))
1132 (flet ((field-getter (field)
1133 (or (getf index-key-getters field)
1134 (error "field ~A is unknown" field))))
1135 (let ((getters (mapcar #'field-getter fields)))
1136 #'(lambda (index-key)
1137 (mapcar #'(lambda (getter)
1138 (funcall getter index-key))
1139 getters))))))
1140
1141 (defun calc-rows-and-cols (joined-index rows-fields cols-fields)
1142 (let ((rows-fields-getter (make-fields-values-getter rows-fields))
1143 (rows-fields-setter (make-fields-values-setter rows-fields))
1144 (rows (make-hash-table :test #'equal))
1145 (cols-fields-getter (make-fields-values-getter cols-fields))
1146 (cols-fields-setter (make-fields-values-setter cols-fields))
1147 (cols (make-hash-table :test #'equal)))
1148 (maphash #'(lambda (index-key index-value)
1149 (declare (ignore index-value))
1150 (setf (gethash (funcall rows-fields-getter index-key)
1151 rows)
1152 t)
1153 (setf (gethash (funcall cols-fields-getter index-key)
1154 cols)
1155 t))
1156 joined-index)
1157 (values (hash-table-keys rows)
1158 (hash-table-keys cols)
1159 #'(lambda (index-key row-addr col-addr)
1160 (funcall rows-fields-setter index-key row-addr)
1161 (funcall cols-fields-setter index-key col-addr)))))
1162
1163 (defstruct (header-print-helper :conc-name)
1164 (span 0 :type fixnum)
1165 (printed nil))
1166
1167 (defun subaddrs (row-address)
1168 (nreverse (maplist #'reverse (reverse row-address))))
1169
1170 (defun calc-spans (row-or-col-addrs)
1171 (let ((helpers (make-hash-table :test #'equal)))
1172 (dolist (row-or-col-addr row-or-col-addrs)
1173 (dolist (subaddr (subaddrs row-or-col-addr))
1174 (let ((helper (or (gethash subaddr helpers)
1175 (setf (gethash subaddr helpers)
1176 (make-header-print-helper)))))
1177 (incf (span helper)))))
1178 helpers))
1179
1180 (defun print-row-header (row-addr row-spans out)
1181 (dolist (subaddr (subaddrs row-addr))
1182 (let ((helper (gethash subaddr row-spans)))
1183 (when (not (printed helper))
1184 (format out "<td rowspan=\"~A\">~A</td>" (span helper) (car (last subaddr)))
1185 (setf (printed helper) t)))))
1186
1187 (defun print-table-headers (row-field-count col-field-count cols out)
1188 (let ((col-spans (calc-spans cols)))
1189 (dotimes (header-row-num col-field-count)
1190 (princ "<tr>" out)
1191 (dotimes (row-header row-field-count)
1192 (princ "<td>&nbsp;</td>" out))
1193 (dolist (col-addr cols)
1194 (let* ((cell-addr (subseq col-addr 0 (1+ header-row-num)))
1195 (helper (gethash cell-addr col-spans)))
1196 (when (not (printed helper))
1197 (format out "<td colspan=\"~A\">~A</td>" (span helper) (car (last cell-addr)))
1198 (setf (printed helper) t))))
1199 (format out "</tr>~%"))))
1200
1201 (defun pivot-table-html (out
1202 joined-index
1203 row-fields row-fields-sort-predicates
1204 col-fields col-fields-sort-predicates)
1205 (princ "<table border=\"1\">" out)
1206 (let (rows
1207 cols
1208 index-key-setter
1209 (row-comparator #'(lambda (rowa rowb)
1210 (list< row-fields-sort-predicates
1211 rowa rowb)))
1212 (col-comparator #'(lambda (cola colb)
1213 (list< col-fields-sort-predicates
1214 cola colb))))
1215
1216 (setf (values rows cols index-key-setter)
1217 (calc-rows-and-cols joined-index row-fields col-fields))
1218
1219 (setf rows (sort rows row-comparator)
1220 cols (sort cols col-comparator))
1221
1222 (print-table-headers (length row-fields) (length col-fields) cols out)
1223 (let ((row-spans (calc-spans rows))
1224 (index-key (make-sequence 'list (+ (length row-fields)
1225 (length col-fields)))))
1226 (dolist (row rows)
1227 (princ "<tr>" out)
1228 (print-row-header row row-spans out)
1229 (dolist (col cols)
1230 (funcall index-key-setter index-key row col)
1231 (let ((data (gethash index-key joined-index)))
1232 (format out "<td>~A</td>" data)))
1233 (format out "</tr>~%"))))
1234 (princ "</table>" out))
1235
1236 (defun print-pivot-reports (db)
1237 (let ((joined-index (build-joined-index db))
1238 (reports-dir (reports-dir)))
1239 (flet ((print-report (filename
1240 row-fields row-fields-sort-predicates
1241 col-fields col-fields-sort-predicates)
1242 (with-open-file (out (merge-pathnames filename reports-dir)
1243 :direction :output
1244 :element-type 'character ;'(unsigned-byte 8) + flexi-stream
1245 :if-exists :supersede
1246 :if-does-not-exist :create)
1247 (pivot-table-html out
1248 joined-index
1249 row-fields row-fields-sort-predicates
1250 col-fields col-fields-sort-predicates))))
1251
1252 (print-report "pivot_ql_lisp-lib.html"
1253 '(:lib-world) (list #'string<)
1254 '(:lisp :libname) (list #'string< #'string<))
1255 (print-report "pivot_ql_lib-lisp.html"
1256 '(:lib-world) (list #'string<)
1257 '(:libname :lisp) (list #'string< #'string<))
1258
1259 (print-report "pivot_lisp_lib-ql.html"
1260 '(:lisp) (list #'string<)
1261 '(:libname :lib-world) (list #'string< #'string<))
1262 (print-report "pivot_lisp_ql-lib.html"
1263 '(:lisp) (list #'string<)
1264 '(:lib-world :libname) (list #'string< #'string<))
1265
1266 (print-report "pivot_lib_lisp-ql.html"
1267 '(:libname) (list #'string<)
1268 '(:lisp :lib-world) (list #'string< #'string<))
1269 (print-report "pivot_lib_ql-lisp.html"
1270 '(:libname) (list #'string<)
1271 '(:lib-world :lisp) (list #'string< #'string<))
1272
1273 (print-report "pivot_ql-lisp_lib.html"
1274 '(:lib-world :lisp) (list #'string<)
1275 '(:libname) (list #'string< #'string<))
1276 (print-report "pivot_ql-lib_lisp.html"
1277 '(:lib-world :libname) (list #'string<)
1278 '(:lisp) (list #'string< #'string<))
1279
1280 (print-report "pivot_lisp-lib_ql.html"
1281 '(:lisp :libname) (list #'string<)
1282 '(:lib-world) (list #'string< #'string<))
1283 (print-report "pivot_lisp-ql_lib.html"
1284 '(:lisp :lib-world) (list #'string<)
1285 '(:libname) (list #'string< #'string<))
1286
1287 (print-report "pivot_lib-lisp_ql.html"
1288 '(:libname :lisp) (list #'string<)
1289 '(:lib-world) (list #'string< #'string<))
1290 (print-report "pivot_lib-ql_lisp.html"
1291 '(:libname :lib-world) (list #'string<)
1292 '(:lisp) (list #'string< #'string<)))))
1293
85e3774 Anton Vodonosov Store the reports generated in the repository, in the directory reports-...
avodonosov authored
1294 (defun generate-reports (&optional (db *db*))
1295
1296 (with-open-file (out (merge-pathnames "test-runs-report.html"
1297 (reports-dir))
1298 :direction :output
1299 :if-exists :supersede
1300 :if-does-not-exist :create)
d88a518 d35h finally finished function get-user-email, added new property in result l...
d35h authored
1301 (write-sequence (test-runs-report db) out))
85e3774 Anton Vodonosov Store the reports generated in the repository, in the directory reports-...
avodonosov authored
1302
1303 (with-open-file (out (merge-pathnames "export.csv"
1304 (reports-dir))
1305 :direction :output
1306 :if-exists :supersede
1307 :if-does-not-exist :create)
fa1c4e1 Anton Vodonosov Pivot reports. First working version; no comments, data cell is rendered...
avodonosov authored
1308 (export-to-csv out))
1309
1310 (print-pivot-reports db))
85e3774 Anton Vodonosov Store the reports generated in the repository, in the directory reports-...
avodonosov authored
1311
1312 (defun reports-dir ()
1313 (merge-pathnames "reports-generated/"
1314 test-grid-config:*src-base-dir*))
1315
5be7639 Anton Vodonosov first commit
avodonosov authored
1316 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
1317
1318 #|
1319
1320 Quicklisp download statistics:
1321
1322 http://blog.quicklisp.org/2010/11/project-download-statistics.html
1323
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
1324 colunmns: download count, has common-lisp test suite (as of quicklisp 2011-07-30).
5be7639 Anton Vodonosov first commit
avodonosov authored
1325
1326 714 + alexandria
1327 596 + babel
1328 520 + trivial-features
1329 503 + cffi
1330 450 + cl-ppcre
1331 423 - trivial-gray-streams
1332 404 + usocket
1333 403 + flexi-streams
1334 398 + bordeaux-threads
1335 393 - slime
4055e27 Anton Vodonosov Added cl-base64 to the tests.
avodonosov authored
1336 386 - cl+ssl (thre is a test.lisp, but it's not automated, and no (asdf:operate (op asdf:test-op) ...)
1337 371 - chunga
1338 370 + cl-base64
7ab792d Anton Vodonosov Added trivial-backtrace to the tests. (Always fails; it tries to create ...
avodonosov authored
1339 361 - cl-fad
1340 339 - md5
1341 327 - quicklisp-slime-helper
1342 323 + trivial-backtrace
1639b53 Anton Vodonosov Added puri. A typo fixed in the LIBTEST for trivial-backtrace. Decided t...
avodonosov authored
1343 321 - rfc2388 (there is a test.lisp, but there is no asdf:test-op, and the code in test.lisp
7ab792d Anton Vodonosov Added trivial-backtrace to the tests. (Always fails; it tries to create ...
avodonosov authored
1344 doesn't return fail/ok status, it jsut prints something to the console)
0a04998 Anton Vodonosov Clarified a little bit more the comment about why hunchentoot is not inc...
avodonosov authored
1345 317 - hunchentoot (there are tests and asdf:test-op, but I am affrait it might take
7622116 Anton Vodonosov Clarified a little the comment about why hunchentoot is not included.
avodonosov authored
1346 lot of work to automate it: test-op starts server and doesn't
0a04998 Anton Vodonosov Clarified a little bit more the comment about why hunchentoot is not inc...
avodonosov authored
1347 stop; I am also afraid it might hang sometimes; implementation
1639b53 Anton Vodonosov Added puri. A typo fixed in the LIBTEST for trivial-backtrace. Decided t...
avodonosov authored
1348 would also require checking for single-threaded lisps
1349 (by hunchentoot::*supports-threadss-p* ?)
7622116 Anton Vodonosov Clarified a little the comment about why hunchentoot is not included.
avodonosov authored
1350 and returning :no-resource. Leave hunchentoot for a later
1639b53 Anton Vodonosov Added puri. A typo fixed in the LIBTEST for trivial-backtrace. Decided t...
avodonosov authored
1351 stage)
1352 293 - salza2
1353 289 + puri
1d6be7b Anton Vodonosov Added anaphora
avodonosov authored
1354 285 - closer-mop (no asdf:test-op. there is a folder "test" with some file jeffs-code.lisp,
1355 but it's a code to reproduce some particular issue. It does not seem
1356 to be intended for automated regression testing of closer-mop)
1357 225 + anaphora
44b1c75 Anton Vodonosov Added parenscript
avodonosov authored
1358 224 + parenscript
d7c17f3 Anton Vodonosov Added trivial-garbage.
avodonosov authored
1359 221 - cl-who
1360 207 + trivial-garbage
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
1361 201 + iterate
1362 193 - cl-vectors
1363 190 - zpng
1364 177 - asdf-system-connections
1365 174 - zpb-ttf
1366 173 + uffi But the test suite is non trivial (for example, it defines asdf:compile-op
1367 for C files using make). Probably that's why quickisp does not
1368 make the uffi-tests.asd availabel for ql:quickload. A study is needed about
1369 how to include this system, therefore I avoid it for now.
1370 173 + metabang-bind
1371 170 - split-sequence
1372 164 - vecto (there is a test.lisp, but it's not automated, intended for manual run and eye-testing of the resulting images)
1373 163 + cl-json
1374 162 + cl-containers
1375 161 + metatilities-base
1376 159 - fare-utils
1377 156 + weblocks (do these tests start hunchentoot? seems no, it creates mock objects for request, response, etc.)
efd325f Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
1378 Unfortunately, as of today, (quicklisp:quickload :weblock-test) doesn't work,
1379 bacause the test suite does not compile.
5f523a0 Anton Vodonosov added metatilities-base cl-cont
avodonosov authored
1380 https://github.com/quicklisp/quicklisp-projects/issues/232
d4fc132 Anton Vodonosov added cl-json
avodonosov authored
1381 156 - fare-matcher (no test-op, but there is fare-matcher-test.asd with one test defined using stefil)
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
1382 148 - drakma
1383 144 + cl-cont
1384 143 - closure-common
1385 140 + moptilities
1386 138 - f-underscore
1387 137 + trivial-timeout
1388 136 + metatilities
efd325f Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
1389 135 + clsql (big test suite, requires database server(s). Runs the tests
1390 tests on the DB servers you specified in the configiration.
1391 Therefore we need to think how to represent results - we can't
696936b Anton Vodonosov Added iterate and metabang-bind.
avodonosov authored
1392 just collect results under the same name "clsql", because
d4fc132 Anton Vodonosov added cl-json
avodonosov authored
1393 different agents might have tested different servers.
1394 Conclusion: very useful test suite to include into
1395 our test set, but we will do it later).
efd325f Anton Vodonosov added moptilities, trivial-timeout, metatilities
avodonosov authored
1396 133 + cxml (But testing it requires manual preparation of test data:
1397 checkout the XML test suite from w3.org CVS repostory,
1398 patch it, build with ant. Conclusion: desirable to fix
1399 improve the situation, e.g. by providing ready
1400 to use test data files in a .tar archive
1401 and downloading it using quicklisp' http utility).
5be7639 Anton Vodonosov first commit
avodonosov authored
1402 |#
Something went wrong with that request. Please try again.