Collaborative testing of Common Lisp libraries.
- for library developer: status of your library on different lisps
- for lisp implementation developer:
- for Quicklisp maintainer: compare test results of two quicklisp dist versions on various lisps
- more: work with the results database to build custom reports
Improve stability of the Common Lisp ecosystem by performing automated tests on as wide set of environments as possible. Environments vary mainly in 3 dimensions:
- Common Lisp implementations. They have incompatibilities, sometimes allowed by the standard (features specified as implementation-dependent), sometimes due to bugs, and sometimes in non-standardized features (threading, sockets, FFI, etc.). Moreover, implementations are evolving over the time, we should care about different versions of the same implementation.
- Growing number of CL libraries (which are also evolving over the time).
- Compatibility between library versions - libraries depend on other libraries and work correctly only with particular versions of the dependencies.
Therefore, when we run test suite of some library, we can speak about success or failure only in context of given Common Lisp implementation, and versions of all the dependency libraries.
Lets call the set of libraries with specified versions a “lib-world”. Important example of lib-world are Quicklisp distros.
It is hoped that constantly running tests on wide variety of environments will help the CL community by:
- Fast response to the library authors in case new changes cause bugs on implementations not available to the author. Otherwise author may receive notification about the bug years after he made the change, and thus the cost of fixing the bug may be much higher than fixing it week or two after the change.
- The same benefit for CL implementors - when they release new version, run test suites of large number of libraries and quickly detect possible regressions in the new release.
- Help to discover and maintain compatible set of library versions (e.g. Quicklisp distros).
Of course, we should understand that test sute success does not always mean the library is workable - there might be bugs which are not covered by the tests. And the other way around - failed tests not always means the library is broken - it may be just a bug in the tests themselves. Reducing this gap increases the utility of automated testing.
The Implementation Idea
Everyone can run a simple command which will run tests of Common Lisp libraries and publishes results online.
That way, instead of setting up a central build farm with all the possible hardware/OS/Lisp implementation combinations, we provide a way for Common Lisp users to contribute test results from their systems, and collectively monitor the CL world. (Of course, if anyone whould want to setup a test farm, cl-test-grid simplifies this task too, by providing required building blocks).
We provide tools to:
- run tests
- publish results online and access them programmatically
- perform massive comparison of results and print HTML resports. For example, compare results of two Quicklisp versions, or compare two versions a lisp implementation on the same Quicklisp.
The tests are run by a lisp program called
User configures it with a list of CL implementations installed
on his machine, and
test-grid-agent runs tests of common lisp
libraries on these implementations.
The tests performed by agent include:
- fresh recompilation and loading of every ASDF system found in Quicklisp;
- also testsuites of some of the libraries.
For testsuites we have so far considered 114 libraries, starting from more often downloaded ones. 57 of these libraries have appropriate test suites (fully automated, no user interaction is needed) which are added to the cl-test-grid.
The test results represented as lisp data are stored online
in so called
test-grid-storage. By default
named “main” is where the test results are published.
You may also choose other storage name if you want to do some private testing and don’t want to mix your results with “main” results. The use cases for this are pre-release testing of CL implementations, ASDF, or patches to particular libraries.
test-grid-storage named “main” is intended to contain resutls
for the last 3 Quicklisp distros and is cleared periodically.
Old results are archvied in a plain lisp file in a separate git repository:
Besides this, we store online the output produced by each lisp process running test suites or compiling ASDF systems. The logs are referenced from the test results lisp data. This allows interested parties to navigate to the corresponding log to study the failure details.
Some HTML reports are demonstrated above, in the Test Results
section. More information about reporting may be found
test-gird-reporting module documentation.
The issue tracker at https://bugs.launchpad.net/common-lisp/ is used to record the bugs we submitted to various CL projects. We record bugs there to avoid duplicated bug reports.
cl-test-grid consists of the following components:
test-grid-agent- the program able to run tests an upload the results online
test-grid-testsuites- testgrid adapters for testsuites of all the included libraries; used by agent
test-grid-storage- online storage for test results; based on
sptm- simple in-memory persistence for lisp data based on transaction log stored online at Amazon Web Services
test-grid-reporting- tools to generate reports from test results
test-grid-data- support code for test results datastructures shared between other components;
datais produced by
agent, stored in
storageand used by
test-grid-gae-blobstore- online storage for test logs; runs on Google App Engine
The packages also have have nicknames prefixed with
tg-testsuites, and so on.
More detailed documentation for most of the modules may be found in the module subdirectory.
Described in the
agent module documentation.
We are looking for contributors who would agree to run
test-grid-agent periodically (ideally once a day, but even
once a month is OK).
Adding testsuite of your library
It’s necessary to contact the cl-test-grid mailing list
and add a method for the
libtest generic function to this file:
libtest function converts
results of library test suite to a unified format
understood by test-grid.
If you use some of the popular CL test frameworks, the implementation is usually simple, like this:
(defmethod libtest ((library-name (eql :cl-containers))) ;; The test framework used: lift. (quicklisp:quickload :cl-containers-test) (run-lift-test-suite :cl-containers-test))
If you are not the library author and don’t know how
it’s test suite is run, look how the library tests are
started in the
asdf:perform method for
defined in the library .asd file.
In any case, contact the mailing list, we will help.
fix in this README:
Terminology - I say “quicklisp distro”, but if be precise, quicklisp calls it “quicklisp distro version”. But if I say “lib-world is a set of libraries with specified versions. An example of lib-world is a quicklisp distro version” the word “version” is repeated twice with diffirent sense - confusing.