Collaborative testing of Common Lisp libraries.
Improve stability of the Common Lisp ecosystem by performing automated tests on as wide set of envionments as possible. Environments vary mainly in 3 dimensions:
- Common Lisp implementations. They have incompatibilities, sometimes allowed by the standard (features specified as implementation-dependent), somtimes due to bugs, and sometimes in non-standartized features (threading, sockets, FFI, etc.). Moreover, impementations are evolving over the time, we should care about different versions of the same implementation.
- Growing number of CL libraries (which are also evolving over the time).
- Compatibility between library versions - libraries depend on other libraries and work correctly only with particular versions of the dependencies.
Therefore, when we run test suite of some library, we can speak about success or failure only in context of given Common Lisp implementation, and versions of all the dependency libraries.
Lets call the set of libraries with specified versions a “lib-world”. Important example of lib-world are Quicklisp distros.
It is hoped that constantly running tests on wide variety of environments will help the CL community by:
- Fast response to the library authors in case new changes cause bugs on implementations not available to the author. Otherwise author may receive notification about the bug years after he made the change, and thus the cost of fixing the bug may be much higher than fixing it week or two after the change.
- The same benefit for CL implementors - when they release new version, run test suites of large number of libraries and quickly detect possible regressions in the new release.
- Help to discover and maintain compatible set of library versions (e.g. Quicklisp distros).
Of course, we should understand that test sute success does not always mean the library is workable - there might be bugs which are not covered by the tests. And the other way around - failed tests not always means the library is broken - it may be just a bug in the tests themselves. Reducing this gap increases the utility of automated testing.
The Implementaion Idea
Everyone can run a simple command which will run tests of Common Lisp libraries and upload results to the central server.
That way, instead of setting up a central build farm with all the possible hardware/OS/Lisp implementation combinations, we provide a way for Common Lisp users to contribute test results from their systems, and collectively monitor the CL world. (Of course, if anyone whould want to setup a test farm, cl-test-grid simplifies this task too, by providing required building blocks).
We have a lisp program called test-grid-agent. User configures it with a list of CL implementations installed on his machine, and the test-grid-agent runs tests on these implenentations, starting each test suite in a separate lisp process.
Test-grid-agent may be run manually from command line or configured as a periodical task with cron or similary service. The agent remembers what lisp implementations where already tested on what quicklisp distros, and doesn’t repeat the work it has already done.
Quicklisp is used to download the libraries to be tested (test-grid-agent bootstraps a private quicklisp installation for use by the child lisp processes running testsuites; therefore no interference happen with a quicklisp installation user might have on his computer).
The test results are uploaded to the central server and published as a lisp data file and as HTML reports here: http://common-lisp.net/project/cl-test-grid/. Clicking library test status (OK/FAIL) in any report refers to the library test logs with the failure details.
We have considered 113 libraries. 56 of these libraries have appropriate test suites (fully automated, no user interaction is needed) and are added to the cl-test-grid.
The most appreciated way to participate.
The first time you will need do the following:
- git clone git://github.com/cl-test-grid/cl-test-grid.git
- cd cl-test-grid
- cp run-agent.sh.sample run-agent.sh; chmod +x run-agent.sh
- Edit the run-agent.sh (edit one line).
- cp run-agent.sample.lisp run-agent.lisp
- Edit the run-agent.lisp (specfify all the lisp implementations you have, and some other settings)
Next time all you need is to just start ./run-agent.sh. It will update the cl-test-grid from git, update quicklisp to the latest distro, run tests and upload the results.
Agent keeps log files in the cl-test-grid/work-dir/agent/logs/, where you may control what it has done.
Example crontab record to run agent at 10 o’clock every day:
# m h dom mon dow command 0 10 * * * cd /home/testgrid/cl-test-grid/ && ./run-agent.sh
Feel free contact us if you have any questions or difficulties (see the mailing list address below).
We are looking for contributors who would agree to run test-grid-agent periodically (ideally once a day, but even once a month is OK).
Discussing the project
Feedback, discussions of the apporach and suggestesion for the open problems are very welcome.
Everyone interested is invited to the “mailing list” - http://groups.google.com/group/cl-test-grid.
Examples of the problems which need solution:
- Currently we run tests only on the quicklisp release. But it is very desirable to run tests on the latest library versions from the source control too. For example if we found a bug and the library author has fixed it, he might want to issue a request to cl-test-grid to run tests of the recent version of his library on all the platforms available. This feature would also help to ensure quicklisp distro quality before releasing the distro.
Adding more libraries
It is quite easy - few lines of code.
Look how the library tests are started in the asdf:perform method for asdf:test-op defined in the library .asd file. Then use the same approach to define a method test-grid::libtest eql specilized for that library and send us this code.
See examples for the already added libraries in the test-grid.lisp.
Lot of things may be done in this project. But the project has no independent value, it is only useful if it helps to improve the CL ecosystem quality. Fixing bugs in the CL libraries, writing more tests is the most important.
fix in this README:
Terminology - I say “quicklisp distro”, but if be precise, quicklisp calls it “quicklisp distro version”. But if I say “lib-world is a set of libraries with specified versions. An example of lib-world is a quicklisp distro version” the word “version” is repeated twice with diffirent sense - confusing.