Collaborative testing of Common Lisp libraries.
Improve stability of the Common Lisp ecosystem by performing automated tests on as wide set of environments as possible. Environments vary mainly in 3 dimensions:
- Common Lisp implementations. They have incompatibilities, sometimes allowed by the standard (features specified as implementation-dependent), sometimes due to bugs, and sometimes in non-standardized features (threading, sockets, FFI, etc.). Moreover, implementations are evolving over the time, we should care about different versions of the same implementation.
- Growing number of CL libraries (which are also evolving over the time).
- Compatibility between library versions - libraries depend on other libraries and work correctly only with particular versions of the dependencies.
Therefore, when we run test suite of some library, we can speak about success or failure only in context of given Common Lisp implementation, and versions of all the dependency libraries.
Lets call the set of libraries with specified versions a “lib-world”. Important example of lib-world are Quicklisp distros.
It is hoped that constantly running tests on wide variety of environments will help the CL community by:
- Fast response to the library authors in case new changes cause bugs on implementations not available to the author. Otherwise author may receive notification about the bug years after he made the change, and thus the cost of fixing the bug may be much higher than fixing it week or two after the change.
- The same benefit for CL implementors - when they release new version, run test suites of large number of libraries and quickly detect possible regressions in the new release.
- Help to discover and maintain compatible set of library versions (e.g. Quicklisp distros).
Of course, we should understand that test sute success does not always mean the library is workable - there might be bugs which are not covered by the tests. And the other way around - failed tests not always means the library is broken - it may be just a bug in the tests themselves. Reducing this gap increases the utility of automated testing.
The Implementation Idea
Everyone can run a simple command which will run tests of Common Lisp libraries and upload results to the central server.
That way, instead of setting up a central build farm with all the possible hardware/OS/Lisp implementation combinations, we provide a way for Common Lisp users to contribute test results from their systems, and collectively monitor the CL world. (Of course, if anyone whould want to setup a test farm, cl-test-grid simplifies this task too, by providing required building blocks).
We have a lisp program called test-grid-agent. User configures it with a list of CL implementations installed on his machine, and the test-grid-agent runs tests of common lisp libraries on these implementations.
The test results are uploaded to the central server and published as a lisp data file and as HTML reports here: http://common-lisp.net/project/cl-test-grid/. Clicking library test status (OK/FAIL) in any report refers to the library test logs with the failure details.
We have so far considered 113 libraries. 56 of these libraries have appropriate test suites (fully automated, no user interaction is needed) and are added to the cl-test-grid.
Test-grid-agent may be run manually from command line or configured as a periodical task with cron or similar service. Each test suite is run in a separate lisp process Quicklisp is used to download the libraries to be tested (test-grid-agent bootstraps a private quicklisp and have no interference with a quicklisp installation user might have on his computer). The agent remembers what lisp implementations where already tested on what quicklisp distros, and doesn’t repeat the work it has already done.
The most appreciated way to participate.
test-grid-agent:agent is a lisp object able
to manage test exectuion by subordinate lisp
implementations (executables) and submit test
resutls to server.
It is created with function
and has 3 configuration properties:
lisps- Paths to the lisp implementations that should be used to run tests.
preferred-lisp- The lisp implementation used when it is necessary to perform an auxiliary task requiring a separte lisp process, for example downloading libraries to be tested. It is therefore desirable to specify here a lisp implementation known to work reliable on your platform.
user-email- Your email so that we know who is contributing the test results and can contact you. The email is also published in the test results reports so that library authors or other interested parties can contact you with questions about your platform. If you are strongly opposed to publish your email, you can specify just some nickname here.
test-grid-agent:main runs the agent.
It is necessary to perform
git pull on agent sources
often - that’s how agent receives new tasks from admin.
We provide template scripts demonstrating how to load, cofigure and run agent by a single commant.
The first time you will need do these steps:
git clone git://github.com/cl-test-grid/cl-test-grid.git
cp run-agent.sh.sample run-agent.sh; chmod +x run-agent.sh
cp run-agent.sample.lisp run-agent.lisp
- Edit the run-agent.sh (edit one line - the path to CCL).
- Edit the run-agent.lisp (paths to the lisp implementations, your email)
Next time all you need is to just invoke
./run-agent.sh. It will update the
cl-test-grid from git, run tests and upload the results.
Agent keeps log files in the cl-test-grid/work-dir/agent/logs/, where you can control what it has done.
Example crontab record to run agent at 10 o’clock every day:
# minute hour day_of_month month day_of_week command 0 10 * * * cd /home/testgrid/cl-test-grid/ && ./run-agent.sh
Feel free to contact us if you have any questions or difficulties (see the mailing list address below).
We are looking for contributors who would agree to run
test-grid-agent periodically (ideally once a day, but even
once a month is OK).
Discussing the project
Feedback, discussions of the approach and suggestion for the open problems are very welcome.
Everyone interested is invited to the “mailing list” - http://groups.google.com/group/cl-test-grid.
Examples of the problems which need solution:
- Currently we run tests only on the quicklisp release. But it is very desirable to run tests on the latest library versions from the source control too. For example if we found a bug and the library author has fixed it, he might want to issue a request to cl-test-grid to run tests of the recent version of his library on all the platforms available. This feature would also help to ensure quicklisp distro quality before releasing the distro.
Adding more libraries
It is quite easy - few lines of code.
Look how the library tests are started in the asdf:perform method for asdf:test-op defined in the library .asd file. Then use the same approach to define a method test-grid::libtest eql specialized for that library and send us this code.
See examples for the already added libraries in the testsuites/testsuites.lisp.
Lot of things may be done in this project. But the project has no independent value, it is only useful if it helps to improve the CL ecosystem quality. Fixing bugs in the CL libraries, writing more tests is the most important.
fix in this README:
Terminology - I say “quicklisp distro”, but if be precise, quicklisp calls it “quicklisp distro version”. But if I say “lib-world is a set of libraries with specified versions. An example of lib-world is a quicklisp distro version” the word “version” is repeated twice with diffirent sense - confusing.