Collaborative testing of Common Lisp libraries.
Improve stability of the Common Lisp ecosystem by performing automated tests on as wide set of environments as possible. Environments vary mainly in 3 dimensions:
- Common Lisp implementations. They have incompatibilities, sometimes allowed by the standard (features specified as implementation-dependent), sometimes due to bugs, and sometimes in non-standardized features (threading, sockets, FFI, etc.). Moreover, implementations are evolving over the time, we should care about different versions of the same implementation.
- Growing number of CL libraries (which are also evolving over the time).
- Compatibility between library versions - libraries depend on other libraries and work correctly only with particular versions of the dependencies.
Therefore, when we run test suite of some library, we can speak about success or failure only in context of given Common Lisp implementation, and versions of all the dependency libraries.
Lets call the set of libraries with specified versions a “lib-world”. Important example of lib-world are Quicklisp distros.
It is hoped that constantly running tests on wide variety of environments will help the CL community by:
- Fast response to the library authors in case new changes cause bugs on implementations not available to the author. Otherwise author may receive notification about the bug years after he made the change, and thus the cost of fixing the bug may be much higher than fixing it week or two after the change.
- The same benefit for CL implementors - when they release new version, run test suites of large number of libraries and quickly detect possible regressions in the new release.
- Help to discover and maintain compatible set of library versions (e.g. Quicklisp distros).
Of course, we should understand that test sute success does not always mean the library is workable - there might be bugs which are not covered by the tests. And the other way around - failed tests not always means the library is broken - it may be just a bug in the tests themselves. Reducing this gap increases the utility of automated testing.
The Implementation Idea
Everyone can run a simple command which will run tests of Common Lisp libraries and upload results to the central server.
That way, instead of setting up a central build farm with all the possible hardware/OS/Lisp implementation combinations, we provide a way for Common Lisp users to contribute test results from their systems, and collectively monitor the CL world. (Of course, if anyone whould want to setup a test farm, cl-test-grid simplifies this task too, by providing required building blocks).
We have a lisp program called test-grid-agent. User configures it with a list of CL implementations installed on his machine, and the test-grid-agent runs tests of common lisp libraries on these implementations.
The tests performed by agent include fresh recompilation and loading of every ASDF system found in Quicklisp, and also testsuites of some of the libraries.
We have so far considered 113 libraries. 56 of these libraries have appropriate test suites (fully automated, no user interaction is needed) which are added to the cl-test-grid.
Test-grid-agent may be run manually from command line or configured as a periodical task with cron or similar service. Each test suite is run in a separate lisp process Quicklisp is used to download the libraries to be tested (test-grid-agent bootstraps a private quicklisp and have no interference with a quicklisp installation user might have on his computer). The agent remembers what lisp implementations where already tested on what quicklisp distros, and doesn’t repeat the work it has already done.
The test results including:
- compile/load statuses of ASDF systems of every project in Quicklisp,
- results of the test suites (OK or a list of FAILed test cases)
for the tested lisp implementations / quicklisp distros are uploaded to admin and then published as a plain lisp data file in a separate git repository: https://github.com/cl-test-grid/cl-test-grid-results. Besides this, we store online the output produced by the lisp processes when running test suites or compiling ASDF systems. The logs are referenced from the test results lisp data. This allows interested parties to navigate to the corresponding log to study the failure details.
Having the pure lisp data it is easy to extract useful information from it. For example, compare how two versions of particular lisp implementation behave and detect regressions between them; detect regressions between Quicklisp distibutions, find out status of particular library on all the tested lisp implementations.
Some HTML reports deomostrating this are published here: http://common-lisp.net/project/cl-test-grid/. Clicking test status (OK/FAIL) in any report refers to the corresponding log file with the failure details.
Our current task in progress is to precisely document the data collected by agent, and provide more report examples demostrating how to analyze the data.
Meantime, feel free to send plain english queries to the mailing list.
test-grid-agent:agent is a lisp object able
to manage test exectuion by subordinate lisp
implementations (executables) and submit test
resutls to server.
It is created with function
and has 3 required configuration properties:
lisps- Paths to the lisp implementations that should be used to run tests.
preferred-lisp- The lisp implementation used when it is necessary to perform an auxiliary task requiring a separte lisp process, for example downloading libraries to be tested. It is therefore desirable to specify here a lisp implementation known to work reliable on your platform.
user-email- Your email so that we know who is contributing the test results and can contact you. The email is also published in the test results reports so that library authors or other interested parties can contact you with questions about your platform. If you are strongly opposed to publish your email, you can specify just some nickname here.
test-grid-agent:main runs the agent.
It is necessary to perform
git pull on agent sources
There are template scripts demonstrating how to load, cofigure and run agent by a single commant.
So, the steps:
git clone git://github.com/cl-test-grid/cl-test-grid.git
cp run-agent.sh.sample run-agent.sh; chmod +x run-agent.sh
cp run-agent.sample.lisp run-agent.lisp
- Edit the run-agent.sh (edit one line - the path to CCL).
- Edit the run-agent.lisp (paths to the lisp implementations, your email)
Next time all you need is to just invoke
./run-agent.sh. It will update the
cl-test-grid from git, run tests and upload the results.
Agent keeps log files in the cl-test-grid/work-dir/agent/logs/, where you can control what it has done.
Example crontab record to run agent at 10 o’clock every day:
# minute hour day_of_month month day_of_week command 0 10 * * * cd /home/testgrid/cl-test-grid/ && ./run-agent.sh
Details of what agent actually does
Simplified, the agent mode of operation may be represened by the following pseudo code:
(let ((current-quicklisp (update-quicklisp))) (loop for lisp in my-lisp-implementations (when (not (tested-already lisp current-quicklisp)) (let ((results-dir (complete-test-run lisp (or (find-unfinished-test-run lisp current-quicklisp) (make-new-test-run lisp current-quicklisp))))) (submit-results results-dir) (remember-tested lisp current-quicklisp) (cl-fad:delete-directory-and-files results-dir)))))
As you can see, the agent submits test results after completing full test set on a single lisp implementation.
The code, including the internal implementaton
complete-test-run is organized so that agent can
be interrupted (computer rebooted or hibernated,
agent process killed). When started again, it continues
the work from the point of interruption.
Testing single lisp implementation may take from 1-2 hours up to 10 hours or more (for ABCL - ABCL has long startup time, which becomes significant in our use case as we run every test suite or ASDF system compilation in a fresh lisp process).
Caveat of killing the agent: if you killed the agent process, (without rebooting the machine), the subordinate process running current testsute or compiling current ASDF system remains alive. Typically it takes less than a minute for it to finish, but sometimes it may take longer (the testsuite or library compilation may require longer time; or, in the worst case, test suite may hang). If you start agent again, it spawns new test running process, which can interfere with the old one via file system (.fasl files, output logs). Therefore it’s better to give the old child process time to finish before starting the agent again.
Parallel execution of multiple agents
Agent operates sequentially.
During its work, agent keeps it’s working data in a directory specified by the cofiguration property
work-dir- Defaults to the <cl-test-grid source code root>/work-dir/agent
The agent takes measures to ensure there is only one agent instance using this working directory.
This is acheaved by using a TCP port as a inter-process lock. When started agent tries to open a socket on the port. If it is successful, the agent continues. If the port is busy, the agent deduces there is another agent instance running, logs a warning and exists.
The port number is specified by configuration property
singleton-lock-portdefaults to 7685.
If you want to run several agent processes and distirbute testing work between them, you can assign each agent different set of lisp implemenations and give each agent different working directory and lock port.
Feel free to contact us if you have any questions or difficulties (see the mailing list address below).
We are looking for contributors who would agree to run
test-grid-agent periodically (ideally once a day, but even
once a month is OK).
Discussing the project
Feedback, discussions of the approach and suggestion for the open problems are very welcome.
Everyone interested is invited to the “mailing list” - http://groups.google.com/group/cl-test-grid.
Examples of the problems which need solution:
- Currently we run tests only on the quicklisp release. But it is very desirable to run tests on the latest library versions from the source control too. For example if we found a bug and the library author has fixed it, he might want to issue a request to cl-test-grid to run tests of the recent version of his library on all the platforms available. This feature would also help to ensure quicklisp distro quality before releasing the distro.
Adding testsuite of your library
It is quite easy - few lines of code.
Look how the library tests are started in the asdf:perform method for asdf:test-op defined in the library .asd file. Then use the same approach to define a method test-grid::libtest eql specialized for that library and send us this code.
See examples for the already added libraries in the testsuites/testsuites.lisp.
Lot of things may be done in this project. But the project has no independent value, it is only useful if it helps to improve the CL ecosystem quality. Fixing bugs in the CL libraries, writing more tests is the most important.
fix in this README:
Terminology - I say “quicklisp distro”, but if be precise, quicklisp calls it “quicklisp distro version”. But if I say “lib-world is a set of libraries with specified versions. An example of lib-world is a quicklisp distro version” the word “version” is repeated twice with diffirent sense - confusing.