Skip to content
This repository

Collaborative testing of Common Lisp libraries

Octocat-spinner-32 agent agent/proc-update-quicklisp.lisp - fix typo in the commented out sect… March 07, 2014
Octocat-spinner-32 data tg-data:read-archive, register new archive file - db4.lisp December 23, 2013
Octocat-spinner-32 docs Update docs to reflect addition of jsown and bencode testsuites March 26, 2014
Octocat-spinner-32 gae-blobstore gae-blobstore: improve comment November 07, 2013
Octocat-spinner-32 reporting Default library dependency info is from quicklisp 2014-02-11 March 26, 2014
Octocat-spinner-32 reports-generated reporting: allow word wrap for notes in generated reports - works bet… December 22, 2013
Octocat-spinner-32 server Working on CL-implemented server, to run it on Heroku February 13, 2013
Octocat-spinner-32 sptm Some clarifications in sptm/README.md February 22, 2014
Octocat-spinner-32 storage Ensure utf-8 is used to store database. September 21, 2013
Octocat-spinner-32 testsuites Added jsown test suite March 23, 2014
Octocat-spinner-32 utils More readable test for tg-utils::ordering-comparator March 06, 2014
Octocat-spinner-32 .gitignore Added .gitignore October 14, 2012
Octocat-spinner-32 LICENSE first commit November 15, 2011
Octocat-spinner-32 Procfile heroky deployment: change Procfile to start the app from the saved li… March 01, 2013
Octocat-spinner-32 README.org Update docs to reflect addition of jsown and bencode testsuites March 26, 2014
Octocat-spinner-32 heroku-compile.lisp heroku deployment: remove any attempts to keep asdf:system-relative-p… March 01, 2013
Octocat-spinner-32 run-agent.bat.sample Specified more clearly the need to perform git update of agent sources. July 28, 2012
Octocat-spinner-32 run-agent.sample.lisp Fix a comment typo March 06, 2014
Octocat-spinner-32 run-agent.sh.sample close #12 - run-agent.sh.sample ensures the current directory is the … January 27, 2013
Octocat-spinner-32 sptm.asd Added some logging to main SPTM operations. December 19, 2012
Octocat-spinner-32 test-grid-agent.asd Send email notifications through the Heroku server. February 14, 2013
Octocat-spinner-32 test-grid-data.asd db schema change 6: test run description: the :contact (:email "someo… January 27, 2013
Octocat-spinner-32 test-grid-reporting.asd notes - new feature allowing to manually annotate certain failures an… September 24, 2013
Octocat-spinner-32 test-grid-server.asd Working on CL-implemented server, to run it on Heroku February 13, 2013
Octocat-spinner-32 test-grid-storage.asd Online storage for test results, implemented in the new module test-g… December 10, 2012
Octocat-spinner-32 test-grid-tests.asd Added clunit support to test-grid-testsuites. Adjusted cl-num-utils t… March 07, 2013
Octocat-spinner-32 test-grid-tests.lisp Sperate code to run stefil suites and hu.dwim.stefil. The same code c… December 22, 2013
Octocat-spinner-32 test-grid-testsuites.asd Sperate code to run stefil suites and hu.dwim.stefil. The same code c… December 22, 2013
Octocat-spinner-32 test-grid-utils.asd Splitting the monolitic module test-grid into: testsuites, agent, blo… August 01, 2012
README.org

Collaborative testing of Common Lisp libraries.

Test Results:

Bugs Reported

The Goal

Improve stability of the Common Lisp ecosystem by performing automated tests on as wide set of environments as possible. Environments vary mainly in 3 dimensions:

  1. Common Lisp implementations. They have incompatibilities, sometimes allowed by the standard (features specified as implementation-dependent), sometimes due to bugs, and sometimes in non-standardized features (threading, sockets, FFI, etc.). Moreover, implementations are evolving over the time, we should care about different versions of the same implementation.
  2. Growing number of CL libraries (which are also evolving over the time).
  3. Compatibility between library versions - libraries depend on other libraries and work correctly only with particular versions of the dependencies.

Therefore, when we run test suite of some library, we can speak about success or failure only in context of given Common Lisp implementation, and versions of all the dependency libraries.

Lets call the set of libraries with specified versions a “lib-world”. Important example of lib-world are Quicklisp distros.

It is hoped that constantly running tests on wide variety of environments will help the CL community by:

  1. Fast response to the library authors in case new changes cause bugs on implementations not available to the author. Otherwise author may receive notification about the bug years after he made the change, and thus the cost of fixing the bug may be much higher than fixing it week or two after the change.
  2. The same benefit for CL implementors - when they release new version, run test suites of large number of libraries and quickly detect possible regressions in the new release.
  3. Help to discover and maintain compatible set of library versions (e.g. Quicklisp distros).

Limitations.

Test suite success does not always mean the library is workable - there might be bugs which are not covered by the tests.

And the other way around - failed tests not always means the library is broken - it may be just a bug in the tests themselves.

Reducing this gap increases the utility of automated testing.

The Implementation Idea

Everyone can run a simple command which will run tests of Common Lisp libraries and publishes results online.

That way, instead of setting up a central build farm with all the possible hardware/OS/Lisp implementation combinations, we provide a way for Common Lisp users to contribute test results from their systems, and collectively monitor the CL world. (Of course, if anyone would want to setup a test farm, cl-test-grid simplifies this task too, by providing required building blocks).

Status

We provide tools to:

  • run tests
  • publish results online and access them programmatically
  • perform massive comparison of results and print HTML reports. For example, compare results of two Quicklisp versions, or compare two versions of a lisp implementation on the same Quicklisp.

The tests are run by a lisp program called test-grid-agent. User configures it with a list of CL implementations installed on his machine, and test-grid-agent runs tests of common lisp libraries on these implementations.

The tests performed by agent include:

  • fresh recompilation and loading of every ASDF system found in Quicklisp;
  • testsuites of some of the libraries.

For testsuites we have so far considered 125 libraries, starting from more often downloaded ones. 70 of these libraries have appropriate test suites (fully automated, no user interaction is needed) which are added to the cl-test-grid.

The test results represented as lisp data are stored online in so called test-grid-storage. By default test-grid-storage named “main” is where the test results are published.

You may also choose other storage name if you want to do some private testing and don’t want to mix your results with “main” results. The use cases for this are pre-release testing of CL implementations, ASDF, or patches to particular libraries.

The test-grid-storage named “main” is intended to contain results for the last 3 Quicklisp distros and is cleared periodically. Old results are archived in a plain lisp files in a separate git repository: https://github.com/cl-test-grid/cl-test-grid-results.

Besides this, we store online the output produced by each lisp process running test suites or compiling ASDF systems. The logs are referenced from the test results lisp data. This allows interested parties to navigate to the corresponding log to study the failure details.

Some HTML reports are demonstrated above, in the Test Results section. More information about reporting may be found in the test-gird-reporting module documentation.

The issue tracker at https://bugs.launchpad.net/common-lisp/ is used to record the bugs we submitted to various CL projects. We record bugs there to avoid duplicated bug reports.

cl-test-grid consists of the following components:

  • test-grid-agent - the program able to run tests and upload the results online.
  • test-grid-testsuites - testgrid adapters for testsuites of all the included libraries; used by agent.
  • test-grid-storage - online storage for test results (Lisp data). Based on sptm
  • sptm - simple in-memory persistence for lisp data based on transaction log stored online at Amazon Web Services.
  • test-grid-gae-blobstore - online storage for test logs (plain text files). Runs on Google App Engine.
  • test-grid-reporting - tools to generate reports from test results.
  • test-grid-data - support code for test results datastructures shared between other components; data is produced by agent, stored in storage and used by reporting
  • test-grid-server - tiny web application running at Heroku which forwards notification from agents to the cl-test-grid-notifications google group.
  • test-grid-utils - utilities.

The packages also have nicknames prefixed with tg- : tg-agent, tg-testsuites, and so on.

More detailed documentation for most of the modules may be found in the module subdirectory.

Participation

Running tests

Described in the agent module documentation.

We are looking for contributors who would run tests on LispWorks and Allegro.

Adding testsuite of your library

It’s necessary to contact the cl-test-grid mailing list and add a method for the libtest generic function to this file: testsuites/testsuites.lisp. The libtest function converts results of library test suite to a unified format understood by test-grid.

If you use some of the popular CL test frameworks, the implementation is usually simple, like this:

(defmethod libtest ((library-name (eql :cl-containers)))
  ;; The test framework used: lift.
  (quicklisp:quickload :cl-containers-test)
  (run-lift-test-suite :cl-containers-test))

If you are not the library author and don’t know how its test suite is run, look how the library tests are started in the asdf:perform method for asdf:test-op defined in the library .asd file.

In any case, contact the mailing list, we will help.

Resources

Project Planning. Use emacs org-mode to browse the TODO documents, because the github org to html translation is not adequate.

  • docs/TODO.org - The current TODO list
  • docs/TODO.done-archived.org - Archive of done TODO items

Donate

PalPal/Credit Card - handled by Common Lisp Foundation.

Donations can help the project by compensating our energy and expenses.

By default the donations provided are spent at the discretion of the project maintainers. If you want to fund a particular goal, it should be negotiated first.

Thank you.

Credits

  • Anton Vodonosov - the project author, developer and maintainer and test runner.
  • Juan Jose Garcia-Ripoll - his ECL test farm was one of the inspirations for the project. Also Juan Jose provided some very valuable feedback and suggestions about the project.
  • Paul Nathan - maintains Linux and Mac OS X build machines and constantly contributes test results for various Lisps.
  • Robert P. Goldman - for moral support and running tests in the early stage of the project.
  • Erik Huelsmann and efficito.com - for hosting a Linux VM to run our tests.
  • Mark Safronov - contributes test results.
  • Dave Cooper and Genworks International - for donation and running the tests.
  • Common Lisp Foundation - for handling donations
  • Daniel Zaru - implemented some code for retrieving test results from email box and the DB printing.
  • Patrick Stein - for donation, for integration with the nst test framework. Testsutes of nst and track-best. Contributing test results.
  • Timo Myyrä - contributes test results.
  • Jan Moringen - contributes test restuls.

Sorry if I missed anyone in this list. Please send me a note in this case.

Something went wrong with that request. Please try again.