Skip to content
Fetching contributors…
Cannot retrieve contributors at this time
399 lines (381 sloc) 18.7 KB

;;;; -*- Mode: org; indent-tabs-mode: nil; coding: utf-8; show-trailing-whitespace: t -*-

Milestone 0

information about test run:

lisp-version-string, lib-world, author contact (get it from some settings file), date, run-duration

organize the database file format

format the DB equally on all the Lisps

so that diff shows only new records. (use pprint ?) 4h

better decision for library name representation.

This representation is used in the:
  • libtest method parameter (eql specialized)
  • in the database and testrun datastructure.

Possible alternatives: a keyword good for READ (package independent), good for EQL specializer good for GETF when working with the database a symbol from the test-grid package

  • good for EQL specializer
  • package dependent in READ
  • good for GETF when working with the database
  • adds one more (unnecessary) way to represent a library in addition the specified for ASDF and Quicklisp

or a downcased string

  • needs special handling in libtest eql specialization
  • good ro READ (package independent)
  • needs care to work with GETF when working with the database

A tool to generate fake DB content to test reporting

simpliest reporting to overview the libraries test statuses [6/6]

Test Runs report: every test run as a row in a table

  • legend or a tooltip in the report for test statuses
  • color for statuses
  • use the online blob URL in the report

A pivot -like table report of library test results, allowing

rows/columns to be any of quicklisp distro, lisp version library name. With grouping and sorging.

Fix rows and columns highlight in the pivot reports

CSV export of the database to use it then with spreadsheets,

google fusion tables, etc. Initial intent was to format it as a pivot for various projections (by quicklisp releases, by platform, etc). But neither google docs spreadsheet, nor google fusion table allow as to format results as we want (the main problem, it is impossible to use a custom aggregation function for pivot cells, because standard aggregation functions are numeric, but we want a report cell to represent test result(s) for a particular library, i.e. :ok, :fail, :no-resource). 5h

Test that the test-duration field value

(Common Lisp rational) can be read by spreadsheet software (MS/Open Offices, Google Spreadsheets).

an overview page with brief explanation of and links to all the reports

simple UI (command line) with guiding messages

for the user who runs the tests. Spend as little efforts as possible on this task, to release quickly. 4h

readme with explanation of the project goal and

how to use it 5h

change db format

  • test run as plist (:descr <descr> :run-results <run-results>) instead of just (<descr> <run-results>)
  • run-results as a list instead of plist; libname which was a plist key is now a property of the lib-result object. It is more convenient for standard mapping functions, instead of current do-lib-results.

add more libraries: total number of 20 libraries

is enough for the beginning. Result: we have 23 libraries.

when loading of a library or library test system

fails, ensure we have the error description in the output 0.5h

The “thank you” message: where exactly to submit test results?

Specify an email or issue tracker of the cl-test-grid project.

how to store public (central) database and failed library

outputs (files). An appealing way is to store it in the same git repository on github, but with the std-out files the repository will quickly grow to an unconvenient size (for new people the checkout procedure will be too long to be considered convenient) 5h Solution: files are stored in Google App Engine blob store.

More detailed output for libraries using the RT test

framework. Ensure the libs with other test framework are all have sufficiently detailed output too.

Log of the BABEL tests (generated by the Stefil test

framework) does not contain information about errors. Add these details.

Some libraries (babel and cl-json) stil print messages to

console, meaning their output is not only standard-output and standard-error. Fix that, all the output should be in the log files, but not on console.

run the tests on all the implementations available for us.

publish the reports

test-run-info.lisp as attachment, instead of letter body

Fix lisp name sorting in pivot_lib-lisp_ql.html

finalize the decision what command user runs

to performs the tests. Describe this main command in the README (in the first paragraph). The solution: user runs (load “agent.lisp”); precondition: quicklisp installed.

When the settings file is populated, inform user where

it is stored (and to be sure user understood us, ask him to press Enter before continuing).

When asking user for email, explain him more carefully

how the email will be used and ensure there is a way to avoid providing email.

ECL: recover from stack alexandria overflow and continute tests

ABCL: recover from cl-ppcre hung-up and continue tests

Optimize horizontal space in the reports: the headers

take to much space

more abstract accessor to parts of DB info instead of

getf by properties: run-descr, run-results. 1h

safe-read database

create a project with asdf system


DB file path based on the asdf system location


accumulate failed library output


run-libtests: redirect the library output to log file directly

without caching it in memory. It is more convenient when you are watching the testing process, you can observe the file being populated with logs (because some libraries, like flexi-streams, take time about minute to finish, and if during this minute nithing happens it is not user-friendly)

Milestone 0.1

move closed TODO items to another (archive) file, so that the

web presentation of the TODO file will be understandable for the public.

new status, :expected-failure (or better :know-failure).

Also maybe :unexpected-ok.

clean up the pivot reports Lisp code

Milestone 0.3.1

automatically find regressions between quicklisp disto versions

test results detailed to individual tests for remaining test frameworks

add more libraries

Added 34 libraries, the total number now is 57.

shorten the blob keys of online library logs

0.3.2 Reports Improve 1

floating report table headers, so that when we scroll,

headers are always visible and we always see what lisp/libary/lib-world corresponds to the table cell we are interested in.

Yellow color for known-fail and unexpected-ok aggregated statuses

0.3.3 Current:

automate processing of library results submited by email

0.3.4 Test result for Quicklisp 2012-03, 2012-04

run tests with the newly added libraries on the quicklisp 2012-03-07,

so that we can compare their results with the new quicklisp release when it is out.

collect the test results for the next quicklisp release (2012-04-xx)

analyze the failures of Quicklisp 2012-04.

0.4.0 Test Running Robustness - lisp-agent

top level error handler in test-grid-agent:main

write the unhandled error stacktrace (with causes) into log and exit - never enter debugger as the program should be suitable to run in background with cron.

prevent starting of two agents (try to open a socket on a constant port)

create test-grid-agent:api-compatible-p

check what lisps other than CCL can run lisp-agent

update to reflect the current status and priorities (in

other words, reflect the changes during the last month)

prepare the announcement for the mailing-list

publish the lisp-agent - merge lisp-agent to master

Give every agent and unique ID. store the ID in the agent persistence.

error handling in the main process - during a test run and in other situations

For example, after testing some library, the output file with library status is not found. Record :FAIL and proceed to further libraries. How to handle error during test results submission?

Agent: when test results upload failed, do not mark this lisp as tested in persistence.

run each test suite in a separate process

enforce the libraries source code recompilation on every test run

by configuring asdf:output-translations to a temporary subdirectory in the test-run directory

0.4.1 Code cleanup after finishing lisp-agent

review test suites which have hard-coded failures

due to hang-ups or other problems and consider removing the hardcoded failures as we now have timeouts for hang-ups.

push forward breaking-up the old monolitic module test-grid:

test grid systems will consist of modules for:
  • testsuites a function libtest(libname) runs test suite of the specified library, downloading the lib from quicklisp if necessary, and returns test result in a unified format. I.e. the module knows how to invoke the library test suite and has adapters for test results of various test frameworks to the testgrid test result format.
  • agent the program with can run test in child lisp processes, knows which lisps the contributor has on the machine, decides when to run tests, and can upload the test results to the central server. [the changes required are small - agent will receive some code from the current test-grid.lisp, for example test suite log header/footer printing. Maybe the test suite log will include whole lisp process output, from the lisp startup (now we start logging bit later, right before invoking the test suite)]
  • admin code to receive test results and add to db [remains unchanged]
  • reporting [reamains unchanged]
  • blobstore [remains unchanges]
  • data objects moving between other modules: test run, test results, printing for these objects; and the database
  • utils

agent fs structure functions: make the workdir root

a configurable property of agent, and fs structure functions shoud accept agent as a parameter.

Fixes user request by Greg Pfeil: don’t install quicklisp in ASDF source registry,

In future it will allow for increased speed on multiprocessor systems by starting several agent processes, each assigned different set of lisp implementations. It would require another agent config property - lock-port, for the socket whcih assures agent is a signelton. Every agent have different work-dir and different port.

Several agents may even run in different threads of a single lisp process (if global shared libraries like asdf whould allow this).

0.5.0 Separate status value for compilation failure

Important in general and a user reques by Juan Jose Garcia-Ripoll.

Test status should include separate indicator for the library compilation and loading.

This allows to add even libraries without test cases to the test grid.

0.5.1 Test loading (including fresh recompilation) of every ASDF system Quicklisp.

This means even if the library doesn’t have a test suite, or the test suite is not yet integrated into the test-grid, we will know it’s compilation status.

Complications: one project may have more than one ASDF system, and in such case it’s not clear how to relate compilation failures with test suite status, if only part of ASDF systems failed to compile.

Change agent to perform load test of every ASDF system in Quicklisp

Continuable test runs

Don’t loose test run results if it was interrupted (due to errors, network outage or agent termination)

Workaround the GAE issue

rework ABCL report

make sure old reports work when DB contains ASDF systems load results

collect results for libraries missed in Quicklisp 2012-09-09 results

Some are missed because of agent bug: list of project was retrieved from wrong (old) quicklisp: cartesian-product-switch, cl-cheshire-cat, cl-grace, coleslaw, com.clearly-useful.iterator-protocol, com.clearly-useful.protocols, com.clearly-useful.sequence-protocol, formlets, glu-tessellate, infix-dollar-reader, lambda-lift, optima, place-modifiers, pzmq.

bknr-datastore testsuite was not run because it was added to testgrid as bknr.datastore.

Also run bknr-datastore on quicklisp 2012-08-11.

The list of projects to test should be reteieved from the quicklisp

tested by agent in child processes, but not the quicklisp used to run the agent itself.

remove old versions of perform-test-run, run-tests, submit-logs.

rework and announce new quicklisp diff report

<in progress>

remove the TODO list from the bottom of perform-test-run.lisp

when killing child process, sometimes we may have “No such process” simple-error

if the process already finished. Handle it.

Testsuite timeout detection should sustain laptop hibernation

db.lisp: deterministic formatting of load-results

db.lisp: don’t print testsuite values if there is no testsuite (values are NIL)

don’t print testing status/duration/log-key when absent (relevant for new resuls where most of the systems don’t have test suite)

db.lisp: don’t print load-results when absent

(only releant for old results submitted before agent become able to test loading)

open launchpad tickets for the issues revealed by the quicklisp diff report

the load failures report, with libraries sorted topologically by dependencies

test fresh ECL to validate this fix:

profile the reporing, to see why it’s so slow now and fix

The reason was set-exclusive-or, which takes minutes on large lists. Introduced fast-exclusive-or which uses hash tables if the arguments allow - :TEST is a designator of EQ, EQL, EQUAL or EQUALP.

document the new version of agent (how to run it, that it takes long and may be restarted)

increase the load-test timeout from 1 to 2 minutes

increase the load-test timeout from 2 to 5 minutes

Jump to Line
Something went wrong with that request. Please try again.