;;;; -*- Mode: org; indent-tabs-mode: nil; coding: utf-8; show-trailing-whitespace: t -*-
lisp-version-string, lib-world, author contact (get it from some settings file), date, run-duration information about test run:
organize the database file format
so that diff shows only new records. (use pprint ?) 4h format the DB equally on all the Lisps
This representation is used in the: better decision for library name representation.
- libtest method parameter (eql specialized)
- in the database and testrun datastructure.
Possible alternatives: a keyword good for READ (package independent), good for EQL specializer good for GETF when working with the database a symbol from the test-grid package
- good for EQL specializer
- package dependent in READ
- good for GETF when working with the database
- adds one more (unnecessary) way to represent a library in addition the specified for ASDF and Quicklisp
or a downcased string
- needs special handling in libtest eql specialization
- good ro READ (package independent)
- needs care to work with GETF when working with the database
A tool to generate fake DB content to test reporting
simpliest reporting to overview the libraries test statuses [6/6]
Test Runs report: every test run as a row in a table
- legend or a tooltip in the report for test statuses
- color for statuses
- use the online blob URL in the report
rows/columns to be any of quicklisp distro, lisp version library name. With grouping and sorging. A pivot -like table report of library test results, allowing
Fix rows and columns highlight in the pivot reports
google fusion tables, etc. Initial intent was to format it as a pivot for various projections (by quicklisp releases, by platform, etc). But neither google docs spreadsheet, nor google fusion table allow as to format results as we want (the main problem, it is impossible to use a custom aggregation function for pivot cells, because standard aggregation functions are numeric, but we want a report cell to represent test result(s) for a particular library, i.e. :ok, :fail, :no-resource). 5h CSV export of the database to use it then with spreadsheets,
(Common Lisp rational) can be read by spreadsheet software (MS/Open Offices, Google Spreadsheets). Test that the test-duration field value
an overview page with brief explanation of and links to all the reports
for the user who runs the tests. Spend as little efforts as possible on this task, to release quickly. 4h simple UI (command line) with guiding messages
how to use it 5h readme with explanation of the project goal and
change db format
- test run as plist (:descr <descr> :run-results <run-results>) instead of just (<descr> <run-results>)
- run-results as a list instead of plist; libname which was a plist key is now a property of the lib-result object. It is more convenient for standard mapping functions, instead of current do-lib-results.
is enough for the beginning. Result: we have 23 libraries. add more libraries: total number of 20 libraries
fails, ensure we have the error description in the output 0.5h when loading of a library or library test system
Specify an email or issue tracker of the cl-test-grid project. The “thank you” message: where exactly to submit test results?
outputs (files). An appealing way is to store it in the same git repository on github, but with the std-out files the repository will quickly grow to an unconvenient size (for new people the checkout procedure will be too long to be considered convenient) 5h Solution: files are stored in Google App Engine blob store. how to store public (central) database and failed library
framework. Ensure the libs with other test framework are all have sufficiently detailed output too. More detailed output for libraries using the RT test
framework) does not contain information about errors. Add these details. Log of the BABEL tests (generated by the Stefil test
console, meaning their output is not only standard-output and standard-error. Fix that, all the output should be in the log files, but not on console. Some libraries (babel and cl-json) stil print messages to
run the tests on all the implementations available for us.
publish the reports
test-run-info.lisp as attachment, instead of letter body
Fix lisp name sorting in pivot_lib-lisp_ql.html
to performs the tests. Describe this main command in the README (in the first paragraph). The solution: user runs (load “agent.lisp”); precondition: quicklisp installed. finalize the decision what command user runs
it is stored (and to be sure user understood us, ask him to press Enter before continuing). When the settings file is populated, inform user where
how the email will be used and ensure there is a way to avoid providing email. When asking user for email, explain him more carefully
ECL: recover from stack alexandria overflow and continute tests
ABCL: recover from cl-ppcre hung-up and continue tests
take to much space Optimize horizontal space in the reports: the headers
getf by properties: run-descr, run-results. 1h more abstract accessor to parts of DB info instead of
0.5h create a project with asdf system
0.5h DB file path based on the asdf system location
1h accumulate failed library output
without caching it in memory. It is more convenient when you are watching the testing process, you can observe the file being populated with logs (because some libraries, like flexi-streams, take time about minute to finish, and if during this minute nithing happens it is not user-friendly) run-libtests: redirect the library output to log file directly
web presentation of the TODO file will be understandable for the public. move closed TODO items to another (archive) file, so that the
Also maybe :unexpected-ok. new status, :expected-failure (or better :know-failure).
clean up the pivot reports Lisp code
automatically find regressions between quicklisp disto versions
test results detailed to individual tests for remaining test frameworks
Added 34 libraries, the total number now is 57. add more libraries
shorten the blob keys of online library logs
0.3.2 Reports Improve 1
headers are always visible and we always see what lisp/libary/lib-world corresponds to the table cell we are interested in. floating report table headers, so that when we scroll,
Yellow color for known-fail and unexpected-ok aggregated statuses
automate processing of library results submited by email
0.3.4 Test result for Quicklisp 2012-03, 2012-04
so that we can compare their results with the new quicklisp release when it is out. run tests with the newly added libraries on the quicklisp 2012-03-07,
collect the test results for the next quicklisp release (2012-04-xx)
analyze the failures of Quicklisp 2012-04.
0.4.0 Test Running Robustness - lisp-agent
write the unhandled error stacktrace (with causes) into log and exit - never enter debugger as the program should be suitable to run in background with cron. top level error handler in test-grid-agent:main
prevent starting of two agents (try to open a socket on a constant port)
check what lisps other than CCL can run lisp-agent
other words, reflect the changes during the last month) update README.org to reflect the current status and priorities (in
prepare the announcement for the mailing-list
publish the lisp-agent - merge lisp-agent to master
Give every agent and unique ID. store the ID in the agent persistence.
For example, after testing some library, the output file with library status is not found. Record :FAIL and proceed to further libraries. How to handle error during test results submission? error handling in the main process - during a test run and in other situations
Agent: when test results upload failed, do not mark this lisp as tested in persistence.
run each test suite in a separate process
by configuring asdf:output-translations to a temporary subdirectory in the test-run directory enforce the libraries source code recompilation on every test run
0.4.1 Code cleanup after finishing lisp-agent
due to hang-ups or other problems and consider removing the hardcoded failures as we now have timeouts for hang-ups. review test suites which have hard-coded failures
test grid systems will consist of modules for: push forward breaking-up the old monolitic module test-grid:
- testsuites a function libtest(libname) runs test suite of the specified library, downloading the lib from quicklisp if necessary, and returns test result in a unified format. I.e. the module knows how to invoke the library test suite and has adapters for test results of various test frameworks to the testgrid test result format.
- agent the program with can run test in child lisp processes, knows which lisps the contributor has on the machine, decides when to run tests, and can upload the test results to the central server. [the changes required are small - agent will receive some code from the current test-grid.lisp, for example test suite log header/footer printing. Maybe the test suite log will include whole lisp process output, from the lisp startup (now we start logging bit later, right before invoking the test suite)]
- admin code to receive test results and add to db [remains unchanged]
- reporting [reamains unchanged]
- blobstore [remains unchanges]
- data objects moving between other modules: test run, test results, printing for these objects; and the database
a configurable property of agent, and fs structure functions shoud accept agent as a parameter. agent fs structure functions: make the workdir root
Fixes user request by Greg Pfeil: don’t install quicklisp in ASDF source registry, https://github.com/cl-test-grid/cl-test-grid/issues/5
In future it will allow for increased speed on multiprocessor systems by starting several agent processes, each assigned different set of lisp implementations. It would require another agent config property - lock-port, for the socket whcih assures agent is a signelton. Every agent have different work-dir and different port.
Several agents may even run in different threads of a single lisp process (if global shared libraries like asdf whould allow this).
0.5.0 Separate status value for compilation failure
Important in general and a user reques by Juan Jose Garcia-Ripoll.
This allows to add even libraries without test cases to the test grid. Test status should include separate indicator for the library compilation and loading.
0.5.1 Test loading (including fresh recompilation) of every ASDF system Quicklisp.
This means even if the library doesn’t have a test suite, or the test suite is not yet integrated into the test-grid, we will know it’s compilation status.
Complications: one project may have more than one ASDF system, and in such case it’s not clear how to relate compilation failures with test suite status, if only part of ASDF systems failed to compile.
Change agent to perform load test of every ASDF system in Quicklisp
Don’t loose test run results if it was interrupted (due to errors, network outage or agent termination) Continuable test runs
rework ABCL report
make sure old reports work when DB contains ASDF systems load results
Some are missed because of agent bug: list of project was retrieved from wrong (old) quicklisp: cartesian-product-switch, cl-cheshire-cat, cl-grace, coleslaw, com.clearly-useful.iterator-protocol, com.clearly-useful.protocols, com.clearly-useful.sequence-protocol, formlets, glu-tessellate, infix-dollar-reader, lambda-lift, optima, place-modifiers, pzmq. collect results for libraries missed in Quicklisp 2012-09-09 results
bknr-datastore testsuite was not run because it was added to testgrid as bknr.datastore.
Also run bknr-datastore on quicklisp 2012-08-11.