Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Updated TODOs

  • Loading branch information...
commit 2d53ee6838b54c2e0945e93e14f8f2a3d93866a9 1 parent 69f7014
Anton Vodonosov avodonosov authored

Showing 2 changed files with 133 additions and 124 deletions. Show diff stats Hide diff stats

  1. +80 0 docs/TODO.done-archived.org
  2. +53 124 docs/TODO.org
80 docs/TODO.done-archived.org
Source Rendered
@@ -316,3 +316,83 @@
316 316 This allows to add even libraries without test cases to the
317 317 test grid.
318 318
  319 +* 0.5.1 Test loading (including fresh recompilation) of every ASDF system Quicklisp.
  320 + This means even if the library doesn't have a test suite,
  321 + or the test suite is not yet integrated into the test-grid,
  322 + we will know it's compilation status.
  323 +
  324 + Complications: one project may have more than one ASDF system,
  325 + and in such case it's not clear how to relate compilation
  326 + failures with test suite status, if only part of ASDF systems
  327 + failed to compile.
  328 +** DONE Change agent to perform load test of every ASDF system in Quicklisp
  329 + CLOSED: [2012-09-01 06:43]
  330 +** DONE Continuable test runs
  331 + CLOSED: [2012-09-04 Вт 02:06]
  332 + Don't loose test run results if it was interrupted (due to errors, network outage or agent termination)
  333 +** DONE Workaround the GAE issue https://code.google.com/p/googleappengine/issues/detail?id=8032
  334 + CLOSED: [2012-09-04 Вт 02:05]
  335 +** DONE rework ABCL report
  336 + CLOSED: [2012-09-14 Пт 06:39]
  337 +** DONE make sure old reports work when DB contains ASDF systems load results
  338 + CLOSED: [2012-09-10 06:43]
  339 +** DONE collect results for libraries missed in Quicklisp 2012-09-09 results
  340 + CLOSED: [2012-09-17 Пн 03:39]
  341 + Some are missed because of agent bug: list of project was retrieved
  342 + from wrong (old) quicklisp:
  343 + cartesian-product-switch, cl-cheshire-cat, cl-grace, coleslaw,
  344 + com.clearly-useful.iterator-protocol,
  345 + com.clearly-useful.protocols, com.clearly-useful.sequence-protocol,
  346 + formlets, glu-tessellate, infix-dollar-reader, lambda-lift,
  347 + optima, place-modifiers, pzmq.
  348 +
  349 + bknr-datastore testsuite was not run because it was added to testgrid
  350 + as bknr.datastore.
  351 +
  352 + Also run bknr-datastore on quicklisp 2012-08-11.
  353 +
  354 +** DONE The list of projects to test should be reteieved from the quicklisp
  355 + CLOSED: [2012-09-17 Пн 02:09]
  356 + tested by agent in child processes, but not the quicklisp used
  357 + to run the agent itself.
  358 +** DONE remove old versions of perform-test-run, run-tests, submit-logs.
  359 + CLOSED: [2012-09-17 Пн 02:09]
  360 +** DONE rework and announce new quicklisp diff report
  361 + CLOSED: [2012-09-18 Вт 00:21]
  362 + <in progress>
  363 +** DONE remove the TODO list from the bottom of perform-test-run.lisp
  364 + CLOSED: [2012-09-18 Вт 02:52]
  365 +** DONE when killing child process, sometimes we may have "No such process" simple-error
  366 + CLOSED: [2012-09-18 Вт 06:34]
  367 + if the process already finished. Handle it.
  368 +** DONE Testsuite timeout detection should sustain laptop hibernation
  369 + CLOSED: [2012-09-18 Вт 21:49]
  370 +** DONE db.lisp: deterministic formatting of load-results
  371 + CLOSED: [2012-09-19 Ср 00:21]
  372 +** DONE db.lisp: don't print testsuite values if there is no testsuite (values are NIL)
  373 + CLOSED: [2012-09-19 Ср 00:21]
  374 + don't print testing status/duration/log-key when absent
  375 + (relevant for new resuls where most of the systems don't have test suite)
  376 +** DONE db.lisp: don't print load-results when absent
  377 + CLOSED: [2012-09-19 Ср 00:21]
  378 + (only releant for old results submitted before agent become able to test loading)
  379 +** DONE open launchpad tickets for the issues revealed by the quicklisp diff report
  380 + CLOSED: [2012-09-19 Ср 01:27]
  381 +
  382 +** DONE the load failures report, with libraries sorted topologically by dependencies
  383 + CLOSED: [2012-09-25 Вт 22:10]
  384 +** DONE test fresh ECL to validate this fix:
  385 + CLOSED: [2012-09-25 Вт 06:03]
  386 + http://sourceforge.net/mailarchive/forum.php?thread_name=CANejTzrbiG71bStNg54J-sJ7QDBRwPgoOOGOWt7unrHfwMkByQ%40mail.gmail.com&forum_name=ecls-list
  387 +** DONE profile the reporing, to see why it's so slow now and fix
  388 + CLOSED: [2012-09-26 Ср 03:58]
  389 + The reason was set-exclusive-or, which takes minutes
  390 + on large lists. Introduced fast-exclusive-or
  391 + which uses hash tables if the arguments allow -
  392 + :TEST is a designator of EQ, EQL, EQUAL or EQUALP.
  393 +** DONE document the new version of agent (how to run it, that it takes long and may be restarted)
  394 + CLOSED: [2012-10-01 Пн 04:56]
  395 +** DONE increase the load-test timeout from 1 to 2 minutes
  396 + CLOSED: [2012-09-15 Сб 10:02]
  397 +** DONE increase the load-test timeout from 2 to 5 minutes
  398 + CLOSED: [2012-09-21 Пт 07:37]
177 docs/TODO.org
Source Rendered
... ... @@ -1,82 +1,14 @@
1 1 ;;;; -*- Mode: org; indent-tabs-mode: nil; coding: utf-8; show-trailing-whitespace: t -*-
2   -* 0.5.1 Test loading (including fresh recompilation) of all the libraries in quicklisp.
3   -** Test loading of every ASDF system and publish results in db.lisp.
4   - This means even if the library doesn't have a test suite,
5   - or the test suite is not yet integrated into the test-grid,
6   - we will know it's compilation status.
7   -
8   - Complications: one project may have more than one ASDF system,
9   - and in such case it's not clear how to relate compilation
10   - failures with test suite status, if only part of ASDF systems
11   - failed to compile.
12   -
13   -** DONE Change agent to perform load test of every ASDF system in Quicklisp
14   - CLOSED: [2012-09-01 06:43]
15   -** DONE Continuable test runs
16   - CLOSED: [2012-09-04 Вт 02:06]
17   - Don't loose test run results if it was interrupted (due to errors, network outage or agent termination)
18   -** DONE Workaround the GAE issue https://code.google.com/p/googleappengine/issues/detail?id=8032
19   - CLOSED: [2012-09-04 Вт 02:05]
20   -** DONE rework ABCL report
21   - CLOSED: [2012-09-14 Пт 06:39]
22   -** DONE make sure old reports work when DB contains ASDF systems load results
23   - CLOSED: [2012-09-10 06:43]
24   -** DONE collect results for libraries missed in Quicklisp 2012-09-09 results
25   - CLOSED: [2012-09-17 Пн 03:39]
26   - Some are missed because of agent bug: list of project was retrieved
27   - from wrong (old) quicklisp:
28   - cartesian-product-switch, cl-cheshire-cat, cl-grace, coleslaw,
29   - com.clearly-useful.iterator-protocol,
30   - com.clearly-useful.protocols, com.clearly-useful.sequence-protocol,
31   - formlets, glu-tessellate, infix-dollar-reader, lambda-lift,
32   - optima, place-modifiers, pzmq.
33   -
34   - bknr-datastore testsuite was not run because it was added to testgrid
35   - as bknr.datastore.
36   -
37   - Also run bknr-datastore on quicklisp 2012-08-11.
38   -
39   -** DONE The list of projects to test should be reteieved from the quicklisp
40   - CLOSED: [2012-09-17 Пн 02:09]
41   - tested by agent in child processes, but not the quicklisp used
42   - to run the agent itself.
43   -** DONE remove old versions of perform-test-run, run-tests, submit-logs.
44   - CLOSED: [2012-09-17 Пн 02:09]
45   -** DONE rework and announce new quicklisp diff report
46   - CLOSED: [2012-09-18 Вт 00:21]
47   - <in progress>
48   -** DONE remove the TODO list from the bottom of perform-test-run.lisp
49   - CLOSED: [2012-09-18 Вт 02:52]
50   -** DONE when killing child process, sometimes we may have "No such process" simple-error
51   - CLOSED: [2012-09-18 Вт 06:34]
52   - if the process already finished. Handle it.
53   -** DONE Testsuite timeout detection should sustain laptop hibernation
54   - CLOSED: [2012-09-18 Вт 21:49]
55   -** DONE db.lisp: deterministic formatting of load-results
56   - CLOSED: [2012-09-19 Ср 00:21]
57   -** DONE db.lisp: don't print testsuite values if there is no testsuite (values are NIL)
58   - CLOSED: [2012-09-19 Ср 00:21]
59   - don't print testing status/duration/log-key when absent
60   - (relevant for new resuls where most of the systems don't have test suite)
61   -** DONE db.lisp: don't print load-results when absent
62   - CLOSED: [2012-09-19 Ср 00:21]
63   - (only releant for old results submitted before agent become able to test loading)
64   -** DONE open launchpad tickets for the issues revealed by the quicklisp diff report
65   - CLOSED: [2012-09-19 Ср 01:27]
66   -
67   -** DONE the load failures report, with libraries sorted topologically by dependencies
68   - CLOSED: [2012-09-25 Вт 22:10]
69   -** DONE test fresh ECL to validate this fix:
70   - CLOSED: [2012-09-25 Вт 06:03]
71   - http://sourceforge.net/mailarchive/forum.php?thread_name=CANejTzrbiG71bStNg54J-sJ7QDBRwPgoOOGOWt7unrHfwMkByQ%40mail.gmail.com&forum_name=ecls-list
72   -** DONE profile the reporing, to see why it's so slow now and fix
73   - CLOSED: [2012-09-26 Ср 03:58]
74   - The reason was set-exclusive-or, which takes minutes
75   - on large lists. Introduced fast-exclusive-or
76   - which uses hash tables if the arguments allow -
77   - :TEST is a designator of EQ, EQL, EQUAL or EQUALP.
  2 +* 0.6.0 Reports Improve 2
  3 +** TODO library mainainer page: :report:informer:
  4 + with reports about the test statuses
  5 + for this single library on various platforms with
  6 + various quicklisp versions,
  7 + history of regressions (by quicklisp versions)
  8 +
  9 + There is some draft (uncommited) code for this task.
  10 +
78 11 ** TODO remove the :load-failed status from DB and reporting
79   -** TODO document the new version of agent (how to run it, that it takes long and may be restarted)
80 12 ** TODO terminology improvement
81 13 - db format: rename?
82 14 :libname -> :project
@@ -85,42 +17,7 @@
85 17 :log-byte-length -> :test-log-byte-length
86 18 - project name: a keyword, or string? Currently a keyword; but ASDF system names are strings.
87 19 - rename test-grid-testsuites:*all-libs* to *all-testsuites*?
88   -
89   -** DONE increase the load-test timeout from 1 to 2 minutes
90   - CLOSED: [2012-09-15 Сб 10:02]
91   -** DONE increase the load-test timeout from 2 to 5 minutes
92   - CLOSED: [2012-09-21 Пт 07:37]
93   -** TODO foreign library load errors should be recognized as :no-resource
94   - Catch CFFI conditions, and in case of ABCL, failure to load JNA classes.
95   - This should be done for load test and for testsuite run.
96   -
97   - After this is implemented, it is desirable to re-run the tests on
98   - the current and the previous Quicklisps, so that we don't redundant
99   - items in the diff report (what previous was a load failure
100   - now becomes :no-resource - not a failure).
101   -
102   -** TODO test run duration should be decreased by the time of hibernation
103   -* 0.5.4 Introduce an option to limit agent run time
104   - Use case: someone wants to run agents at night, but have
105   - the machine free from agent during dayly work.
106   - We will provide and option the user can configure in run-agent.lisp
107   - which limits agent run time so that agent exits after this duration.
108   -
109   - Before terminating agent should sumbit the results accumulated to server
110   - (even partial test runs). This is necessary becase we can not
111   - be sure the user will ever start the agent again.
112   -
113   - This also means that if next time agent is started, it should
114   - know what part of test run is completed, and continue for
115   - remaining projects (record the completenes status
116   - per library in persistence.lisp instead of whole
117   - quicklisp distros?)
118   -* 0.5.5 Install more lisps on my VPS
119   - I already have acl 8.2a express, ccl 1.8, sbcl 1.57, ecl from git, cmucl 20c
120   -** DONE ABCL
121   - CLOSED: [2012-08-20 Пн 01:48]
122   -** TODO CLISP (build a multithreaded version)
123   -* 0.6.0 Reports Improve 2
  20 +** TODO remove or wrap nicely the global variable *failures* from the dependencies-and-blockers.lisp
124 21 ** TODO document, for each CL community role, how he can befig from testgrid
125 22 - library mainaner:
126 23 - Status of your library ASDF systems loaded by various lisps
@@ -149,14 +46,6 @@
149 46 Still, the end user may:
150 47 - see what libraries work on your platform
151 48
152   -** TODO library mainainer page: :report:informer:
153   - with reports about the test statuses
154   - for this single library on various platforms with
155   - various quicklisp versions,
156   - history of regressions (by quicklisp versions)
157   -
158   - There is some draft (uncommited) code for this task.
159   -
160 49 ** TODO Prevent HTML injection via test-run-info :report:
161 50 ** TODO combine information about load failures with information about system dependencies
162 51 It will allow us to find "root" compilation blockers -
@@ -225,8 +114,7 @@
225 114 code generating this report, so that all the users
226 115 see and understand how to query data and can follow
227 116 these examples.
228   -** DONE document the data format, data access functions, reporting
229   - CLOSED: [2012-08-17 Пт 21:47]
  117 +** TODO document the data format, data access functions, reporting
230 118 functions and utilities.
231 119 ** TODO the diff report (aka regressions report) - diff along various dimentions
232 120 Currently we can only find differences between test statuses
@@ -281,6 +169,11 @@
281 169 Do we really need WEB UI for filters? If so, it
282 170 is low priority anyway.
283 171
  172 +* 0.6.1 Install more lisps on my VPS
  173 + I already have acl 8.2a express, ccl 1.8, sbcl 1.57, ecl from git, cmucl 20c
  174 +** DONE ABCL
  175 + CLOSED: [2012-08-20 Пн 01:48]
  176 +** TODO CLISP (build a multithreaded version)
284 177 * 0.7.0 Add more test suites
285 178 ** TODO add testsuites of more libraries from quicklisp
286 179
@@ -373,7 +266,43 @@
373 266
374 267 Although, for the most important libraries which don't have
375 268 enough maintainers we could provide implementation.
376   -* lisp-agent postponed issues
  269 +* agent postponed issues
  270 +** TODO Introduce an option to limit agent run time
  271 + Use case: someone wants to run agents at night, but have
  272 + the machine free from agent during dayly work.
  273 + We will provide and option the user can configure in run-agent.lisp
  274 + which limits agent run time so that agent exits after this duration.
  275 +
  276 + Before terminating agent should sumbit the results accumulated to server
  277 + (even partial test runs). This is necessary becase we can not
  278 + be sure the user will ever start the agent again.
  279 +
  280 + This also means that if next time agent is started, it should
  281 + know what part of test run is completed, and continue for
  282 + remaining projects (record the completenes status
  283 + per library in persistence.lisp instead of whole
  284 + quicklisp distros?)
  285 +** TODO A "quit" command for soft termination of the agent instead of killing it
  286 + Will ensure agent finishes only afther all his child processes
  287 + are finished, so that starting agent again is safe.
  288 +
  289 + How the command should be sent to agent? Via web interface?
  290 + Lisp command from REPL?
  291 +
  292 +** TODO foreign library load errors should be recognized and represented
  293 + in test results (maybe as :no-resource status, but it's better
  294 + to have a special status, including the library name. that way
  295 + we will have up to date list of foreign libraries necessary
  296 + to have quicklisp fully working).
  297 +
  298 + Catch CFFI conditions, and in case of ABCL, failure to load JNA classes.
  299 + This should be done for load test and for testsuite run.
  300 +
  301 + After this is implemented, it is desirable to re-run the tests on
  302 + the current and the previous Quicklisps, so that we don't redundant
  303 + items in the diff report (what previous was a load failure
  304 + now becomes :no-resource - not a failure).
  305 +** TODO test run duration should be decreased by the time of hibernation
377 306 ** TODO memoization of the implementation-identifier generic function is not portable
378 307 in particular, it's known that fare-memoization can't memoize it on ACL,
379 308 see https://groups.google.com/d/topic/cl-test-grid/Vnl3kHJbQ0M/discussion

0 comments on commit 2d53ee6

Please sign in to comment.
Something went wrong with that request. Please try again.