Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Updated README.org

  • Loading branch information...
commit 69f701418b63ef9f4d87a78265c3c226847b3a0e 1 parent e413058
Anton Vodonosov avodonosov authored

Showing 1 changed file with 125 additions and 23 deletions. Show diff stats Hide diff stats

  1. +125 23 README.org
148 README.org
Source Rendered
... ... @@ -1,5 +1,5 @@
1 1 * Collaborative testing of Common Lisp libraries.
2   -[[http://common-lisp.net/project/cl-test-grid/][Test Results]] | [[https://bugs.launchpad.net/common-lisp][Bugs Found]]
  2 +[[http://common-lisp.net/project/cl-test-grid/][Test Results]] | [[https://bugs.launchpad.net/common-lisp][Bugs Reported]]
3 3
4 4 * The Goal
5 5 Improve stability of the Common Lisp ecosystem
@@ -72,37 +72,65 @@
72 72 and the test-grid-agent runs tests of common lisp libraries
73 73 on these implementations.
74 74
75   - The test results are uploaded to the central server and
76   - published as a lisp data file and as HTML reports here:
77   - [[http://common-lisp.net/project/cl-test-grid/]]. Clicking library test
78   - status (OK/FAIL) in any report refers to the library test logs
79   - with the failure details.
80   -
81   - We have so far considered 113 libraries. 56 of these libraries
82   - have appropriate test suites (fully automated, no user interaction is needed)
83   - and are added to the cl-test-grid.
  75 + The tests performed by agent include fresh recompilation
  76 + and loading of every ASDF system found in Quicklisp,
  77 + and also testsuites of some of the libraries.
84 78
  79 + We have so far considered 113 libraries. 56 of these libraries
  80 + have appropriate test suites (fully automated, no user
  81 + interaction is needed) which are added to the cl-test-grid.
  82 +
85 83 Test-grid-agent may be run manually from command line or
86 84 configured as a periodical task with cron or similar service.
87 85 Each test suite is run in a separate lisp process
88 86 Quicklisp is used to download the libraries to be tested
89 87 (test-grid-agent bootstraps a private quicklisp and have no
90   - interference with a quicklisp installation user might have
  88 + interference with a quicklisp installation user might have
91 89 on his computer). The agent remembers what lisp implementations
92 90 where already tested on what quicklisp distros, and doesn't repeat
93 91 the work it has already done.
94 92
  93 + The test results including:
  94 + - compile/load statuses of ASDF systems of every project in Quicklisp,
  95 + - results of the test suites (OK or a list of FAILed test cases)
  96 + for the tested lisp implementations / quicklisp distros
  97 + are uploaded to admin and then published as a plain lisp
  98 + data file in a separate git repository:
  99 + https://github.com/cl-test-grid/cl-test-grid-results.
  100 + Besides this, we store online the output produced by the
  101 + lisp processes when running test suites or compiling ASDF systems.
  102 + The logs are referenced from the test results lisp data.
  103 + This allows interested parties to navigate to the corresponding
  104 + log to study the failure details.
  105 +
  106 + Having the pure lisp data it is easy to extract useful
  107 + information from it. For example, compare how two versions
  108 + of particular lisp implementation behave and detect
  109 + regressions between them; detect regressions between
  110 + Quicklisp distibutions, find out status of particular
  111 + library on all the tested lisp implementations.
  112 +
  113 + Some HTML reports deomostrating this are published here:
  114 + [[http://common-lisp.net/project/cl-test-grid/]]. Clicking test
  115 + status (OK/FAIL) in any report refers to the corresponding
  116 + log file with the failure details.
  117 +
  118 + Our current task in progress is to precisely document
  119 + the data collected by agent, and provide more report
  120 + examples demostrating how to analyze the data.
  121 +
  122 + Meantime, feel free to send plain english queries
  123 + to the mailing list.
  124 +
95 125 * Participation
96 126 ** Running tests
97   - The most appreciated way to participate.
98   -
99 127 =test-grid-agent:agent= is a lisp object able
100 128 to manage test exectuion by subordinate lisp
101 129 implementations (executables) and submit test
102 130 resutls to server.
103 131
104 132 It is created with function =test-grid-agent:make-agent=
105   - and has 3 configuration properties:
  133 + and has 3 required configuration properties:
106 134
107 135 - =lisps= - Paths to the lisp implementations
108 136 that should be used to run tests.
@@ -126,16 +154,16 @@
126 154 Function =test-grid-agent:main= runs the agent.
127 155
128 156 It is necessary to perform =git pull= on agent sources
129   - often - that's how agent receives new tasks from admin.
  157 + often.
130 158
131   - We provide template scripts demonstrating how to
  159 + There are template scripts demonstrating how to
132 160 load, cofigure and run agent by a single commant.
133 161
134   - In our example we suggest to use [[http://ccl.clozure.com/][CCL]] and assume
135   - [[http://www.quicklisp.org/beta/][Quicklisp]] is installed and added to the CCL
136   - init file. (SBCL was aslo tested successfully).
  162 + Please use [[http://ccl.clozure.com/][CCL]] - it is the development platform and the only
  163 + lisp known to run agent successfully. The template scripts
  164 + assume [[http://www.quicklisp.org/beta/][Quicklisp]] is installed and added to the CCL init file.
137 165
138   - The first time you will need do these steps:
  166 + So, the steps:
139 167
140 168 1. =git clone git://github.com/cl-test-grid/cl-test-grid.git=
141 169 2. =cd cl-test-grid=
@@ -157,8 +185,82 @@
157 185 0 10 * * * cd /home/testgrid/cl-test-grid/ && ./run-agent.sh
158 186 #+END_SRC
159 187
160   - Feel free to contact us if you have any questions or difficulties
161   - (see the mailing list address below).
  188 +*** Details of what agent actually does
  189 +
  190 + Simplified, the agent mode of operation may be represened
  191 + by the following pseudo code:
  192 +
  193 +#+BEGIN_SRC common-lisp
  194 + (let ((current-quicklisp (update-quicklisp)))
  195 + (loop for lisp in my-lisp-implementations
  196 + (when (not (tested-already lisp current-quicklisp))
  197 + (let ((results-dir (complete-test-run lisp (or (find-unfinished-test-run lisp current-quicklisp)
  198 + (make-new-test-run lisp current-quicklisp)))))
  199 + (submit-results results-dir)
  200 + (remember-tested lisp current-quicklisp)
  201 + (cl-fad:delete-directory-and-files results-dir)))))
  202 +#+END_SRC
  203 +
  204 + As you can see, the agent submits test results after
  205 + completing full test set on a single lisp implementation.
  206 +
  207 + The code, including the internal implementaton
  208 + of =complete-test-run= is organized so that agent can
  209 + be interrupted (computer rebooted or hibernated,
  210 + agent process killed). When started again, it continues
  211 + the work from the point of interruption.
  212 +
  213 + Testing single lisp implementation may take from 1-2
  214 + hours up to 10 hours or more (for ABCL - ABCL has long
  215 + startup time, which becomes significant in our use case
  216 + as we run every test suite or ASDF system compilation
  217 + in a fresh lisp process).
  218 +
  219 + Caveat of killing the agent: if you killed the agent process,
  220 + (without rebooting the machine), the subordinate process
  221 + running current testsute or compiling current ASDF system
  222 + remains alive. Typically it takes less than a minute for
  223 + it to finish, but sometimes it may take longer (the
  224 + testsuite or library compilation may require longer
  225 + time; or, in the worst case, test suite may hang).
  226 + If you start agent again, it spawns new test running
  227 + process, which can interfere with the old one via file
  228 + system (.fasl files, output logs). Therefore it's better
  229 + to give the old child process time to finish before
  230 + starting the agent again.
  231 +
  232 +*** Parallel execution of multiple agents
  233 + Agent operates sequentially.
  234 +
  235 + During its work, agent keeps it's working data in
  236 + a directory specified by the cofiguration property
  237 + - =work-dir= - Defaults to the /<cl-test-grid source code root>/work-dir/agent/
  238 +
  239 + The agent takes measures to ensure there is only
  240 + one agent instance using this working directory.
  241 +
  242 + This is acheaved by using a TCP port as a inter-process
  243 + lock. When started agent tries to open a socket on
  244 + the port. If it is successful, the agent continues.
  245 + If the port is busy, the agent deduces there is
  246 + another agent instance running, logs a warning
  247 + and exists.
  248 +
  249 + The port number is specified by configuration
  250 + property
  251 + - =singleton-lock-port= defaults to 7685.
  252 +
  253 + If you want to run several agent processes
  254 + and distirbute testing work between them,
  255 + you can assign each agent different set
  256 + of lisp implemenations and give each
  257 + agent different working directory and lock
  258 + port.
  259 +
  260 +*** Getting assistance
  261 +
  262 + Feel free to contact us if you have any questions or
  263 + difficulties (see the mailing list address below).
162 264
163 265 We are looking for contributors who would agree to run
164 266 =test-grid-agent= periodically (ideally once a day, but even
@@ -185,7 +287,7 @@
185 287
186 288 - ...
187 289
188   -** Adding more libraries
  290 +** Adding testsuite of your library
189 291 It is quite easy - few lines of code.
190 292
191 293 Look how the library tests are started in the asdf:perform method

0 comments on commit 69f7014

Please sign in to comment.
Something went wrong with that request. Please try again.