Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 314 lines (254 sloc) 13.702 kb
291ec27 @avodonosov Updated README.org
avodonosov authored
1 * Collaborative testing of Common Lisp libraries.
69f7014 @avodonosov Updated README.org
avodonosov authored
2 [[http://common-lisp.net/project/cl-test-grid/][Test Results]] | [[https://bugs.launchpad.net/common-lisp][Bugs Reported]]
8515dae @avodonosov Updated README.org
avodonosov authored
3
9a2e28e @avodonosov Started README.org
avodonosov authored
4 * The Goal
5 Improve stability of the Common Lisp ecosystem
6 by performing automated tests on as wide set of
bbd8084 @avodonosov Updated README.org
avodonosov authored
7 environments as possible. Environments vary
9a2e28e @avodonosov Started README.org
avodonosov authored
8 mainly in 3 dimensions:
9
10 1. Common Lisp implementations. They have incompatibilities,
11 sometimes allowed by the standard (features specified
bbd8084 @avodonosov Updated README.org
avodonosov authored
12 as implementation-dependent), sometimes due to bugs,
13 and sometimes in non-standardized features (threading,
14 sockets, FFI, etc.). Moreover, implementations are
9a2e28e @avodonosov Started README.org
avodonosov authored
15 evolving over the time, we should care about different
6b04bfc @avodonosov Working on the README
avodonosov authored
16 versions of the same implementation.
9a2e28e @avodonosov Started README.org
avodonosov authored
17 2. Growing number of CL libraries (which are also evolving
18 over the time).
19 3. Compatibility between library versions - libraries
20 depend on other libraries and work correctly
3bf4f91 @avodonosov Working on the README
avodonosov authored
21 only with particular versions of the dependencies.
9a2e28e @avodonosov Started README.org
avodonosov authored
22
3bf4f91 @avodonosov Working on the README
avodonosov authored
23 Therefore, when we run test suite of some library, we can speak
24 about success or failure only in context of given Common Lisp
25 implementation, and versions of all the dependency libraries.
9a2e28e @avodonosov Started README.org
avodonosov authored
26
3bf4f91 @avodonosov Working on the README
avodonosov authored
27 Lets call the set of libraries with specified versions a "lib-world".
28 Important example of lib-world are Quicklisp distros.
9a2e28e @avodonosov Started README.org
avodonosov authored
29
3bf4f91 @avodonosov Working on the README
avodonosov authored
30 It is hoped that constantly running tests on wide variety
9a2e28e @avodonosov Started README.org
avodonosov authored
31 of environments will help the CL community by:
32
33 1. Fast response to the library authors in case new
34 changes cause bugs on implementations not available
35 to the author. Otherwise author may receive notification
36 about the bug years after he made the change, and
d7c29b6 @avodonosov Fix typo in README.org
avodonosov authored
37 thus the cost of fixing the bug may be much higher than
3bf4f91 @avodonosov Working on the README
avodonosov authored
38 fixing it week or two after the change.
9a2e28e @avodonosov Started README.org
avodonosov authored
39 2. The same benefit for CL implementors - when they
40 release new version, run test suites of large
41 number of libraries and quickly detect
42 possible regressions in the new release.
43 3. Help to discover and maintain compatible
44 set of library versions (e.g. Quicklisp distros).
45
6b04bfc @avodonosov Working on the README
avodonosov authored
46 Limitations.
47
14edb0d @avodonosov Working on the README
avodonosov authored
48 Of course, we should understand that test sute success
6b04bfc @avodonosov Working on the README
avodonosov authored
49 does not always mean the library is workable -
50 there might be bugs which are not covered by the tests.
10eb50e @avodonosov Working on the README
avodonosov authored
51 And the other way around - failed tests not always
6b04bfc @avodonosov Working on the README
avodonosov authored
52 means the library is broken - it may be just
53 a bug in the tests themselves.
6c7b291 @avodonosov Working on the README
avodonosov authored
54 Reducing this gap increases the utility of automated testing.
6b04bfc @avodonosov Working on the README
avodonosov authored
55
bbd8084 @avodonosov Updated README.org
avodonosov authored
56 * The Implementation Idea
9a2e28e @avodonosov Started README.org
avodonosov authored
57 Everyone can run a simple command which will run tests
58 of Common Lisp libraries and upload results to
59 the central server.
60
61 That way, instead of setting up a central build farm with
62 all the possible hardware/OS/Lisp implementation combinations,
63 we provide a way for Common Lisp users to contribute
076f68d @avodonosov Typo in README.org
avodonosov authored
64 test results from their systems, and collectively
9a2e28e @avodonosov Started README.org
avodonosov authored
65 monitor the CL world. (Of course, if anyone whould want
3bf4f91 @avodonosov Working on the README
avodonosov authored
66 to setup a test farm, cl-test-grid simplifies this task too,
67 by providing required building blocks).
9a2e28e @avodonosov Started README.org
avodonosov authored
68
69 * Status
02af791 @avodonosov Updated README.org
avodonosov authored
70 We have a lisp program called test-grid-agent. User configures
3eeeb29 @avodonosov Editing the Status section in the README.org
avodonosov authored
71 it with a list of CL implementations installed on his machine,
72 and the test-grid-agent runs tests of common lisp libraries
73 on these implementations.
02af791 @avodonosov Updated README.org
avodonosov authored
74
69f7014 @avodonosov Updated README.org
avodonosov authored
75 The tests performed by agent include fresh recompilation
76 and loading of every ASDF system found in Quicklisp,
77 and also testsuites of some of the libraries.
6b04bfc @avodonosov Working on the README
avodonosov authored
78
69f7014 @avodonosov Updated README.org
avodonosov authored
79 We have so far considered 113 libraries. 56 of these libraries
80 have appropriate test suites (fully automated, no user
81 interaction is needed) which are added to the cl-test-grid.
82
3eeeb29 @avodonosov Editing the Status section in the README.org
avodonosov authored
83 Test-grid-agent may be run manually from command line or
84 configured as a periodical task with cron or similar service.
85 Each test suite is run in a separate lisp process
86 Quicklisp is used to download the libraries to be tested
48373aa @avodonosov Editing the Status section in the README.org
avodonosov authored
87 (test-grid-agent bootstraps a private quicklisp and have no
69f7014 @avodonosov Updated README.org
avodonosov authored
88 interference with a quicklisp installation user might have
3eeeb29 @avodonosov Editing the Status section in the README.org
avodonosov authored
89 on his computer). The agent remembers what lisp implementations
90 where already tested on what quicklisp distros, and doesn't repeat
91 the work it has already done.
92
69f7014 @avodonosov Updated README.org
avodonosov authored
93 The test results including:
94 - compile/load statuses of ASDF systems of every project in Quicklisp,
95 - results of the test suites (OK or a list of FAILed test cases)
96 for the tested lisp implementations / quicklisp distros
97 are uploaded to admin and then published as a plain lisp
98 data file in a separate git repository:
99 https://github.com/cl-test-grid/cl-test-grid-results.
100 Besides this, we store online the output produced by the
101 lisp processes when running test suites or compiling ASDF systems.
102 The logs are referenced from the test results lisp data.
103 This allows interested parties to navigate to the corresponding
104 log to study the failure details.
105
106 Having the pure lisp data it is easy to extract useful
107 information from it. For example, compare how two versions
108 of particular lisp implementation behave and detect
109 regressions between them; detect regressions between
110 Quicklisp distibutions, find out status of particular
111 library on all the tested lisp implementations.
112
113 Some HTML reports deomostrating this are published here:
114 [[http://common-lisp.net/project/cl-test-grid/]]. Clicking test
115 status (OK/FAIL) in any report refers to the corresponding
116 log file with the failure details.
117
118 Our current task in progress is to precisely document
119 the data collected by agent, and provide more report
120 examples demostrating how to analyze the data.
121
122 Meantime, feel free to send plain english queries
123 to the mailing list.
124
9a2e28e @avodonosov Started README.org
avodonosov authored
125 * Participation
942a737 @avodonosov Working on the README
avodonosov authored
126 ** Running tests
7a25fd5 @avodonosov clarify user instructions
avodonosov authored
127 =test-grid-agent:agent= is a lisp object able
036c427 @avodonosov Updated README.org
avodonosov authored
128 to manage test exectuion by subordinate lisp
129 implementations (executables) and submit test
130 resutls to server.
7a25fd5 @avodonosov clarify user instructions
avodonosov authored
131
132 It is created with function =test-grid-agent:make-agent=
69f7014 @avodonosov Updated README.org
avodonosov authored
133 and has 3 required configuration properties:
7a25fd5 @avodonosov clarify user instructions
avodonosov authored
134
f8371cd @avodonosov Clarified user instructions.
avodonosov authored
135 - =lisps= - Paths to the lisp implementations
136 that should be used to run tests.
7a25fd5 @avodonosov clarify user instructions
avodonosov authored
137
138 - =preferred-lisp= - The lisp implementation used when
0a2dec3 @avodonosov clarify user instructions
avodonosov authored
139 it is necessary to perform an auxiliary task
140 requiring a separte lisp process, for example
c34ffb7 @avodonosov clarify user instructions
avodonosov authored
141 downloading libraries to be tested.
7a25fd5 @avodonosov clarify user instructions
avodonosov authored
142 It is therefore desirable to specify here
143 a lisp implementation known to work reliable
144 on your platform.
145
146 - =user-email= - Your email so that we know who is contributing
147 the test results and can contact you. The
037ae28 @avodonosov clarify user instructions
avodonosov authored
148 email is also published in the test results
7a25fd5 @avodonosov clarify user instructions
avodonosov authored
149 reports so that library authors or other interested
037ae28 @avodonosov clarify user instructions
avodonosov authored
150 parties can contact you with questions about your platform.
7a25fd5 @avodonosov clarify user instructions
avodonosov authored
151 If you are strongly opposed to publish your email,
152 you can specify just some nickname here.
153
154 Function =test-grid-agent:main= runs the agent.
155
c6965e1 @avodonosov Specified more clearly the need to perform git update of agent sources.
avodonosov authored
156 It is necessary to perform =git pull= on agent sources
69f7014 @avodonosov Updated README.org
avodonosov authored
157 often.
4ba4e28 @avodonosov Specified more clearly the need to perform git update of agent sources.
avodonosov authored
158
69f7014 @avodonosov Updated README.org
avodonosov authored
159 There are template scripts demonstrating how to
8b32ea5 @avodonosov clarify user instructions
avodonosov authored
160 load, cofigure and run agent by a single commant.
63f4ce2 @avodonosov Simplify user instuctions a little.
avodonosov authored
161
69f7014 @avodonosov Updated README.org
avodonosov authored
162 Please use [[http://ccl.clozure.com/][CCL]] - it is the development platform and the only
163 lisp known to run agent successfully. The template scripts
164 assume [[http://www.quicklisp.org/beta/][Quicklisp]] is installed and added to the CCL init file.
bbd8084 @avodonosov Updated README.org
avodonosov authored
165
69f7014 @avodonosov Updated README.org
avodonosov authored
166 So, the steps:
1822343 @avodonosov Working on the README
avodonosov authored
167
5d5c6a1 @avodonosov Improving formatting in README.org
avodonosov authored
168 1. =git clone git://github.com/cl-test-grid/cl-test-grid.git=
169 2. =cd cl-test-grid=
8b32ea5 @avodonosov clarify user instructions
avodonosov authored
170 3. =cp run-agent.sh.sample run-agent.sh; chmod +x run-agent.sh=
171 4. =cp run-agent.sample.lisp run-agent.lisp=
3c4ce14 @avodonosov Simplify user instuctions a little.
avodonosov authored
172 5. Edit the /run-agent.sh/ (edit one line - the path to CCL).
c7b33ea @avodonosov Clarified user instructions.
avodonosov authored
173 6. Edit the /run-agent.lisp/ (paths to the lisp implementations, your email)
5d5c6a1 @avodonosov Improving formatting in README.org
avodonosov authored
174 7. =./run-agent.sh=
175
0d29c97 @avodonosov Improving formatting in README.org
avodonosov authored
176 Next time all you need is to just invoke =./run-agent.sh=. It will update the
5d5c6a1 @avodonosov Improving formatting in README.org
avodonosov authored
177 =cl-test-grid= from git, run tests and upload the results.
178
179 Agent keeps log files in the /cl-test-grid/work-dir/agent/logs//,
7cbc96b @avodonosov Fixed comments and the user instractions.
avodonosov authored
180 where you can control what it has done.
02af791 @avodonosov Updated README.org
avodonosov authored
181
5ca6f43 @avodonosov Updated README.org
avodonosov authored
182 Example crontab record to run agent at 10 o'clock every day:
183 #+BEGIN_SRC shell
9bcf9fa @avodonosov clarify user instructions
avodonosov authored
184 # minute hour day_of_month month day_of_week command
5ca6f43 @avodonosov Updated README.org
avodonosov authored
185 0 10 * * * cd /home/testgrid/cl-test-grid/ && ./run-agent.sh
186 #+END_SRC
793eaac @avodonosov Reflected the new way of runningn agent in the README.org
avodonosov authored
187
69f7014 @avodonosov Updated README.org
avodonosov authored
188 *** Details of what agent actually does
189
190 Simplified, the agent mode of operation may be represened
191 by the following pseudo code:
192
193 #+BEGIN_SRC common-lisp
194 (let ((current-quicklisp (update-quicklisp)))
195 (loop for lisp in my-lisp-implementations
196 (when (not (tested-already lisp current-quicklisp))
197 (let ((results-dir (complete-test-run lisp (or (find-unfinished-test-run lisp current-quicklisp)
198 (make-new-test-run lisp current-quicklisp)))))
199 (submit-results results-dir)
200 (remember-tested lisp current-quicklisp)
201 (cl-fad:delete-directory-and-files results-dir)))))
202 #+END_SRC
203
204 As you can see, the agent submits test results after
205 completing full test set on a single lisp implementation.
206
207 The code, including the internal implementaton
208 of =complete-test-run= is organized so that agent can
209 be interrupted (computer rebooted or hibernated,
210 agent process killed). When started again, it continues
211 the work from the point of interruption.
212
213 Testing single lisp implementation may take from 1-2
214 hours up to 10 hours or more (for ABCL - ABCL has long
215 startup time, which becomes significant in our use case
216 as we run every test suite or ASDF system compilation
217 in a fresh lisp process).
218
6d13d67 @avodonosov Typo in README.org
avodonosov authored
219 *Caveat of killing the agent:* if you killed the agent process
69f7014 @avodonosov Updated README.org
avodonosov authored
220 (without rebooting the machine), the subordinate process
221 running current testsute or compiling current ASDF system
222 remains alive. Typically it takes less than a minute for
223 it to finish, but sometimes it may take longer (the
224 testsuite or library compilation may require longer
225 time; or, in the worst case, test suite may hang).
226 If you start agent again, it spawns new test running
227 process, which can interfere with the old one via file
228 system (.fasl files, output logs). Therefore it's better
229 to give the old child process time to finish before
230 starting the agent again.
231
232 *** Parallel execution of multiple agents
233 Agent operates sequentially.
234
235 During its work, agent keeps it's working data in
236 a directory specified by the cofiguration property
237 - =work-dir= - Defaults to the /<cl-test-grid source code root>/work-dir/agent/
238
239 The agent takes measures to ensure there is only
240 one agent instance using this working directory.
241
07968a2 @avodonosov Typo in README.org
avodonosov authored
242 This is acheaved by using a TCP port as an inter-process
243 lock. When started, agent tries to open a socket on
69f7014 @avodonosov Updated README.org
avodonosov authored
244 the port. If it is successful, the agent continues.
245 If the port is busy, the agent deduces there is
246 another agent instance running, logs a warning
247 and exists.
248
07968a2 @avodonosov Typo in README.org
avodonosov authored
249 The port number is specified by the configuration
69f7014 @avodonosov Updated README.org
avodonosov authored
250 property
251 - =singleton-lock-port= defaults to 7685.
252
253 If you want to run several agent processes
254 and distirbute testing work between them,
255 you can assign each agent different set
256 of lisp implemenations and give each
257 agent different working directory and lock
258 port.
259
260 *** Getting assistance
261
262 Feel free to contact us if you have any questions or
263 difficulties (see the mailing list address below).
02af791 @avodonosov Updated README.org
avodonosov authored
264
265 We are looking for contributors who would agree to run
5d5c6a1 @avodonosov Improving formatting in README.org
avodonosov authored
266 =test-grid-agent= periodically (ideally once a day, but even
02af791 @avodonosov Updated README.org
avodonosov authored
267 once a month is OK).
1822343 @avodonosov Working on the README
avodonosov authored
268
6b04bfc @avodonosov Working on the README
avodonosov authored
269 ** Discussing the project
bbd8084 @avodonosov Updated README.org
avodonosov authored
270 Feedback, discussions of the approach and suggestion
6b04bfc @avodonosov Working on the README
avodonosov authored
271 for the open problems are very welcome.
272
273 Everyone interested is invited to the "mailing list" -
274 [[http://groups.google.com/group/cl-test-grid]].
9a2e28e @avodonosov Started README.org
avodonosov authored
275
6b04bfc @avodonosov Working on the README
avodonosov authored
276 Examples of the problems which need solution:
277
278 - Currently we run tests only on the quicklisp release.
279 But it is very desirable to run tests on the latest
fdd7e43 @avodonosov Working on the README
avodonosov authored
280 library versions from the source control too. For
281 example if we found a bug and the library author has
282 fixed it, he might want to issue a request to cl-test-grid
6b04bfc @avodonosov Working on the README
avodonosov authored
283 to run tests of the recent version of his library
284 on all the platforms available. This feature would
3bba583 @avodonosov Working on the README
avodonosov authored
285 also help to ensure quicklisp distro quality before
286 releasing the distro.
6b04bfc @avodonosov Working on the README
avodonosov authored
287
288 - ...
9a2e28e @avodonosov Started README.org
avodonosov authored
289
69f7014 @avodonosov Updated README.org
avodonosov authored
290 ** Adding testsuite of your library
6b04bfc @avodonosov Working on the README
avodonosov authored
291 It is quite easy - few lines of code.
292
293 Look how the library tests are started in the asdf:perform method
294 for asdf:test-op defined in the library .asd file. Then use the
bbd8084 @avodonosov Updated README.org
avodonosov authored
295 same approach to define a method test-grid::libtest eql specialized
6b04bfc @avodonosov Working on the README
avodonosov authored
296 for that library and send us this code.
297
fdd99a6 @avodonosov Working on the README
avodonosov authored
298 See examples for the already added libraries in the
0514aad @avodonosov README.org: reference to example of libtest function now points to the t...
avodonosov authored
299 [[https://github.com/cl-test-grid/cl-test-grid/blob/master/testsuites/testsuites.lisp][testsuites/testsuites.lisp]].
6b04bfc @avodonosov Working on the README
avodonosov authored
300
301 ** More
302 Lot of things may be done in this project. But the project
303 has no independent value, it is only useful if it helps
304 to improve the CL ecosystem quality. Fixing bugs in the
305 CL libraries, writing more tests is the most important.
9a2e28e @avodonosov Started README.org
avodonosov authored
306
6b04bfc @avodonosov Working on the README
avodonosov authored
307 * TODO fix in this README:
9a2e28e @avodonosov Started README.org
avodonosov authored
308 Terminology - I say "quicklisp distro", but if be precise,
309 quicklisp calls it "quicklisp distro version". But
310 if I say "lib-world is a set of libraries with specified
311 versions. An example of lib-world is a quicklisp
312 distro version" the word "version" is repeated twice
313 with diffirent sense - confusing.
Something went wrong with that request. Please try again.