Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 204 lines (164 sloc) 8.895 kb
5be7639 Anton Vodonosov first commit
avodonosov authored
1 Collaborative testing of Common Lisp libraries.
2
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
3 (This README file is a work-in-progress)
4
5 * The Goal
6 Improve stability of the Common Lisp ecosystem
7 by performing automated tests on as wide set of
8 envionments as possible. Environments vary
9 mainly in 3 dimensions:
10
11 1. Common Lisp implementations. They have incompatibilities,
12 sometimes allowed by the standard (features specified
13 as implementation-dependent), somtimes due to bugs,
14 and sometimes in non-standartized features (threading,
15 sockets, FFI, etc.). Moreover, impementations are
16 evolving over the time, we should care about different
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
17 versions of the same implementation.
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
18 2. Growing number of CL libraries (which are also evolving
19 over the time).
20 3. Compatibility between library versions - libraries
21 depend on other libraries and work correctly
3bf4f91 Anton Vodonosov Working on the README
avodonosov authored
22 only with particular versions of the dependencies.
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
23
3bf4f91 Anton Vodonosov Working on the README
avodonosov authored
24 Therefore, when we run test suite of some library, we can speak
25 about success or failure only in context of given Common Lisp
26 implementation, and versions of all the dependency libraries.
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
27
3bf4f91 Anton Vodonosov Working on the README
avodonosov authored
28 Lets call the set of libraries with specified versions a "lib-world".
29 Important example of lib-world are Quicklisp distros.
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
30
3bf4f91 Anton Vodonosov Working on the README
avodonosov authored
31 It is hoped that constantly running tests on wide variety
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
32 of environments will help the CL community by:
33
34 1. Fast response to the library authors in case new
35 changes cause bugs on implementations not available
36 to the author. Otherwise author may receive notification
37 about the bug years after he made the change, and
d7c29b6 Anton Vodonosov Fix typo in README.org
avodonosov authored
38 thus the cost of fixing the bug may be much higher than
3bf4f91 Anton Vodonosov Working on the README
avodonosov authored
39 fixing it week or two after the change.
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
40 2. The same benefit for CL implementors - when they
41 release new version, run test suites of large
42 number of libraries and quickly detect
43 possible regressions in the new release.
44 3. Help to discover and maintain compatible
45 set of library versions (e.g. Quicklisp distros).
46
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
47 Limitations.
48
14edb0d Anton Vodonosov Working on the README
avodonosov authored
49 Of course, we should understand that test sute success
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
50 does not always mean the library is workable -
51 there might be bugs which are not covered by the tests.
10eb50e Anton Vodonosov Working on the README
avodonosov authored
52 And the other way around - failed tests not always
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
53 means the library is broken - it may be just
54 a bug in the tests themselves.
6c7b291 Anton Vodonosov Working on the README
avodonosov authored
55 Reducing this gap increases the utility of automated testing.
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
56
57 * The Implementaion Idea
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
58 Everyone can run a simple command which will run tests
59 of Common Lisp libraries and upload results to
60 the central server.
61
62 That way, instead of setting up a central build farm with
63 all the possible hardware/OS/Lisp implementation combinations,
64 we provide a way for Common Lisp users to contribute
076f68d Anton Vodonosov Typo in README.org
avodonosov authored
65 test results from their systems, and collectively
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
66 monitor the CL world. (Of course, if anyone whould want
3bf4f91 Anton Vodonosov Working on the README
avodonosov authored
67 to setup a test farm, cl-test-grid simplifies this task too,
68 by providing required building blocks).
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
69
70 * Status
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
71 We have function test-grid::run-libtests wich runs tests
e5db648 Anton Vodonosov Working on the README
avodonosov authored
72 of the CL libraries. Quicklisp is used to download the libraries.
73 The test results are uploaded to the central server and
74 published as reports here:
374cd6d Anton Vodonosov Working on the README
avodonosov authored
75 [[http://common-lisp.net/project/cl-test-grid/]]. Clicking library test
76 status (OK/FAIL) in any report refers to the library test logs
77 with the failure details.
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
78
6c7b291 Anton Vodonosov Working on the README
avodonosov authored
79 Currently we have considered the top 50 of
80 [[http://blog.quicklisp.org/2010/11/project-download-statistics.html][most often quicklisp-downloaded libraries]]. 23 of these libraries
059717c Anton Vodonosov Working on the README
avodonosov authored
81 have appropriate test suites (fully automated, no user interaction is needed)
82 and are added to the cl-test-grid.
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
83
6c7b291 Anton Vodonosov Working on the README
avodonosov authored
84 File [[https://github.com/cl-test-grid/cl-test-grid/blob/master/run.sh.sample][run.sh.sample]] shows what parameters to use for particular
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
85 CL implementations so that it bootstraps quicklisp in a local
86 directory and updates it to the recent version (if necessary),
87 and then preforms test-grid::run-libtests using this local
88 quicklisp installation. The CL implementation parameters
5cec1b3 Anton Vodonosov Working on the README
avodonosov authored
89 suppress loading the standard init files of the implementaion
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
90 (to avoid interference with the quicklisp installation
91 user might have on his computer).
92
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
93 * Participation
942a737 Anton Vodonosov Working on the README
avodonosov authored
94 ** Running tests
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
95 The most appreciated way to participate.
96
1822343 Anton Vodonosov Working on the README
avodonosov authored
97 The first time you will need do the following:
98
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
99 1. git clone git://github.com/cl-test-grid/cl-test-grid.git
100 2. cd cl-test-grid
fdd99a6 Anton Vodonosov Working on the README
avodonosov authored
101 3. cp run.sh.sample run.sh; chmod +x run.sh
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
102 4. Edit the run.sh (you may need to change paths to your CL implementation(s)).
103 5. ./run.sh
104
1822343 Anton Vodonosov Working on the README
avodonosov authored
105 Next time all you need is to just start ./run.sh. It will update the
106 cl-test-grid from git, update quicklisp to the latest distro
107 (and of course run tests and upload the results).
108
109 We are looking for contributors who would agree to run
110 this command periodically, for example once a month.
111
112 Note also, that this is not 100% automated system.
113 For example uploading results uses [[http://weitz.de/drakma/][drakma]]. But if
114 drakma is broken on your planform and the upload fails,
a55d852 Anton Vodonosov Working on the README
avodonosov authored
115 you will be asked to send us the resuls manually. Or
116 if some test sute hangs, you will need to kill the
a6c104b Anton Vodonosov Working on the README
avodonosov authored
117 lisp process.
118
119 If you decided to give this project a try and run tests,
b376aba Anton Vodonosov Working on the README
avodonosov authored
120 feel free contact us, we will be glad to assist you
121 (actually it is even recommended, because the project
122 is in the early stage of the development and problems
123 are possible; and also we are interested to see
124 how it goes and how to make test running more convenient).
1822343 Anton Vodonosov Working on the README
avodonosov authored
125
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
126 ** Discussing the project
127 Feedback, discussions of the apporach and suggestesion
128 for the open problems are very welcome.
129
130 Everyone interested is invited to the "mailing list" -
131 [[http://groups.google.com/group/cl-test-grid]].
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
132
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
133 Examples of the problems which need solution:
134
135 - Currently we run tests only on the quicklisp release.
136 But it is very desirable to run tests on the latest
fdd7e43 Anton Vodonosov Working on the README
avodonosov authored
137 library versions from the source control too. For
138 example if we found a bug and the library author has
139 fixed it, he might want to issue a request to cl-test-grid
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
140 to run tests of the recent version of his library
141 on all the platforms available. This feature would
3bba583 Anton Vodonosov Working on the README
avodonosov authored
142 also help to ensure quicklisp distro quality before
143 releasing the distro.
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
144
145 - Protecting from hanging tests or crashes.
146
147 All the test sutes run in a single Lisp session.
3bba583 Anton Vodonosov Working on the README
avodonosov authored
148 If some test hangs, or crases unrecoverably
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
149 (core dump for example), it will block other tests.
150
151 Possible solution would be to run every test
152 as a separate process, but it would require
153 a portable way to start lisp process and making
3bba583 Anton Vodonosov Working on the README
avodonosov authored
154 it run particular lisp code.
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
155
f604f75 Anton Vodonosov Working on the README
avodonosov authored
156 - Distributing the work. Currently we considered
5c48854 Anton Vodonosov Working on the README
avodonosov authored
157 only 50 libraries and 23 of them have test suites
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
158 appropriate to be added to the cl-test-grid.
159 Running these 23 test sutes (including download
5c48854 Anton Vodonosov Working on the README
avodonosov authored
160 and compilation) takes 6-10 minutes on a single
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
161 CL implementation.
162
163 Quicklisp today contains almost 650 libraries.
164 Based on this, we may estimate that running
165 all the test suites available in quicklisp
166 may take around 1.5 hours on a single implementation.
167
168 This work might be distributed between several
169 contributors - everyone runs part of the tests.
170
850e1a6 Fixed a minor spelling error in the README
jiacobucci authored
171 [Today this feature has quite low priority]
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
172
173 - ...
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
174
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
175 ** Adding more libraries
176 It is quite easy - few lines of code.
177
178 Look how the library tests are started in the asdf:perform method
179 for asdf:test-op defined in the library .asd file. Then use the
180 same approach to define a method test-grid::libtest eql specilized
181 for that library and send us this code.
182
fdd99a6 Anton Vodonosov Working on the README
avodonosov authored
183 See examples for the already added libraries in the
38c88cd Anton Vodonosov Working on the README
avodonosov authored
184 [[https://github.com/cl-test-grid/cl-test-grid/blob/master/test-grid.lisp][test-grid.lisp]].
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
185
186 Today adding more libraries has low priority because we are
187 testing how cl-test-grid works, polishing corner cases
188 and reporting already found bugs to the libraries/CL implementation
189 mainteners. After we stabilize these libaries, we can add more.
190
191 ** More
192 Lot of things may be done in this project. But the project
193 has no independent value, it is only useful if it helps
194 to improve the CL ecosystem quality. Fixing bugs in the
195 CL libraries, writing more tests is the most important.
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
196
6b04bfc Anton Vodonosov Working on the README
avodonosov authored
197 * TODO fix in this README:
9a2e28e Anton Vodonosov Started README.org
avodonosov authored
198 Terminology - I say "quicklisp distro", but if be precise,
199 quicklisp calls it "quicklisp distro version". But
200 if I say "lib-world is a set of libraries with specified
201 versions. An example of lib-world is a quicklisp
202 distro version" the word "version" is repeated twice
203 with diffirent sense - confusing.
Something went wrong with that request. Please try again.