Skip to content
This repository
Newer
Older
100644 260 lines (180 sloc) 8.051 kb
89a54fe5 » bschmalhofer
2007-05-21 More elaboration in doc/tests.pod.
1 # Copyright (C) 2001-2007, The Perl Foundation.
9d8cce43 » Michael Scott
2004-02-28 a bit more tidying up
2 # $Id$
9b4e5d00 » Dan Sugalski
2002-08-27 Fixups to make parrotcode.org's autogenerated pages happier
3
9d8cce43 » Michael Scott
2004-02-28 a bit more tidying up
4 =head1 NAME
5
6 docs/tests.pod - Testing Parrot
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
7
e48bb2e0 » chromatic
2005-12-12 Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
8 =head1 A basic guide to writing and running tests for Parrot
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
9
aab50225 » bschmalhofer
2005-12-28 docs: Going over docs/tests.pod.
10 This is quick and dirty pointer to how the Parrot test suite is executed and
11 to how new tests for Parrot should be written.
12 The testing system is liable to change in the future, but tests written
13 following the guidelines below should be easy to port into a new test suite.
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
14
e48bb2e0 » chromatic
2005-12-12 Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
15 =head1 How to test parrot
16
17 The easy way to test parrot is running C<make test>. If you have
18 updated your code recently and tests began failing, go for a C<make
19 realclean> and recompile parrot before complaining.
20
21 If your architecture supports JIT, you can test parrot JIT engine using C<make
22 testj>. It works just like C<make test>, but uses the JIT engine when possible.
23
c8ca9f07 » bschmalhofer
2005-12-12 Mention the make targets 'languages-test' and 'languages-smoke'
24 C<make languages-test> runs the test suite for most language implementations
25 in the languages directory.
26
e48bb2e0 » chromatic
2005-12-12 Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
27 =head2 Submitting smoke test results
28
29 Parrot has a status page with smoke test results
38d4ce0a » coke
2005-12-12 Fix the URL to the smoke server.
30 L<http://smoke.parrotcode.org/smoke/>. You can supply new tests
c138099a » bschmalhofer
2006-03-21 Move Parrot_Docs.t from t/docs to t/perl, as
31 results by just running C<make smoke>. It will run the same tests as
32 C<make test> would, but will additionally create a HTML table with the test
e48bb2e0 » chromatic
2005-12-12 Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
33 results. At the end, it will try to upload the test results to the
34 smoke server.
35
36 It is also possible to run a smoke test on JIT. For that, try running
87105637 » jhoblitt
2005-12-13 add smokej jit smoke test target
37 C<make smokej>.
e48bb2e0 » chromatic
2005-12-12 Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
38
c8ca9f07 » bschmalhofer
2005-12-12 Mention the make targets 'languages-test' and 'languages-smoke'
39 C<make languages-smoke> does smoke testing for most language implementations
40 in the languages directory.
41
aab50225 » bschmalhofer
2005-12-28 docs: Going over docs/tests.pod.
42 =head1 Location of the test files
43
44 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
45 A quick overview over the subdirs in F<t> can be found in F<t/README>.
1acea255 » Leopold Toetsch
2004-02-26 update info on pir tests in main; don't generate packfile-perl.pod
46
aab50225 » bschmalhofer
2005-12-28 docs: Going over docs/tests.pod.
47 The language implementations usually have their test files in F<languages/*/t>.
48
49 New tests should be added to an existing F<*.t> file.
50 If a previously untested feature is tested,
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
51 it might also make sense to create a new F<*.t> file.
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
52
aab50225 » bschmalhofer
2005-12-28 docs: Going over docs/tests.pod.
53 =head1 How to write a test
54
89a54fe5 » bschmalhofer
2007-05-21 More elaboration in doc/tests.pod.
55 Test scripts must emit text that conforms to the C<Test Anything Protocol>.
56 Test scripts are currently usually written in Perl 5 or PIR.
57 The Perl 5 module C<Parrot::Test>
58 and the PIR module C<Test;More> help with writing tests.
59
05a7e7bd » bschmalhofer
2005-09-07 This patch is huge because of all the whitespace reformatting. This was
60 The testing framework needs to know how many tests it should expect. So the
c138099a » bschmalhofer
2006-03-21 Move Parrot_Docs.t from t/docs to t/perl, as
61 number of planned tests needs to be incremented when adding a new test. This
05a7e7bd » bschmalhofer
2005-09-07 This patch is huge because of all the whitespace reformatting. This was
62 is done near the top of a test file, in a line that looks like:
553e254a » Nicholas Clark
2002-03-19 Patch from Simon Glover <scog@amnh.org>:
63
64 use Parrot::Test tests => 8;
65
89a54fe5 » bschmalhofer
2007-05-21 More elaboration in doc/tests.pod.
66 for Perl 5 based test scripts.
67
68 =head2 Testing Parrot Assembler
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
69
e48bb2e0 » chromatic
2005-12-12 Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
70 PASM tests are mostly used for testing ops. Appropriate test files for basic
05a7e7bd » bschmalhofer
2005-09-07 This patch is huge because of all the whitespace reformatting. This was
71 ops are F<t/op/*.t>. Perl Magic Cookies are tested in F<t/pmc/*.t>. Add the
72 new test like this:
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
73
b38c8aa8 » bschmalhofer
2005-03-04 Some fiddling with test suite:
74 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
75 *** a big chunk of assembler, eg:
76 print 1
77 print "\n" # you can even comment it if it's obscure
78 end # don't forget this...!
79 CODE
80 *** what you expect the output of the chunk to be, eg.
81 1
82 OUTPUT
83
89a54fe5 » bschmalhofer
2007-05-21 More elaboration in doc/tests.pod.
84 =head2 Testing Parrot Intermediate Representation
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
85
67c929f9 » ambs
2006-08-22 While discussion continues on p2, pod files should be at most
86 Writing tests in B<PIR> is more convenient. This is done with
87 C<pir_output_is> and friends.
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
88
89 pir_output_is(<<'CODE',<<'OUT','nothing useful');
eb9e7dd6 » particle
2005-12-28 #37520: [TODO] rename library files from .imc to .pir
90 .include 'library/config.pir'
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
91
3721ccc4 » Leopold Toetsch
2005-10-11 [PATCH] @directive -> :directive - part 2 src, docs, tools
92 .sub main :main
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
93 print "hi\n"
94 .end
95 CODE
96 hi
97 OUT
1acea255 » Leopold Toetsch
2004-02-26 update info on pir tests in main; don't generate packfile-perl.pod
98
89a54fe5 » bschmalhofer
2007-05-21 More elaboration in doc/tests.pod.
99 =head2 Testing C source
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
100
ea0630c1 » jhoblitt
2005-10-05 reformat all Pod files under docs with podtidy (modified to also remo…
101 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750b » jhoblitt
2005-10-05 fix verbatim block in docs/tests.pod
102
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
103 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
104 #include <stdio.h>
105 #include "parrot/parrot.h"
106 #include "parrot/embed.h"
58c0aa8c » Leopold Toetsch
2004-02-26 update info on src tests
107
108 static opcode_t *the_test(Parrot_Interp, opcode_t *, opcode_t *);
109
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
110 int main(int argc, char* argv[]) {
6de588fd » paultcochrane
2007-06-29 [docs] Instances of 'interpreter' changed to 'interp' as per convention
111 Parrot_Interp interp;
01f4e37a » Jens Rieks
2004-09-16 Updated doc and some code to reflect new Parrot_new() interface
112 interpreter = Parrot_new(NULL);
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
113
58c0aa8c » Leopold Toetsch
2004-02-26 update info on src tests
114 if (!interpreter)
a25ee79a » bschmalhofer
2006-03-29 Replace a couple of calls to
115 return 1;
58c0aa8c » Leopold Toetsch
2004-02-26 update info on src tests
116
6de588fd » paultcochrane
2007-06-29 [docs] Instances of 'interpreter' changed to 'interp' as per convention
117 Parrot_run_native(interp, the_test);
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
118 printf("done\n");
9d8cce43 » Michael Scott
2004-02-28 a bit more tidying up
119 fflush(stdout);
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
120 return 0;
121 }
58c0aa8c » Leopold Toetsch
2004-02-26 update info on src tests
122
123 static opcode_t*
6de588fd » paultcochrane
2007-06-29 [docs] Instances of 'interpreter' changed to 'interp' as per convention
124 the_test(Parrot_Interp interp,
9d8cce43 » Michael Scott
2004-02-28 a bit more tidying up
125 opcode_t *cur_op, opcode_t *start)
58c0aa8c » Leopold Toetsch
2004-02-26 update info on src tests
126 {
127 /* Your test goes here. */
128
129 return NULL; /* always return NULL */
130 }
c543d6fa » boemmels
2003-09-30 Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
131 CODE
132 # Anything that might be output prior to "done".
133 done
134 OUTPUT
135
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
136 Note that it's always a good idea to output "done" to confirm that the compiled
05a7e7bd » bschmalhofer
2005-09-07 This patch is huge because of all the whitespace reformatting. This was
137 code executed completely. When mixing C<printf> and C<PIO_printf> always append
138 a C<fflush(stdout);> after the former.
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
139
89a54fe5 » bschmalhofer
2007-05-21 More elaboration in doc/tests.pod.
140 =head2 Testing Perl5 components
b9beb9d3 » bschmalhofer
2006-03-14 Explain what t/perl is for.
141
7cc48021 » bschmalhofer
2007-05-21 #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
142 At the present time most, if not all, of the programs used to configure, build
143 and install Parrot are written in Perl 5. These programs take the form of
c13cb724 » jkeenan
2007-05-22 Following further discussion in RT 43008: In sections relating to tes…
144 program files (F<*.pl>) and Perl modules (F<*.pm>) holding subroutines and
145 other variables imported into the program files. Examples of such
146 program files can be found under F<tools/>; examples of such Perl modules
7cc48021 » bschmalhofer
2007-05-21 #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
147 can be found under F<lib/Parrot/>.
148
149 All of these Perl 5 components ought to be tested. Fortunately, over the last
150 decade, under the leadership of Michael Schwern, chromatic, Andy Lester and
151 many others, the Perl 5 community has developed a rigorous approach to testing
152 in which:
153
154 =over 4
155
156 =item a
157
158 Subroutines found in F<*.pl> files are extracted and placed in F<*.pm>
159 modules.
160
161 =item b
162
c13cb724 » jkeenan
2007-05-22 Following further discussion in RT 43008: In sections relating to tes…
163 Those subroutines are then imported back into the program file.
7cc48021 » bschmalhofer
2007-05-21 #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
164
165 =item c
166
167 Those subroutines are also imported into test files (F<*.t>) where are tests
168 are run by Test::Builder-based modules such as Test::Simple and Test::More.
169
170 =item d
171
172 Those test files are run by Test::Harness-based functionality such as
173 ExtUtils::MakeMaker's F<make test>, Module::Build's F<build test>, or
174 Test::Harness's F<prove>.
175
176 =item e
177
178 The extent to which the test files exercise all statements in the Perl modules
179 being tested is measured in coverage analysis using CPAN module Devel::Cover.
180
181 =item f
182
183 The underlying code is refactored and improved on the basis of the results of
184 tests and coverage analysis.
185
186 =back
187
188 Tests reflecting this approach can be found in F<t/configure/>,
189 F<t/postconfigure/>, F<t/tools/>, and so on.
190
191 It is our objective to test all Perl 5 components of the Parrot distribution
192 using the methodology above.
b9beb9d3 » bschmalhofer
2006-03-14 Explain what t/perl is for.
193
c4608d5c » simoncozens
2008-01-10 [BUILD] Missing opcodes should be an error - (#48497) fix build test …
194 =head3 Build Tools Tests
195
196 The files in F<t/postconfigure> are tests for build system. The build tools
197 tests are intended to be run after someone has made changes in modules such as
198 F<lib/Parrot/Pmc2cUtils/>, F<Ops2cUtils/> and F<Ops2pmutils/>. They're set up
199 to be run after F<Configure.pl> has completed but before make has been invoked.
200 (In fact, they will generate errors if make has completed.) You can run them
201 with any of the following:
202
203 perl Configure.pl --test
204 perl Configure.pl --test=build
205 make buildtools_tests (following Configure.pl)
206
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
207 =head2 Testing language implementations
208
c138099a » bschmalhofer
2006-03-21 Move Parrot_Docs.t from t/docs to t/perl, as
209 Language implementations are usually tested with
89a54fe5 » bschmalhofer
2007-05-21 More elaboration in doc/tests.pod.
210 C<language_output_is> and friends.
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
211
1acea255 » Leopold Toetsch
2004-02-26 update info on pir tests in main; don't generate packfile-perl.pod
212 =head1 Ideal tests:
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
213
214 =over 4
215
afc981d4 » chromatic
2005-12-04 Fixed some POD formatting errors.
216 =item *
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
217
05a7e7bd » bschmalhofer
2005-09-07 This patch is huge because of all the whitespace reformatting. This was
218 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
219 code they're testing. These should include potentially out of band input
220 unless we decide that compilers should check for this themselves.
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
221
afc981d4 » chromatic
2005-12-04 Fixed some POD formatting errors.
222 =item *
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
223
05a7e7bd » bschmalhofer
2005-09-07 This patch is huge because of all the whitespace reformatting. This was
224 Are small and self contained, so that if the tested feature breaks we can
225 identify where and why quickly.
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
226
afc981d4 » chromatic
2005-12-04 Fixed some POD formatting errors.
227 =item *
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
228
19b32a9c » Simon Glover
2003-05-14 Minor edits
229 Are valid. Essentially, they should conform to the additional documentation
05a7e7bd » bschmalhofer
2005-09-07 This patch is huge because of all the whitespace reformatting. This was
230 that accompanies the feature (if any). [If there isn't any documentation, then
231 feel free to add some and/or complain to the mailing list].
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
232
afc981d4 » chromatic
2005-12-04 Fixed some POD formatting errors.
233 =item *
8c4a44c5 » quidity
2001-11-07 This is a basic quide to writing tests with assembler.
234
235 Are a chunk of assembler and a chunk of expected output.
236
9b4e5d00 » Dan Sugalski
2002-08-27 Fixups to make parrotcode.org's autogenerated pages happier
237 =back
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
238
239 =head1 TODO tests
240
05a7e7bd » bschmalhofer
2005-09-07 This patch is huge because of all the whitespace reformatting. This was
241 In test driven development, tests are implemented first. So the tests are
242 initially expected to fail. This can be expressed by marking the tests as
aab50225 » bschmalhofer
2005-12-28 docs: Going over docs/tests.pod.
243 TODO. See L<Test::More> on how to do that.
244
245 =head1 SKIP tests
246
247 TODO test actually executed, so that unexpected success can be detected.
248 In the case of missing requirements and in the case of serious breakdowns
249 the execution of tests can be skipped.
250 See L<Test::More> on how to do that.
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
251
252 =head1 SEE ALSO
253
e48bb2e0 » chromatic
2005-12-12 Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
254 L<http://qa.perl.org/>
89a54fe5 » bschmalhofer
2007-05-21 More elaboration in doc/tests.pod.
255 L<http://testanything.org/>
256 L<http://en.wikipedia.org/wiki/Test_Anything_Protocol>
b9beb9d3 » bschmalhofer
2006-03-14 Explain what t/perl is for.
257 F<t/TESTS.STATUS.pod>
c138099a » bschmalhofer
2006-03-21 Move Parrot_Docs.t from t/docs to t/perl, as
258 F<t/README>
a754c8cd » bschmalhofer
2005-04-06 Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
259
260 =cut
Something went wrong with that request. Please try again.