Skip to content
Newer
Older
100644 253 lines (174 sloc) 7.58 KB
8ee1ad7 @allisonrandal [cage] Updating copyright in whole repository to Parrot Foundation.
allisonrandal authored Feb 17, 2009
1 # Copyright (C) 2001-2007, Parrot Foundation.
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
2 # $Id$
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored Aug 27, 2002
3
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
4 =head1 NAME
5
6 docs/tests.pod - Testing Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
7
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
8 =head1 A basic guide to writing and running tests for Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
9
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
10 This is quick and dirty pointer to how the Parrot test suite is executed and
11 to how new tests for Parrot should be written.
12 The testing system is liable to change in the future, but tests written
13 following the guidelines below should be easy to port into a new test suite.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
14
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
15 =head1 How to test parrot
16
17 The easy way to test parrot is running C<make test>. If you have
18 updated your code recently and tests began failing, go for a C<make
19 realclean> and recompile parrot before complaining.
20
21 If your architecture supports JIT, you can test parrot JIT engine using C<make
22 testj>. It works just like C<make test>, but uses the JIT engine when possible.
23
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored Dec 12, 2005
24 C<make languages-test> runs the test suite for most language implementations
25 in the languages directory.
26
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored Dec 21, 2008
27 =head2 Submitting smolder test results
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
28
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored Dec 21, 2008
29 Parrot has a status page with smoke test results at
30 L<http://smolder.plusthree.com/app/public_projects/details/8>.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
31
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored Dec 21, 2008
32 You can supply new tests results by just running C<make smoke>.
33 It will run the same tests as C<make test> would, but will upload
34 the test results to the website.
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored Dec 12, 2005
35
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
36 =head1 Location of the test files
37
38 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
39 A quick overview over the subdirs in F<t> can be found in F<t/README>.
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored Feb 26, 2004
40
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
41 The language implementations usually have their test files in F<languages/*/t>.
42
43 New tests should be added to an existing F<*.t> file.
44 If a previously untested feature is tested,
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
45 it might also make sense to create a new F<*.t> file.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
46
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
47 =head1 How to write a test
48
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored May 21, 2007
49 Test scripts must emit text that conforms to the C<Test Anything Protocol>.
50 Test scripts are currently usually written in Perl 5 or PIR.
51 The Perl 5 module C<Parrot::Test>
52 and the PIR module C<Test;More> help with writing tests.
53
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
54 The testing framework needs to know how many tests it should expect. So the
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored Mar 21, 2006
55 number of planned tests needs to be incremented when adding a new test. This
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
56 is done near the top of a test file, in a line that looks like:
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored Mar 19, 2002
57
58 use Parrot::Test tests => 8;
59
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored May 21, 2007
60 for Perl 5 based test scripts.
61
62 =head2 Testing Parrot Assembler
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
63
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
64 PASM tests are mostly used for testing ops. Appropriate test files for basic
976ec68 @allisonrandal [doc] More conversions for PMC backronym.
allisonrandal authored Feb 9, 2009
65 ops are F<t/op/*.t>. Polymorphic Containers are tested in F<t/pmc/*.t>. Add the
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
66 new test like this:
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
67
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored Mar 4, 2005
68 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
69 *** a big chunk of assembler, eg:
70 print 1
71 print "\n" # you can even comment it if it's obscure
72 end # don't forget this...!
73 CODE
74 *** what you expect the output of the chunk to be, eg.
75 1
76 OUTPUT
77
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored May 21, 2007
78 =head2 Testing Parrot Intermediate Representation
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
79
67c929f @ambs While discussion continues on p2, pod files should be at most
ambs authored Aug 22, 2006
80 Writing tests in B<PIR> is more convenient. This is done with
81 C<pir_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
82
83 pir_output_is(<<'CODE',<<'OUT','nothing useful');
3721ccc [PATCH] @directive -> :directive - part 2 src, docs, tools
Leopold Toetsch authored Oct 11, 2005
84 .sub main :main
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
85 print "hi\n"
86 .end
87 CODE
88 hi
89 OUT
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored Feb 26, 2004
90
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored May 21, 2007
91 =head2 Testing C source
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
92
ea0630c @jhoblitt reformat all Pod files under docs with podtidy (modified to also remo…
jhoblitt authored Oct 5, 2005
93 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750 @jhoblitt fix verbatim block in docs/tests.pod
jhoblitt authored Oct 5, 2005
94
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
95 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
96 #include <stdio.h>
97 #include "parrot/parrot.h"
98 #include "parrot/embed.h"
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
99
100 static opcode_t *the_test(Parrot_Interp, opcode_t *, opcode_t *);
101
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
102 int main(int argc, char* argv[]) {
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored Jun 29, 2007
103 Parrot_Interp interp;
01f4e37 Updated doc and some code to reflect new Parrot_new() interface
Jens Rieks authored Sep 16, 2004
104 interpreter = Parrot_new(NULL);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
105
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
106 if (!interpreter)
a25ee79 @bschmalhofer Replace a couple of calls to
bschmalhofer authored Mar 29, 2006
107 return 1;
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
108
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored Jun 29, 2007
109 Parrot_run_native(interp, the_test);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
110 printf("done\n");
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
111 fflush(stdout);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
112 return 0;
113 }
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
114
115 static opcode_t*
f376029 @Infinoid Apply patch from jimmy++ in TT #111.
Infinoid authored Jan 1, 2009
116 the_test(PARROT_INTERP,
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
117 opcode_t *cur_op, opcode_t *start)
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
118 {
119 /* Your test goes here. */
120
121 return NULL; /* always return NULL */
122 }
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
123 CODE
124 # Anything that might be output prior to "done".
125 done
126 OUTPUT
127
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
128 Note that it's always a good idea to output "done" to confirm that the compiled
9f9cea2 @allisonrandal [pdd22io] Merging the pdd22io_part2 branch into trunk for r32922 to r…
allisonrandal authored Dec 9, 2008
129 code executed completely. When mixing C<printf> and C<Parrot_io_printf> always append
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
130 a C<fflush(stdout);> after the former.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
131
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored May 21, 2007
132 =head2 Testing Perl5 components
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored Mar 14, 2006
133
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored May 21, 2007
134 At the present time most, if not all, of the programs used to configure, build
135 and install Parrot are written in Perl 5. These programs take the form of
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to tes…
jkeenan authored May 22, 2007
136 program files (F<*.pl>) and Perl modules (F<*.pm>) holding subroutines and
137 other variables imported into the program files. Examples of such
138 program files can be found under F<tools/>; examples of such Perl modules
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored May 21, 2007
139 can be found under F<lib/Parrot/>.
140
141 All of these Perl 5 components ought to be tested. Fortunately, over the last
142 decade, under the leadership of Michael Schwern, chromatic, Andy Lester and
143 many others, the Perl 5 community has developed a rigorous approach to testing
144 in which:
145
146 =over 4
147
148 =item a
149
150 Subroutines found in F<*.pl> files are extracted and placed in F<*.pm>
151 modules.
152
153 =item b
154
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to tes…
jkeenan authored May 22, 2007
155 Those subroutines are then imported back into the program file.
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored May 21, 2007
156
157 =item c
158
159 Those subroutines are also imported into test files (F<*.t>) where are tests
160 are run by Test::Builder-based modules such as Test::Simple and Test::More.
161
162 =item d
163
164 Those test files are run by Test::Harness-based functionality such as
165 ExtUtils::MakeMaker's F<make test>, Module::Build's F<build test>, or
166 Test::Harness's F<prove>.
167
168 =item e
169
170 The extent to which the test files exercise all statements in the Perl modules
171 being tested is measured in coverage analysis using CPAN module Devel::Cover.
172
173 =item f
174
175 The underlying code is refactored and improved on the basis of the results of
176 tests and coverage analysis.
177
178 =back
179
180 Tests reflecting this approach can be found in F<t/configure/>,
181 F<t/postconfigure/>, F<t/tools/>, and so on.
182
183 It is our objective to test all Perl 5 components of the Parrot distribution
184 using the methodology above.
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored Mar 14, 2006
185
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test …
simoncozens authored Jan 10, 2008
186 =head3 Build Tools Tests
187
188 The files in F<t/postconfigure> are tests for build system. The build tools
189 tests are intended to be run after someone has made changes in modules such as
190 F<lib/Parrot/Pmc2cUtils/>, F<Ops2cUtils/> and F<Ops2pmutils/>. They're set up
191 to be run after F<Configure.pl> has completed but before make has been invoked.
192 (In fact, they will generate errors if make has completed.) You can run them
193 with any of the following:
194
195 perl Configure.pl --test
196 perl Configure.pl --test=build
197 make buildtools_tests (following Configure.pl)
198
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
199 =head2 Testing language implementations
200
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored Mar 21, 2006
201 Language implementations are usually tested with
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored May 21, 2007
202 C<language_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
203
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored Feb 26, 2004
204 =head1 Ideal tests:
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
205
206 =over 4
207
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored Dec 4, 2005
208 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
209
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
210 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
211 code they're testing. These should include potentially out of band input
212 unless we decide that compilers should check for this themselves.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
213
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored Dec 4, 2005
214 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
215
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
216 Are small and self contained, so that if the tested feature breaks we can
217 identify where and why quickly.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
218
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored Dec 4, 2005
219 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
220
19b32a9 Minor edits
Simon Glover authored May 14, 2003
221 Are valid. Essentially, they should conform to the additional documentation
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
222 that accompanies the feature (if any). [If there isn't any documentation, then
223 feel free to add some and/or complain to the mailing list].
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
224
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored Dec 4, 2005
225 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
226
227 Are a chunk of assembler and a chunk of expected output.
228
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored Aug 27, 2002
229 =back
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
230
231 =head1 TODO tests
232
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
233 In test driven development, tests are implemented first. So the tests are
234 initially expected to fail. This can be expressed by marking the tests as
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
235 TODO. See L<Test::More> on how to do that.
236
237 =head1 SKIP tests
238
239 TODO test actually executed, so that unexpected success can be detected.
240 In the case of missing requirements and in the case of serious breakdowns
241 the execution of tests can be skipped.
242 See L<Test::More> on how to do that.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
243
244 =head1 SEE ALSO
245
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
246 L<http://qa.perl.org/>
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored May 21, 2007
247 L<http://testanything.org/>
248 L<http://en.wikipedia.org/wiki/Test_Anything_Protocol>
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored Mar 14, 2006
249 F<t/TESTS.STATUS.pod>
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored Mar 21, 2006
250 F<t/README>
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
251
252 =cut
Something went wrong with that request. Please try again.