Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 250 lines (172 sloc) 7.603 kB
8ee1ad7 @allisonrandal [cage] Updating copyright in whole repository to Parrot Foundation.
allisonrandal authored
1 # Copyright (C) 2001-2007, Parrot Foundation.
9d8cce4 a bit more tidying up
Michael Scott authored
2 # $Id$
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
3
9d8cce4 a bit more tidying up
Michael Scott authored
4 =head1 NAME
5
6 docs/tests.pod - Testing Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
7
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
8 =head1 A basic guide to writing and running tests for Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
9
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
10 This is quick and dirty pointer to how the Parrot test suite is executed and
11 to how new tests for Parrot should be written.
12 The testing system is liable to change in the future, but tests written
13 following the guidelines below should be easy to port into a new test suite.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
14
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
15 =head1 How to test parrot
16
17 The easy way to test parrot is running C<make test>. If you have
18 updated your code recently and tests began failing, go for a C<make
19 realclean> and recompile parrot before complaining.
20
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
21 C<make languages-test> runs the test suite for most language implementations
22 in the languages directory.
23
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
24 =head2 Submitting smolder test results
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
25
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
26 Parrot has a status page with smoke test results at
27 L<http://smolder.plusthree.com/app/public_projects/details/8>.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
28
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
29 You can supply new tests results by just running C<make smoke>.
30 It will run the same tests as C<make test> would, but will upload
31 the test results to the website.
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
32
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
33 =head1 Location of the test files
34
35 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
36 A quick overview over the subdirs in F<t> can be found in F<t/README>.
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
37
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
38 The language implementations usually have their test files in F<languages/*/t>.
39
40 New tests should be added to an existing F<*.t> file.
41 If a previously untested feature is tested,
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
42 it might also make sense to create a new F<*.t> file.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
43
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
44 =head1 How to write a test
45
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
46 Test scripts must emit text that conforms to the C<Test Anything Protocol>.
47 Test scripts are currently usually written in Perl 5 or PIR.
48 The Perl 5 module C<Parrot::Test>
49 and the PIR module C<Test;More> help with writing tests.
50
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
51 The testing framework needs to know how many tests it should expect. So the
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
52 number of planned tests needs to be incremented when adding a new test. This
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
53 is done near the top of a test file, in a line that looks like:
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
54
55 use Parrot::Test tests => 8;
56
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
57 for Perl 5 based test scripts.
58
59 =head2 Testing Parrot Assembler
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
60
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
61 PASM tests are mostly used for testing ops. Appropriate test files for basic
976ec68 @allisonrandal [doc] More conversions for PMC backronym.
allisonrandal authored
62 ops are F<t/op/*.t>. Polymorphic Containers are tested in F<t/pmc/*.t>. Add the
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
63 new test like this:
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
64
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored
65 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
66 *** a big chunk of assembler, eg:
67 print 1
68 print "\n" # you can even comment it if it's obscure
69 end # don't forget this...!
70 CODE
71 *** what you expect the output of the chunk to be, eg.
72 1
73 OUTPUT
74
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
75 =head2 Testing Parrot Intermediate Representation
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
76
67c929f @ambs While discussion continues on p2, pod files should be at most
ambs authored
77 Writing tests in B<PIR> is more convenient. This is done with
78 C<pir_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
79
80 pir_output_is(<<'CODE',<<'OUT','nothing useful');
3721ccc [PATCH] @directive -> :directive - part 2 src, docs, tools
Leopold Toetsch authored
81 .sub main :main
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
82 print "hi\n"
83 .end
84 CODE
85 hi
86 OUT
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
87
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
88 =head2 Testing C source
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
89
ea0630c @jhoblitt reformat all Pod files under docs with podtidy (modified to also remo…
jhoblitt authored
90 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750 @jhoblitt fix verbatim block in docs/tests.pod
jhoblitt authored
91
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
92 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
93 #include <stdio.h>
94 #include "parrot/parrot.h"
95 #include "parrot/embed.h"
58c0aa8 update info on src tests
Leopold Toetsch authored
96
97 static opcode_t *the_test(Parrot_Interp, opcode_t *, opcode_t *);
98
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
99 int main(int argc, char* argv[]) {
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored
100 Parrot_Interp interp;
01f4e37 Updated doc and some code to reflect new Parrot_new() interface
Jens Rieks authored
101 interpreter = Parrot_new(NULL);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
102
58c0aa8 update info on src tests
Leopold Toetsch authored
103 if (!interpreter)
a25ee79 @bschmalhofer Replace a couple of calls to
bschmalhofer authored
104 return 1;
58c0aa8 update info on src tests
Leopold Toetsch authored
105
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored
106 Parrot_run_native(interp, the_test);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
107 printf("done\n");
9d8cce4 a bit more tidying up
Michael Scott authored
108 fflush(stdout);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
109 return 0;
110 }
58c0aa8 update info on src tests
Leopold Toetsch authored
111
112 static opcode_t*
f376029 @Infinoid Apply patch from jimmy++ in TT #111.
Infinoid authored
113 the_test(PARROT_INTERP,
9d8cce4 a bit more tidying up
Michael Scott authored
114 opcode_t *cur_op, opcode_t *start)
58c0aa8 update info on src tests
Leopold Toetsch authored
115 {
116 /* Your test goes here. */
117
118 return NULL; /* always return NULL */
119 }
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
120 CODE
121 # Anything that might be output prior to "done".
122 done
123 OUTPUT
124
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
125 Note that it's always a good idea to output "done" to confirm that the compiled
9f9cea2 @allisonrandal [pdd22io] Merging the pdd22io_part2 branch into trunk for r32922 to r…
allisonrandal authored
126 code executed completely. When mixing C<printf> and C<Parrot_io_printf> always append
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
127 a C<fflush(stdout);> after the former.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
128
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
129 =head2 Testing Perl5 components
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
130
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored
131 At the present time most, if not all, of the programs used to configure, build
132 and install Parrot are written in Perl 5. These programs take the form of
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to te…
jkeenan authored
133 program files (F<*.pl>) and Perl modules (F<*.pm>) holding subroutines and
134 other variables imported into the program files. Examples of such
135 program files can be found under F<tools/>; examples of such Perl modules
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored
136 can be found under F<lib/Parrot/>.
137
138 All of these Perl 5 components ought to be tested. Fortunately, over the last
139 decade, under the leadership of Michael Schwern, chromatic, Andy Lester and
140 many others, the Perl 5 community has developed a rigorous approach to testing
141 in which:
142
143 =over 4
144
145 =item a
146
147 Subroutines found in F<*.pl> files are extracted and placed in F<*.pm>
148 modules.
149
150 =item b
151
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to te…
jkeenan authored
152 Those subroutines are then imported back into the program file.
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored
153
154 =item c
155
156 Those subroutines are also imported into test files (F<*.t>) where are tests
157 are run by Test::Builder-based modules such as Test::Simple and Test::More.
158
159 =item d
160
161 Those test files are run by Test::Harness-based functionality such as
162 ExtUtils::MakeMaker's F<make test>, Module::Build's F<build test>, or
163 Test::Harness's F<prove>.
164
165 =item e
166
167 The extent to which the test files exercise all statements in the Perl modules
168 being tested is measured in coverage analysis using CPAN module Devel::Cover.
169
170 =item f
171
172 The underlying code is refactored and improved on the basis of the results of
173 tests and coverage analysis.
174
175 =back
176
177 Tests reflecting this approach can be found in F<t/configure/>,
178 F<t/postconfigure/>, F<t/tools/>, and so on.
179
180 It is our objective to test all Perl 5 components of the Parrot distribution
181 using the methodology above.
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
182
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test …
simoncozens authored
183 =head3 Build Tools Tests
184
185 The files in F<t/postconfigure> are tests for build system. The build tools
186 tests are intended to be run after someone has made changes in modules such as
187 F<lib/Parrot/Pmc2cUtils/>, F<Ops2cUtils/> and F<Ops2pmutils/>. They're set up
188 to be run after F<Configure.pl> has completed but before make has been invoked.
189 (In fact, they will generate errors if make has completed.) You can run them
190 with any of the following:
191
192 perl Configure.pl --test
193 perl Configure.pl --test=build
194 make buildtools_tests (following Configure.pl)
195
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
196 =head2 Testing language implementations
197
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
198 Language implementations are usually tested with
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
199 C<language_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
200
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
201 =head1 Ideal tests:
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
202
203 =over 4
204
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
205 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
206
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
207 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
208 code they're testing. These should include potentially out of band input
209 unless we decide that compilers should check for this themselves.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
210
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
211 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
212
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
213 Are small and self contained, so that if the tested feature breaks we can
214 identify where and why quickly.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
215
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
216 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
217
19b32a9 Minor edits
Simon Glover authored
218 Are valid. Essentially, they should conform to the additional documentation
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
219 that accompanies the feature (if any). [If there isn't any documentation, then
220 feel free to add some and/or complain to the mailing list].
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
221
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
222 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
223
224 Are a chunk of assembler and a chunk of expected output.
225
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
226 =back
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
227
228 =head1 TODO tests
229
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
230 In test driven development, tests are implemented first. So the tests are
231 initially expected to fail. This can be expressed by marking the tests as
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
232 TODO. See L<Test::More> on how to do that.
233
234 =head1 SKIP tests
235
236 TODO test actually executed, so that unexpected success can be detected.
237 In the case of missing requirements and in the case of serious breakdowns
238 the execution of tests can be skipped.
239 See L<Test::More> on how to do that.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
240
241 =head1 SEE ALSO
242
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
243 L<http://qa.perl.org/>
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
244 L<http://testanything.org/>
245 L<http://en.wikipedia.org/wiki/Test_Anything_Protocol>
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
246 F<t/TESTS.STATUS.pod>
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
247 F<t/README>
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
248
249 =cut
Something went wrong with that request. Please try again.