Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 255 lines (175 sloc) 7.86 kb
8ee1ad7 @allisonrandal [cage] Updating copyright in whole repository to Parrot Foundation.
allisonrandal authored
1 # Copyright (C) 2001-2007, Parrot Foundation.
9d8cce4 a bit more tidying up
Michael Scott authored
2 # $Id$
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
3
9d8cce4 a bit more tidying up
Michael Scott authored
4 =head1 NAME
5
6 docs/tests.pod - Testing Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
7
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
8 =head1 A basic guide to writing and running tests for Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
9
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
10 This is quick and dirty primer on to how the Parrot test suite is executed and
11 to how new tests for Parrot should be written. The testing system is liable to
12 change in the future, but tests written following the guidelines below should be
13 easy to port into a new test suite.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
14
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
15 =head1 How to test parrot
16
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
17 The easy way to test parrot is running C<make test>. If you have updated your
18 code recently and tests began failing, go for a C<make realclean> and recompile
19 parrot before complaining.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
20
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
21 C<make languages-test> runs the test suite for most language implementations
22 in the languages directory.
23
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
24 =head2 Submitting smolder test results
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
25
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
26 Parrot has a status page with smoke test results at
27 L<http://smolder.plusthree.com/app/public_projects/details/8>.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
28
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
29 You can supply new tests results by just running C<make smoke>. It will run the
30 same tests as C<make test> would, but will upload the test results to the
31 website.
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
32
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
33 =head1 Location of the test files
34
35 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
36 A quick overview over the subdirs in F<t> can be found in F<t/README>.
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
37
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
38 The language implementations usually have their test files in F<languages/*/t>.
39
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
40 New tests should be added to an existing F<*.t> file. If a previously untested
41 feature is tested, it might also make sense to create a new F<*.t> file. You
42 may also see tests named like foo-old.t, which are Perl tests that are in the
43 process of being translated to PIR.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
44
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
45 =head1 How to write a test
46
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
47 Test scripts must emit text that conforms to the C<Test Anything Protocol>.
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
48 Test scripts are currently usually written in PIR or Perl 5. The Perl 5 module
49 C<Parrot::Test> and the PIR module C<Test;More> help with writing tests.
50 Writing tests in PIR is preferred, but there are some cases where the
51 proper framework is not available. If you can, write your tests in PIR.
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
52
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
53 The testing framework needs to know how many tests it should expect. So the
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
54 number of planned tests needs to be incremented when adding a new test. This
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
55 is done near the top of a test file, in a line that looks like:
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
56
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
57 plan(42)
58
59 in PIR tests and
60
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
61 use Parrot::Test tests => 8;
62
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
63 for Perl 5 test scripts.
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
64
65 =head2 Testing Parrot Assembler
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
66
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
67 PASM tests are mostly used for testing ops. Appropriate test files for basic
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
68 ops are F<t/op/*.t>. Polymorphic Containers are tested in F<t/pmc/*.t>. Add
69 the new test like this:
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
70
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored
71 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
72 *** a big chunk of assembler, eg:
73 print 1
74 print "\n" # you can even comment it if it's obscure
75 end # don't forget this...!
76 CODE
77 *** what you expect the output of the chunk to be, eg.
78 1
79 OUTPUT
80
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
81 =head2 Testing Parrot Intermediate Representation
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
82
67c929f @ambs While discussion continues on p2, pod files should be at most
ambs authored
83 Writing tests in B<PIR> is more convenient. This is done with
84 C<pir_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
85
86 pir_output_is(<<'CODE',<<'OUT','nothing useful');
3721ccc [PATCH] @directive -> :directive - part 2 src, docs, tools
Leopold Toetsch authored
87 .sub main :main
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
88 print "hi\n"
89 .end
90 CODE
91 hi
92 OUT
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
93
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
94 =head2 Testing C source
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
95
ea0630c @jhoblitt reformat all Pod files under docs with podtidy (modified to also remove ...
jhoblitt authored
96 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750 @jhoblitt fix verbatim block in docs/tests.pod
jhoblitt authored
97
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
98 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
99 #include <stdio.h>
100 #include "parrot/parrot.h"
101 #include "parrot/embed.h"
58c0aa8 update info on src tests
Leopold Toetsch authored
102
103 static opcode_t *the_test(Parrot_Interp, opcode_t *, opcode_t *);
104
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
105 int main(int argc, char* argv[]) {
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored
106 Parrot_Interp interp;
01f4e37 Updated doc and some code to reflect new Parrot_new() interface
Jens Rieks authored
107 interpreter = Parrot_new(NULL);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
108
58c0aa8 update info on src tests
Leopold Toetsch authored
109 if (!interpreter)
a25ee79 @bschmalhofer Replace a couple of calls to
bschmalhofer authored
110 return 1;
58c0aa8 update info on src tests
Leopold Toetsch authored
111
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored
112 Parrot_run_native(interp, the_test);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
113 printf("done\n");
c9c418b @bacek Indent code properly
bacek authored
114 fflush(stdout);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
115 return 0;
116 }
58c0aa8 update info on src tests
Leopold Toetsch authored
117
118 static opcode_t*
f376029 @Infinoid Apply patch from jimmy++ in TT #111.
Infinoid authored
119 the_test(PARROT_INTERP,
9d8cce4 a bit more tidying up
Michael Scott authored
120 opcode_t *cur_op, opcode_t *start)
58c0aa8 update info on src tests
Leopold Toetsch authored
121 {
122 /* Your test goes here. */
123
124 return NULL; /* always return NULL */
125 }
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
126 CODE
127 # Anything that might be output prior to "done".
128 done
129 OUTPUT
130
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
131 Note that it's always a good idea to output "done" to confirm that the compiled
9f9cea2 @allisonrandal [pdd22io] Merging the pdd22io_part2 branch into trunk for r32922 to r336...
allisonrandal authored
132 code executed completely. When mixing C<printf> and C<Parrot_io_printf> always append
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
133 a C<fflush(stdout);> after the former.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
134
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
135 =head2 Testing Perl5 components
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
136
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
137 At the present time most, if not all, of the programs used to configure, build
138 and install Parrot are written in Perl 5. These programs take the form of
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to testi...
jkeenan authored
139 program files (F<*.pl>) and Perl modules (F<*.pm>) holding subroutines and
140 other variables imported into the program files. Examples of such
141 program files can be found under F<tools/>; examples of such Perl modules
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
142 can be found under F<lib/Parrot/>.
143
144 All of these Perl 5 components ought to be tested. Fortunately, over the last
145 decade, under the leadership of Michael Schwern, chromatic, Andy Lester and
146 many others, the Perl 5 community has developed a rigorous approach to testing
147 in which:
148
149 =over 4
150
151 =item a
152
153 Subroutines found in F<*.pl> files are extracted and placed in F<*.pm>
154 modules.
155
156 =item b
157
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to testi...
jkeenan authored
158 Those subroutines are then imported back into the program file.
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
159
160 =item c
161
162 Those subroutines are also imported into test files (F<*.t>) where are tests
163 are run by Test::Builder-based modules such as Test::Simple and Test::More.
164
165 =item d
166
167 Those test files are run by Test::Harness-based functionality such as
168 ExtUtils::MakeMaker's F<make test>, Module::Build's F<build test>, or
169 Test::Harness's F<prove>.
170
171 =item e
172
173 The extent to which the test files exercise all statements in the Perl modules
174 being tested is measured in coverage analysis using CPAN module Devel::Cover.
175
176 =item f
177
178 The underlying code is refactored and improved on the basis of the results of
179 tests and coverage analysis.
180
181 =back
182
183 Tests reflecting this approach can be found in F<t/configure/>,
184 F<t/postconfigure/>, F<t/tools/>, and so on.
185
186 It is our objective to test all Perl 5 components of the Parrot distribution
187 using the methodology above.
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
188
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test fai...
simoncozens authored
189 =head3 Build Tools Tests
190
191 The files in F<t/postconfigure> are tests for build system. The build tools
192 tests are intended to be run after someone has made changes in modules such as
762a0f2 @bacek Merge branch 'ops_pct' back to trunk.
bacek authored
193 F<lib/Parrot/Pmc2cUtils/>. They're set up to be run after F<Configure.pl> has
194 completed but before make has been invoked. (In fact, they will generate
195 errors if make has completed.) You can run them with any of the following:
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test fai...
simoncozens authored
196
197 perl Configure.pl --test
198 perl Configure.pl --test=build
199 make buildtools_tests (following Configure.pl)
200
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
201 =head2 Testing language implementations
202
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
203 Language implementations are usually tested with
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
204 C<language_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
205
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
206 =head1 Ideal tests:
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
207
208 =over 4
209
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
210 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
211
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
212 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
213 code they're testing. These should include potentially out of band input
214 unless we decide that compilers should check for this themselves.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
215
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
216 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
217
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
218 Are small and self contained, so that if the tested feature breaks we can
219 identify where and why quickly.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
220
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
221 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
222
19b32a9 Minor edits
Simon Glover authored
223 Are valid. Essentially, they should conform to the additional documentation
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
224 that accompanies the feature (if any). [If there isn't any documentation, then
225 feel free to add some and/or complain to the mailing list].
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
226
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
227 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
228
229 Are a chunk of assembler and a chunk of expected output.
230
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
231 =back
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
232
233 =head1 TODO tests
234
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
235 In test driven development, tests are implemented first. So the tests are
236 initially expected to fail. This can be expressed by marking the tests as
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
237 TODO. See L<Test::More> on how to do that.
238
239 =head1 SKIP tests
240
241 TODO test actually executed, so that unexpected success can be detected.
242 In the case of missing requirements and in the case of serious breakdowns
243 the execution of tests can be skipped.
244 See L<Test::More> on how to do that.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
245
246 =head1 SEE ALSO
247
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
248 L<http://qa.perl.org/>
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
249 L<http://testanything.org/>
250 L<http://en.wikipedia.org/wiki/Test_Anything_Protocol>
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
251 F<t/TESTS.STATUS.pod>
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
252 F<t/README>
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
253
254 =cut
Something went wrong with that request. Please try again.