Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 245 lines (166 sloc) 7.581 kb
8ee1ad7 @allisonrandal [cage] Updating copyright in whole repository to Parrot Foundation.
allisonrandal authored
1 # Copyright (C) 2001-2007, Parrot Foundation.
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
2
9d8cce4 a bit more tidying up
Michael Scott authored
3 =head1 NAME
4
5 docs/tests.pod - Testing Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
6
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
7 =head1 A basic guide to writing and running tests for Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
8
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
9 This is quick and dirty primer on to how the Parrot test suite is executed and
10 to how new tests for Parrot should be written. The testing system is liable to
11 change in the future, but tests written following the guidelines below should be
12 easy to port into a new test suite.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
13
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
14 =head1 How to test parrot
15
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
16 The easy way to test parrot is running C<make test>. If you have updated your
17 code recently and tests began failing, go for a C<make realclean> and recompile
18 parrot before complaining.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
19
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
20 C<make languages-test> runs the test suite for most language implementations
21 in the languages directory.
22
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
23 =head2 Submitting smolder test results
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
24
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
25 Parrot has a status page with smoke test results at
6120e2f @leto [docs] Fix link to Smolder
leto authored
26 L<http://smolder.parrot.org/app/projects/smoke_reports/1>.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
27
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
28 You can supply new tests results by just running C<make smoke>. It will run the
29 same tests as C<make test> would, but will upload the test results to the
30 website.
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
31
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
32 =head1 Location of the test files
33
34 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
35 A quick overview over the subdirs in F<t> can be found in F<t/README>.
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
36
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
37 The language implementations usually have their test files in F<languages/*/t>.
38
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
39 New tests should be added to an existing F<*.t> file. If a previously untested
40 feature is tested, it might also make sense to create a new F<*.t> file. You
41 may also see tests named like foo-old.t, which are Perl tests that are in the
42 process of being translated to PIR.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
43
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
44 =head1 How to write a test
45
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
46 Test scripts must emit text that conforms to the C<Test Anything Protocol>.
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
47 Test scripts are currently usually written in PIR or Perl 5. The Perl 5 module
48 C<Parrot::Test> and the PIR module C<Test;More> help with writing tests.
49 Writing tests in PIR is preferred, but there are some cases where the
50 proper framework is not available. If you can, write your tests in PIR.
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
51
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
52 The testing framework needs to know how many tests it should expect. So the
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
53 number of planned tests needs to be incremented when adding a new test. This
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
54 is done near the top of a test file, in a line that looks like:
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
55
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
56 plan(42)
57
58 in PIR tests and
59
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
60 use Parrot::Test tests => 8;
61
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
62 for Perl 5 test scripts.
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
63
64 =head2 Testing Parrot Assembler
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
65
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
66 PASM tests are mostly used for testing ops. Appropriate test files for basic
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
67 ops are F<t/op/*.t>. Polymorphic Containers are tested in F<t/pmc/*.t>. Add
68 the new test like this:
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
69
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored
70 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
71 *** a big chunk of assembler, eg:
72 print 1
73 print "\n" # you can even comment it if it's obscure
74 end # don't forget this...!
75 CODE
76 *** what you expect the output of the chunk to be, eg.
77 1
78 OUTPUT
79
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
80 =head2 Testing Parrot Intermediate Representation
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
81
67c929f @ambs While discussion continues on p2, pod files should be at most
ambs authored
82 Writing tests in B<PIR> is more convenient. This is done with
83 C<pir_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
84
85 pir_output_is(<<'CODE',<<'OUT','nothing useful');
3721ccc [PATCH] @directive -> :directive - part 2 src, docs, tools
Leopold Toetsch authored
86 .sub main :main
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
87 print "hi\n"
88 .end
89 CODE
90 hi
91 OUT
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
92
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
93 =head2 Testing C source
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
94
ea0630c @jhoblitt reformat all Pod files under docs with podtidy (modified to also remove ...
jhoblitt authored
95 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750 @jhoblitt fix verbatim block in docs/tests.pod
jhoblitt authored
96
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
97 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
98 #include <stdio.h>
99 #include "parrot/parrot.h"
100 #include "parrot/embed.h"
58c0aa8 update info on src tests
Leopold Toetsch authored
101
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
102 int main(int argc, char* argv[]) {
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored
103 Parrot_Interp interp;
01f4e37 Updated doc and some code to reflect new Parrot_new() interface
Jens Rieks authored
104 interpreter = Parrot_new(NULL);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
105
58c0aa8 update info on src tests
Leopold Toetsch authored
106 if (!interpreter)
a25ee79 @bschmalhofer Replace a couple of calls to
bschmalhofer authored
107 return 1;
58c0aa8 update info on src tests
Leopold Toetsch authored
108
2f0d61c @plobsing remove remaining references to Parrot_run_native
plobsing authored
109 /* Your test goes here. */
110
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
111 printf("done\n");
c9c418b @bacek Indent code properly
bacek authored
112 fflush(stdout);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
113 return 0;
114 }
58c0aa8 update info on src tests
Leopold Toetsch authored
115
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
116 CODE
117 # Anything that might be output prior to "done".
118 done
119 OUTPUT
120
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
121 Note that it's always a good idea to output "done" to confirm that the compiled
9f9cea2 @allisonrandal [pdd22io] Merging the pdd22io_part2 branch into trunk for r32922 to r336...
allisonrandal authored
122 code executed completely. When mixing C<printf> and C<Parrot_io_printf> always append
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
123 a C<fflush(stdout);> after the former.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
124
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
125 =head2 Testing Perl5 components
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
126
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
127 At the present time most, if not all, of the programs used to configure, build
128 and install Parrot are written in Perl 5. These programs take the form of
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to testi...
jkeenan authored
129 program files (F<*.pl>) and Perl modules (F<*.pm>) holding subroutines and
130 other variables imported into the program files. Examples of such
131 program files can be found under F<tools/>; examples of such Perl modules
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
132 can be found under F<lib/Parrot/>.
133
134 All of these Perl 5 components ought to be tested. Fortunately, over the last
135 decade, under the leadership of Michael Schwern, chromatic, Andy Lester and
136 many others, the Perl 5 community has developed a rigorous approach to testing
137 in which:
138
139 =over 4
140
141 =item a
142
143 Subroutines found in F<*.pl> files are extracted and placed in F<*.pm>
144 modules.
145
146 =item b
147
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to testi...
jkeenan authored
148 Those subroutines are then imported back into the program file.
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
149
150 =item c
151
152 Those subroutines are also imported into test files (F<*.t>) where are tests
153 are run by Test::Builder-based modules such as Test::Simple and Test::More.
154
155 =item d
156
157 Those test files are run by Test::Harness-based functionality such as
158 ExtUtils::MakeMaker's F<make test>, Module::Build's F<build test>, or
159 Test::Harness's F<prove>.
160
161 =item e
162
163 The extent to which the test files exercise all statements in the Perl modules
164 being tested is measured in coverage analysis using CPAN module Devel::Cover.
165
166 =item f
167
168 The underlying code is refactored and improved on the basis of the results of
169 tests and coverage analysis.
170
171 =back
172
173 Tests reflecting this approach can be found in F<t/configure/>,
174 F<t/postconfigure/>, F<t/tools/>, and so on.
175
176 It is our objective to test all Perl 5 components of the Parrot distribution
177 using the methodology above.
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
178
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test fai...
simoncozens authored
179 =head3 Build Tools Tests
180
181 The files in F<t/postconfigure> are tests for build system. The build tools
182 tests are intended to be run after someone has made changes in modules such as
762a0f2 @bacek Merge branch 'ops_pct' back to trunk.
bacek authored
183 F<lib/Parrot/Pmc2cUtils/>. They're set up to be run after F<Configure.pl> has
184 completed but before make has been invoked. (In fact, they will generate
185 errors if make has completed.) You can run them with any of the following:
017e3b7 fix some warnings from podchecker
mstevens authored
186
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test fai...
simoncozens authored
187 perl Configure.pl --test
188 perl Configure.pl --test=build
189 make buildtools_tests (following Configure.pl)
190
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
191 =head2 Testing language implementations
192
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
193 Language implementations are usually tested with
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
194 C<language_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
195
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
196 =head1 Ideal tests:
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
197
198 =over 4
199
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
200 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
201
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
202 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
203 code they're testing. These should include potentially out of band input
204 unless we decide that compilers should check for this themselves.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
205
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
206 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
207
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
208 Are small and self contained, so that if the tested feature breaks we can
209 identify where and why quickly.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
210
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
211 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
212
19b32a9 Minor edits
Simon Glover authored
213 Are valid. Essentially, they should conform to the additional documentation
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
214 that accompanies the feature (if any). [If there isn't any documentation, then
215 feel free to add some and/or complain to the mailing list].
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
216
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
217 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
218
219 Are a chunk of assembler and a chunk of expected output.
220
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
221 =back
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
222
223 =head1 TODO tests
224
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
225 In test driven development, tests are implemented first. So the tests are
226 initially expected to fail. This can be expressed by marking the tests as
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
227 TODO. See L<Test::More> on how to do that.
228
229 =head1 SKIP tests
230
231 TODO test actually executed, so that unexpected success can be detected.
232 In the case of missing requirements and in the case of serious breakdowns
233 the execution of tests can be skipped.
234 See L<Test::More> on how to do that.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
235
236 =head1 SEE ALSO
237
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
238 L<http://qa.perl.org/>
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
239 L<http://testanything.org/>
240 L<http://en.wikipedia.org/wiki/Test_Anything_Protocol>
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
241 F<t/TESTS.STATUS.pod>
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
242 F<t/README>
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
243
244 =cut
Something went wrong with that request. Please try again.