Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 239 lines (163 sloc) 7.147 kb
6db42a0 @coke #39217 - copyright cleanup. (docs/)
coke authored
1 # Copyright (C) 2001-2006, The Perl Foundation.
9d8cce4 a bit more tidying up
Michael Scott authored
2 # $Id$
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
3
9d8cce4 a bit more tidying up
Michael Scott authored
4 =head1 NAME
5
6 docs/tests.pod - Testing Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
7
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
8 =head1 A basic guide to writing and running tests for Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
9
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
10 This is quick and dirty pointer to how the Parrot test suite is executed and
11 to how new tests for Parrot should be written.
12 The testing system is liable to change in the future, but tests written
13 following the guidelines below should be easy to port into a new test suite.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
14
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
15 =head1 How to test parrot
16
17 The easy way to test parrot is running C<make test>. If you have
18 updated your code recently and tests began failing, go for a C<make
19 realclean> and recompile parrot before complaining.
20
21 If your architecture supports JIT, you can test parrot JIT engine using C<make
22 testj>. It works just like C<make test>, but uses the JIT engine when possible.
23
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
24 C<make languages-test> runs the test suite for most language implementations
25 in the languages directory.
26
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
27 =head2 Submitting smoke test results
28
29 Parrot has a status page with smoke test results
38d4ce0 @coke Fix the URL to the smoke server.
coke authored
30 L<http://smoke.parrotcode.org/smoke/>. You can supply new tests
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
31 results by just running C<make smoke>. It will run the same tests as
32 C<make test> would, but will additionally create a HTML table with the test
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
33 results. At the end, it will try to upload the test results to the
34 smoke server.
35
36 It is also possible to run a smoke test on JIT. For that, try running
8710563 @jhoblitt add smokej jit smoke test target
jhoblitt authored
37 C<make smokej>.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
38
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
39 C<make languages-smoke> does smoke testing for most language implementations
40 in the languages directory.
41
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
42 =head1 Location of the test files
43
44 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
45 A quick overview over the subdirs in F<t> can be found in F<t/README>.
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
46
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
47 The language implementations usually have their test files in F<languages/*/t>.
48
49 New tests should be added to an existing F<*.t> file.
50 If a previously untested feature is tested,
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
51 it might also make sense to create a new F<*.t> file.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
52
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
53 =head1 How to write a test
54
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
55 The testing framework needs to know how many tests it should expect. So the
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
56 number of planned tests needs to be incremented when adding a new test. This
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
57 is done near the top of a test file, in a line that looks like:
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
58
59 use Parrot::Test tests => 8;
60
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
61 =head2 Parrot Assembler
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
62
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
63 PASM tests are mostly used for testing ops. Appropriate test files for basic
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
64 ops are F<t/op/*.t>. Perl Magic Cookies are tested in F<t/pmc/*.t>. Add the
65 new test like this:
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
66
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored
67 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
68 *** a big chunk of assembler, eg:
69 print 1
70 print "\n" # you can even comment it if it's obscure
71 end # don't forget this...!
72 CODE
73 *** what you expect the output of the chunk to be, eg.
74 1
75 OUTPUT
76
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
77 =head2 Parrot Intermediate Representation
78
67c929f @ambs While discussion continues on p2, pod files should be at most
ambs authored
79 Writing tests in B<PIR> is more convenient. This is done with
80 C<pir_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
81
82 pir_output_is(<<'CODE',<<'OUT','nothing useful');
eb9e7dd @particle #37520: [TODO] rename library files from .imc to .pir
particle authored
83 .include 'library/config.pir'
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
84
3721ccc [PATCH] @directive -> :directive - part 2 src, docs, tools
Leopold Toetsch authored
85 .sub main :main
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
86 print "hi\n"
87 .end
88 CODE
89 hi
90 OUT
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
91
92 =head2 C source tests
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
93
ea0630c @jhoblitt reformat all Pod files under docs with podtidy (modified to also remove ...
jhoblitt authored
94 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750 @jhoblitt fix verbatim block in docs/tests.pod
jhoblitt authored
95
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
96 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
97 #include <stdio.h>
98 #include "parrot/parrot.h"
99 #include "parrot/embed.h"
58c0aa8 update info on src tests
Leopold Toetsch authored
100
101 static opcode_t *the_test(Parrot_Interp, opcode_t *, opcode_t *);
102
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
103 int main(int argc, char* argv[]) {
58c0aa8 update info on src tests
Leopold Toetsch authored
104 Parrot_Interp interpreter;
01f4e37 Updated doc and some code to reflect new Parrot_new() interface
Jens Rieks authored
105 interpreter = Parrot_new(NULL);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
106
58c0aa8 update info on src tests
Leopold Toetsch authored
107 if (!interpreter)
a25ee79 @bschmalhofer Replace a couple of calls to
bschmalhofer authored
108 return 1;
58c0aa8 update info on src tests
Leopold Toetsch authored
109
110 Parrot_run_native(interpreter, the_test);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
111 printf("done\n");
9d8cce4 a bit more tidying up
Michael Scott authored
112 fflush(stdout);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
113 return 0;
114 }
58c0aa8 update info on src tests
Leopold Toetsch authored
115
116 static opcode_t*
117 the_test(Parrot_Interp interpreter,
9d8cce4 a bit more tidying up
Michael Scott authored
118 opcode_t *cur_op, opcode_t *start)
58c0aa8 update info on src tests
Leopold Toetsch authored
119 {
120 /* Your test goes here. */
121
122 return NULL; /* always return NULL */
123 }
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
124 CODE
125 # Anything that might be output prior to "done".
126 done
127 OUTPUT
128
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
129 Note that it's always a good idea to output "done" to confirm that the compiled
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
130 code executed completely. When mixing C<printf> and C<PIO_printf> always append
131 a C<fflush(stdout);> after the former.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
132
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
133 =head2 Test Perl5 components
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
134
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
135 At the present time most, if not all, of the programs used to configure, build
136 and install Parrot are written in Perl 5. These programs take the form of
137 executable files (F<*.pl>) and Perl modules (F<*.pm>) holding subroutines and
138 other variables imported into the executable files. Examples of such
139 executable files can be found under F<tools/>; examples of such Perl modules
140 can be found under F<lib/Parrot/>.
141
142 All of these Perl 5 components ought to be tested. Fortunately, over the last
143 decade, under the leadership of Michael Schwern, chromatic, Andy Lester and
144 many others, the Perl 5 community has developed a rigorous approach to testing
145 in which:
146
147 =over 4
148
149 =item a
150
151 Subroutines found in F<*.pl> files are extracted and placed in F<*.pm>
152 modules.
153
154 =item b
155
156 Those subroutines are then imported back into the executable file.
157
158 =item c
159
160 Those subroutines are also imported into test files (F<*.t>) where are tests
161 are run by Test::Builder-based modules such as Test::Simple and Test::More.
162
163 =item d
164
165 Those test files are run by Test::Harness-based functionality such as
166 ExtUtils::MakeMaker's F<make test>, Module::Build's F<build test>, or
167 Test::Harness's F<prove>.
168
169 =item e
170
171 The extent to which the test files exercise all statements in the Perl modules
172 being tested is measured in coverage analysis using CPAN module Devel::Cover.
173
174 =item f
175
176 The underlying code is refactored and improved on the basis of the results of
177 tests and coverage analysis.
178
179 =back
180
181 Tests reflecting this approach can be found in F<t/configure/>,
182 F<t/postconfigure/>, F<t/tools/>, and so on.
183
184 It is our objective to test all Perl 5 components of the Parrot distribution
185 using the methodology above.
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
186
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
187 =head2 Testing language implementations
188
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
189 Language implementations are usually tested with
190 C<language_output_is> and friends..
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
191
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
192 =head1 Ideal tests:
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
193
194 =over 4
195
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
196 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
197
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
198 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
199 code they're testing. These should include potentially out of band input
200 unless we decide that compilers should check for this themselves.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
201
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
202 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
203
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
204 Are small and self contained, so that if the tested feature breaks we can
205 identify where and why quickly.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
206
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
207 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
208
19b32a9 Minor edits
Simon Glover authored
209 Are valid. Essentially, they should conform to the additional documentation
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
210 that accompanies the feature (if any). [If there isn't any documentation, then
211 feel free to add some and/or complain to the mailing list].
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
212
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
213 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
214
215 Are a chunk of assembler and a chunk of expected output.
216
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
217 =back
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
218
219 =head1 TODO tests
220
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
221 In test driven development, tests are implemented first. So the tests are
222 initially expected to fail. This can be expressed by marking the tests as
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
223 TODO. See L<Test::More> on how to do that.
224
225 =head1 SKIP tests
226
227 TODO test actually executed, so that unexpected success can be detected.
228 In the case of missing requirements and in the case of serious breakdowns
229 the execution of tests can be skipped.
230 See L<Test::More> on how to do that.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
231
232 =head1 SEE ALSO
233
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
234 L<http://qa.perl.org/>
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
235 F<t/TESTS.STATUS.pod>
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
236 F<t/README>
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
237
238 =cut
Something went wrong with that request. Please try again.