Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 241 lines (163 sloc) 7.55 kb
98e627c @leto Remove references to long-gone languages in parrot.git and mention all_h...
leto authored
1 # Copyright (C) 2001-2012, Parrot Foundation.
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
2
9d8cce4 a bit more tidying up
Michael Scott authored
3 =head1 NAME
4
5 docs/tests.pod - Testing Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
6
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
7 =head1 A basic guide to writing and running tests for Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
8
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
9 This is quick and dirty primer on to how the Parrot test suite is executed and
10 to how new tests for Parrot should be written. The testing system is liable to
11 change in the future, but tests written following the guidelines below should be
12 easy to port into a new test suite.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
13
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
14 =head1 How to test parrot
15
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
16 The easy way to test parrot is running C<make test>. If you have updated your
17 code recently and tests began failing, go for a C<make realclean> and recompile
18 parrot before complaining.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
19
98e627c @leto Remove references to long-gone languages in parrot.git and mention all_h...
leto authored
20 C<make all_hll_test> runs the test suite for all known working HLLs and libraries.
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
21
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
22 =head2 Submitting smolder test results
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
23
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
24 Parrot has a status page with smoke test results at
6120e2f @leto [docs] Fix link to Smolder
leto authored
25 L<http://smolder.parrot.org/app/projects/smoke_reports/1>.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
26
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
27 You can supply new tests results by just running C<make smoke>. It will run the
28 same tests as C<make test> would, but will upload the test results to the
29 website.
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
30
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
31 =head1 Location of the test files
32
33 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
98e627c @leto Remove references to long-gone languages in parrot.git and mention all_h...
leto authored
34 A quick overview about each test directory in F<t> can be found in F<t/README>.
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
35
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
36 New tests should be added to an existing F<*.t> file. If a previously untested
37 feature is tested, it might also make sense to create a new F<*.t> file. You
38 may also see tests named like foo-old.t, which are Perl tests that are in the
39 process of being translated to PIR.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
40
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
41 =head1 How to write a test
42
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
43 Test scripts must emit text that conforms to the C<Test Anything Protocol>.
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
44 Test scripts are currently usually written in PIR or Perl 5. The Perl 5 module
45 C<Parrot::Test> and the PIR module C<Test;More> help with writing tests.
46 Writing tests in PIR is preferred, but there are some cases where the
47 proper framework is not available. If you can, write your tests in PIR.
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
48
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
49 The testing framework needs to know how many tests it should expect. So the
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
50 number of planned tests needs to be incremented when adding a new test. This
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
51 is done near the top of a test file, in a line that looks like:
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
52
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
53 plan(42)
54
55 in PIR tests and
56
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
57 use Parrot::Test tests => 8;
58
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
59 for Perl 5 test scripts.
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
60
61 =head2 Testing Parrot Assembler
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
62
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
63 PASM tests are mostly used for testing ops. Appropriate test files for basic
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sprin...
leto authored
64 ops are F<t/op/*.t>. Polymorphic Containers are tested in F<t/pmc/*.t>. Add
65 the new test like this:
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
66
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored
67 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
68 *** a big chunk of assembler, eg:
69 print 1
70 print "\n" # you can even comment it if it's obscure
71 end # don't forget this...!
72 CODE
73 *** what you expect the output of the chunk to be, eg.
74 1
75 OUTPUT
76
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
77 =head2 Testing Parrot Intermediate Representation
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
78
67c929f @ambs While discussion continues on p2, pod files should be at most
ambs authored
79 Writing tests in B<PIR> is more convenient. This is done with
80 C<pir_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
81
82 pir_output_is(<<'CODE',<<'OUT','nothing useful');
3721ccc [PATCH] @directive -> :directive - part 2 src, docs, tools
Leopold Toetsch authored
83 .sub main :main
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
84 print "hi\n"
85 .end
86 CODE
87 hi
88 OUT
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
89
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
90 =head2 Testing C source
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
91
ea0630c @jhoblitt reformat all Pod files under docs with podtidy (modified to also remove ...
jhoblitt authored
92 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750 @jhoblitt fix verbatim block in docs/tests.pod
jhoblitt authored
93
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
94 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
95 #include <stdio.h>
96 #include "parrot/parrot.h"
58c0aa8 update info on src tests
Leopold Toetsch authored
97
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
98 int main(int argc, char* argv[]) {
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored
99 Parrot_Interp interp;
01f4e37 Updated doc and some code to reflect new Parrot_new() interface
Jens Rieks authored
100 interpreter = Parrot_new(NULL);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
101
58c0aa8 update info on src tests
Leopold Toetsch authored
102 if (!interpreter)
a25ee79 @bschmalhofer Replace a couple of calls to
bschmalhofer authored
103 return 1;
58c0aa8 update info on src tests
Leopold Toetsch authored
104
2f0d61c @plobsing remove remaining references to Parrot_run_native
plobsing authored
105 /* Your test goes here. */
106
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
107 printf("done\n");
c9c418b @bacek Indent code properly
bacek authored
108 fflush(stdout);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
109 return 0;
110 }
58c0aa8 update info on src tests
Leopold Toetsch authored
111
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scott@m...
boemmels authored
112 CODE
113 # Anything that might be output prior to "done".
114 done
115 OUTPUT
116
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
117 Note that it's always a good idea to output "done" to confirm that the compiled
9f9cea2 @allisonrandal [pdd22io] Merging the pdd22io_part2 branch into trunk for r32922 to r336...
allisonrandal authored
118 code executed completely. When mixing C<printf> and C<Parrot_io_printf> always append
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
119 a C<fflush(stdout);> after the former.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
120
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
121 =head2 Testing Perl5 components
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
122
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
123 At the present time most, if not all, of the programs used to configure, build
124 and install Parrot are written in Perl 5. These programs take the form of
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to testi...
jkeenan authored
125 program files (F<*.pl>) and Perl modules (F<*.pm>) holding subroutines and
126 other variables imported into the program files. Examples of such
127 program files can be found under F<tools/>; examples of such Perl modules
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
128 can be found under F<lib/Parrot/>.
129
130 All of these Perl 5 components ought to be tested. Fortunately, over the last
131 decade, under the leadership of Michael Schwern, chromatic, Andy Lester and
132 many others, the Perl 5 community has developed a rigorous approach to testing
133 in which:
134
135 =over 4
136
137 =item a
138
139 Subroutines found in F<*.pl> files are extracted and placed in F<*.pm>
140 modules.
141
142 =item b
143
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to testi...
jkeenan authored
144 Those subroutines are then imported back into the program file.
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl 5 c...
bschmalhofer authored
145
146 =item c
147
148 Those subroutines are also imported into test files (F<*.t>) where are tests
149 are run by Test::Builder-based modules such as Test::Simple and Test::More.
150
151 =item d
152
153 Those test files are run by Test::Harness-based functionality such as
154 ExtUtils::MakeMaker's F<make test>, Module::Build's F<build test>, or
155 Test::Harness's F<prove>.
156
157 =item e
158
159 The extent to which the test files exercise all statements in the Perl modules
160 being tested is measured in coverage analysis using CPAN module Devel::Cover.
161
162 =item f
163
164 The underlying code is refactored and improved on the basis of the results of
165 tests and coverage analysis.
166
167 =back
168
169 Tests reflecting this approach can be found in F<t/configure/>,
170 F<t/postconfigure/>, F<t/tools/>, and so on.
171
172 It is our objective to test all Perl 5 components of the Parrot distribution
173 using the methodology above.
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
174
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test fai...
simoncozens authored
175 =head3 Build Tools Tests
176
177 The files in F<t/postconfigure> are tests for build system. The build tools
178 tests are intended to be run after someone has made changes in modules such as
762a0f2 @bacek Merge branch 'ops_pct' back to trunk.
bacek authored
179 F<lib/Parrot/Pmc2cUtils/>. They're set up to be run after F<Configure.pl> has
180 completed but before make has been invoked. (In fact, they will generate
181 errors if make has completed.) You can run them with any of the following:
017e3b7 fix some warnings from podchecker
mstevens authored
182
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test fai...
simoncozens authored
183 perl Configure.pl --test
184 perl Configure.pl --test=build
185 make buildtools_tests (following Configure.pl)
186
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
187 =head2 Testing language implementations
188
c989e15 @Whiteknight Remove embed.h (and all traces of it) and embed.c. Get things building a...
Whiteknight authored
189 Language implementations are usually tested with
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
190 C<language_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
191
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
192 =head1 Ideal tests:
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
193
194 =over 4
195
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
196 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
197
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
198 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
199 code they're testing. These should include potentially out of band input
200 unless we decide that compilers should check for this themselves.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
201
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
202 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
203
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
204 Are small and self contained, so that if the tested feature breaks we can
205 identify where and why quickly.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
206
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
207 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
208
19b32a9 Minor edits
Simon Glover authored
209 Are valid. Essentially, they should conform to the additional documentation
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
210 that accompanies the feature (if any). [If there isn't any documentation, then
211 feel free to add some and/or complain to the mailing list].
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
212
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
213 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
214
215 Are a chunk of assembler and a chunk of expected output.
216
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
217 =back
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
218
219 =head1 TODO tests
220
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
221 In test driven development, tests are implemented first. So the tests are
222 initially expected to fail. This can be expressed by marking the tests as
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
223 TODO. See L<Test::More> on how to do that.
224
225 =head1 SKIP tests
226
227 TODO test actually executed, so that unexpected success can be detected.
228 In the case of missing requirements and in the case of serious breakdowns
229 the execution of tests can be skipped.
230 See L<Test::More> on how to do that.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
231
232 =head1 SEE ALSO
233
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling correcti...
chromatic authored
234 L<http://qa.perl.org/>
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
235 L<http://testanything.org/>
236 L<http://en.wikipedia.org/wiki/Test_Anything_Protocol>
46ec5ef @leto [doc] Linkify
leto authored
237 L<https://github.com/parrot/parrot/blob/master/t/TESTS.STATUS.pod>
238 L<https://github.com/parrot/parrot/blob/master/t/README.pod>
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.pod...
bschmalhofer authored
239
240 =cut
Something went wrong with that request. Please try again.