Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 243 lines (164 sloc) 7.514 kB
98e627c @leto Remove references to long-gone languages in parrot.git and mention al…
leto authored
1 # Copyright (C) 2001-2012, Parrot Foundation.
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
2
983fe83 @ayardley Added DESCRIPTION and upcased '=head1' cps.
ayardley authored
3 =pod
4
9d8cce4 a bit more tidying up
Michael Scott authored
5 =head1 NAME
6
7 docs/tests.pod - Testing Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
8
983fe83 @ayardley Added DESCRIPTION and upcased '=head1' cps.
ayardley authored
9 =head1 DESCRIPTION
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
10
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sp…
leto authored
11 This is quick and dirty primer on to how the Parrot test suite is executed and
12 to how new tests for Parrot should be written. The testing system is liable to
13 change in the future, but tests written following the guidelines below should be
14 easy to port into a new test suite.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
15
983fe83 @ayardley Added DESCRIPTION and upcased '=head1' cps.
ayardley authored
16 =head1 HOW TO TEST PARROT
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
17
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sp…
leto authored
18 The easy way to test parrot is running C<make test>. If you have updated your
19 code recently and tests began failing, go for a C<make realclean> and recompile
20 parrot before complaining.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
21
98e627c @leto Remove references to long-gone languages in parrot.git and mention al…
leto authored
22 C<make all_hll_test> runs the test suite for all known working HLLs and libraries.
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
23
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
24 =head2 Submitting smolder test results
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
25
5b64732 @jkeenan Merging smoke2smolder branch into trunk. Implement's Coke's patch
jkeenan authored
26 Parrot has a status page with smoke test results at
6120e2f @leto [docs] Fix link to Smolder
leto authored
27 L<http://smolder.parrot.org/app/projects/smoke_reports/1>.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
28
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sp…
leto authored
29 You can supply new tests results by just running C<make smoke>. It will run the
30 same tests as C<make test> would, but will upload the test results to the
31 website.
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored
32
983fe83 @ayardley Added DESCRIPTION and upcased '=head1' cps.
ayardley authored
33 =head1 LOCATION OF THE TEST FILES
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
34
35 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
98e627c @leto Remove references to long-gone languages in parrot.git and mention al…
leto authored
36 A quick overview about each test directory in F<t> can be found in F<t/README>.
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
37
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sp…
leto authored
38 New tests should be added to an existing F<*.t> file. If a previously untested
39 feature is tested, it might also make sense to create a new F<*.t> file. You
40 may also see tests named like foo-old.t, which are Perl tests that are in the
41 process of being translated to PIR.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
42
983fe83 @ayardley Added DESCRIPTION and upcased '=head1' cps.
ayardley authored
43 =head1 HOW TO WRITE A TEST
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
44
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
45 Test scripts must emit text that conforms to the C<Test Anything Protocol>.
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sp…
leto authored
46 Test scripts are currently usually written in PIR or Perl 5. The Perl 5 module
47 C<Parrot::Test> and the PIR module C<Test;More> help with writing tests.
48 Writing tests in PIR is preferred, but there are some cases where the
49 proper framework is not available. If you can, write your tests in PIR.
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
50
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
51 The testing framework needs to know how many tests it should expect. So the
c138099 @bschmalhofer Move Parrot_Docs.t from t/docs to t/perl, as
bschmalhofer authored
52 number of planned tests needs to be incremented when adding a new test. This
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
53 is done near the top of a test file, in a line that looks like:
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
54
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sp…
leto authored
55 plan(42)
56
57 in PIR tests and
58
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored
59 use Parrot::Test tests => 8;
60
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sp…
leto authored
61 for Perl 5 test scripts.
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
62
63 =head2 Testing Parrot Assembler
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
64
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
65 PASM tests are mostly used for testing ops. Appropriate test files for basic
a0bba64 @leto [docs] Update tests.pod a bit and remove a wrong comment from t/op/sp…
leto authored
66 ops are F<t/op/*.t>. Polymorphic Containers are tested in F<t/pmc/*.t>. Add
67 the new test like this:
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
68
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored
69 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
70 *** a big chunk of assembler, eg:
71 print 1
72 print "\n" # you can even comment it if it's obscure
73 end # don't forget this...!
74 CODE
75 *** what you expect the output of the chunk to be, eg.
76 1
77 OUTPUT
78
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
79 =head2 Testing Parrot Intermediate Representation
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
80
67c929f @ambs While discussion continues on p2, pod files should be at most
ambs authored
81 Writing tests in B<PIR> is more convenient. This is done with
82 C<pir_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
83
84 pir_output_is(<<'CODE',<<'OUT','nothing useful');
3721ccc [PATCH] @directive -> :directive - part 2 src, docs, tools
Leopold Toetsch authored
85 .sub main :main
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
86 print "hi\n"
87 .end
88 CODE
89 hi
90 OUT
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored
91
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
92 =head2 Testing C source
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
93
ea0630c @jhoblitt reformat all Pod files under docs with podtidy (modified to also remo…
jhoblitt authored
94 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750 @jhoblitt fix verbatim block in docs/tests.pod
jhoblitt authored
95
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
96 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
97 #include <stdio.h>
98 #include "parrot/parrot.h"
58c0aa8 update info on src tests
Leopold Toetsch authored
99
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
100 int main(int argc, char* argv[]) {
6de588f @paultcochrane [docs] Instances of 'interpreter' changed to 'interp' as per convention
paultcochrane authored
101 Parrot_Interp interp;
01f4e37 Updated doc and some code to reflect new Parrot_new() interface
Jens Rieks authored
102 interpreter = Parrot_new(NULL);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
103
58c0aa8 update info on src tests
Leopold Toetsch authored
104 if (!interpreter)
a25ee79 @bschmalhofer Replace a couple of calls to
bschmalhofer authored
105 return 1;
58c0aa8 update info on src tests
Leopold Toetsch authored
106
2f0d61c @plobsing remove remaining references to Parrot_run_native
plobsing authored
107 /* Your test goes here. */
108
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
109 printf("done\n");
c9c418b @bacek Indent code properly
bacek authored
110 fflush(stdout);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
111 return 0;
112 }
58c0aa8 update info on src tests
Leopold Toetsch authored
113
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored
114 CODE
115 # Anything that might be output prior to "done".
116 done
117 OUTPUT
118
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
119 Note that it's always a good idea to output "done" to confirm that the compiled
9f9cea2 @allisonrandal [pdd22io] Merging the pdd22io_part2 branch into trunk for r32922 to r…
allisonrandal authored
120 code executed completely. When mixing C<printf> and C<Parrot_io_printf> always append
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
121 a C<fflush(stdout);> after the former.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
122
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
123 =head2 Testing Perl5 components
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
124
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored
125 At the present time most, if not all, of the programs used to configure, build
126 and install Parrot are written in Perl 5. These programs take the form of
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to te…
jkeenan authored
127 program files (F<*.pl>) and Perl modules (F<*.pm>) holding subroutines and
128 other variables imported into the program files. Examples of such
129 program files can be found under F<tools/>; examples of such Perl modules
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored
130 can be found under F<lib/Parrot/>.
131
132 All of these Perl 5 components ought to be tested. Fortunately, over the last
133 decade, under the leadership of Michael Schwern, chromatic, Andy Lester and
134 many others, the Perl 5 community has developed a rigorous approach to testing
135 in which:
136
137 =over 4
138
139 =item a
140
141 Subroutines found in F<*.pl> files are extracted and placed in F<*.pm>
142 modules.
143
144 =item b
145
c13cb72 @jkeenan Following further discussion in RT 43008: In sections relating to te…
jkeenan authored
146 Those subroutines are then imported back into the program file.
7cc4802 @bschmalhofer #43008: [PATCH] docs/tests.pod: Expand discussion of testing of Perl …
bschmalhofer authored
147
148 =item c
149
150 Those subroutines are also imported into test files (F<*.t>) where are tests
151 are run by Test::Builder-based modules such as Test::Simple and Test::More.
152
153 =item d
154
155 Those test files are run by Test::Harness-based functionality such as
156 ExtUtils::MakeMaker's F<make test>, Module::Build's F<build test>, or
157 Test::Harness's F<prove>.
158
159 =item e
160
161 The extent to which the test files exercise all statements in the Perl modules
162 being tested is measured in coverage analysis using CPAN module Devel::Cover.
163
164 =item f
165
166 The underlying code is refactored and improved on the basis of the results of
167 tests and coverage analysis.
168
169 =back
170
171 Tests reflecting this approach can be found in F<t/configure/>,
172 F<t/postconfigure/>, F<t/tools/>, and so on.
173
174 It is our objective to test all Perl 5 components of the Parrot distribution
175 using the methodology above.
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored
176
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test …
simoncozens authored
177 =head3 Build Tools Tests
178
179 The files in F<t/postconfigure> are tests for build system. The build tools
180 tests are intended to be run after someone has made changes in modules such as
762a0f2 @bacek Merge branch 'ops_pct' back to trunk.
bacek authored
181 F<lib/Parrot/Pmc2cUtils/>. They're set up to be run after F<Configure.pl> has
182 completed but before make has been invoked. (In fact, they will generate
183 errors if make has completed.) You can run them with any of the following:
017e3b7 fix some warnings from podchecker
mstevens authored
184
c4608d5 @simoncozens [BUILD] Missing opcodes should be an error - (#48497) fix build test …
simoncozens authored
185 perl Configure.pl --test
186 perl Configure.pl --test=build
187 make buildtools_tests (following Configure.pl)
188
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
189 =head2 Testing language implementations
190
c989e15 @Whiteknight Remove embed.h (and all traces of it) and embed.c. Get things buildin…
Whiteknight authored
191 Language implementations are usually tested with
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
192 C<language_output_is> and friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
193
983fe83 @ayardley Added DESCRIPTION and upcased '=head1' cps.
ayardley authored
194 =head1 IDEAL TESTS:
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
195
196 =over 4
197
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
198 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
199
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
200 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
201 code they're testing. These should include potentially out of band input
202 unless we decide that compilers should check for this themselves.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
203
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
204 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
205
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
206 Are small and self contained, so that if the tested feature breaks we can
207 identify where and why quickly.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
208
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
209 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
210
19b32a9 Minor edits
Simon Glover authored
211 Are valid. Essentially, they should conform to the additional documentation
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
212 that accompanies the feature (if any). [If there isn't any documentation, then
213 feel free to add some and/or complain to the mailing list].
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
214
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored
215 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored
216
217 Are a chunk of assembler and a chunk of expected output.
218
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored
219 =back
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
220
983fe83 @ayardley Added DESCRIPTION and upcased '=head1' cps.
ayardley authored
221 =head1 TODO TESTS
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
222
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored
223 In test driven development, tests are implemented first. So the tests are
224 initially expected to fail. This can be expressed by marking the tests as
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
225 TODO. See L<Test::More> on how to do that.
226
983fe83 @ayardley Added DESCRIPTION and upcased '=head1' cps.
ayardley authored
227 =head1 SKIP TESTS
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored
228
229 TODO test actually executed, so that unexpected success can be detected.
230 In the case of missing requirements and in the case of serious breakdowns
231 the execution of tests can be skipped.
232 See L<Test::More> on how to do that.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
233
234 =head1 SEE ALSO
235
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored
236 L<http://qa.perl.org/>
89a54fe @bschmalhofer More elaboration in doc/tests.pod.
bschmalhofer authored
237 L<http://testanything.org/>
238 L<http://en.wikipedia.org/wiki/Test_Anything_Protocol>
46ec5ef @leto [doc] Linkify
leto authored
239 L<https://github.com/parrot/parrot/blob/master/t/TESTS.STATUS.pod>
240 L<https://github.com/parrot/parrot/blob/master/t/README.pod>
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored
241
242 =cut
Something went wrong with that request. Please try again.