Skip to content
Newer
Older
100644 189 lines (130 sloc) 5.39 KB
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored Mar 4, 2005
1 # Copyright: 2001-2005 The Perl Foundation. All Rights Reserved.
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
2 # $Id$
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored Aug 27, 2002
3
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
4 =head1 NAME
5
6 docs/tests.pod - Testing Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
7
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
8 =head1 A basic guide to writing and running tests for Parrot
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
9
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
10 This is quick and dirty pointer to how the Parrot test suite is executed and
11 to how new tests for Parrot should be written.
12 The testing system is liable to change in the future, but tests written
13 following the guidelines below should be easy to port into a new test suite.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
14
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
15 =head1 How to test parrot
16
17 The easy way to test parrot is running C<make test>. If you have
18 updated your code recently and tests began failing, go for a C<make
19 realclean> and recompile parrot before complaining.
20
21 If your architecture supports JIT, you can test parrot JIT engine using C<make
22 testj>. It works just like C<make test>, but uses the JIT engine when possible.
23
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored Dec 12, 2005
24 C<make languages-test> runs the test suite for most language implementations
25 in the languages directory.
26
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
27 =head2 Submitting smoke test results
28
29 Parrot has a status page with smoke test results
38d4ce0 @coke Fix the URL to the smoke server.
coke authored Dec 12, 2005
30 L<http://smoke.parrotcode.org/smoke/>. You can supply new tests
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
31 results just running C<make smoke>. It will run the same tests as
32 C<make test> would, but creating a HTML table with the test
33 results. At the end, it will try to upload the test results to the
34 smoke server.
35
36 It is also possible to run a smoke test on JIT. For that, try running
8710563 @jhoblitt add smokej jit smoke test target
jhoblitt authored Dec 13, 2005
37 C<make smokej>.
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
38
c8ca9f0 @bschmalhofer Mention the make targets 'languages-test' and 'languages-smoke'
bschmalhofer authored Dec 12, 2005
39 C<make languages-smoke> does smoke testing for most language implementations
40 in the languages directory.
41
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
42 =head1 Location of the test files
43
44 The parrot test files, the F<*.t> files, can be found in the F<t> directory.
45 A quick overview over the subdirs in F<t> can be found in F<t/README>.
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored Feb 26, 2004
46
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
47 The language implementations usually have their test files in F<languages/*/t>.
48
49 New tests should be added to an existing F<*.t> file.
50 If a previously untested feature is tested,
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
51 it might also make sense to create a new F<*.t> file.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
52
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
53 =head1 How to write a test
54
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
55 The testing framework needs to know how many tests it should expect. So the
56 number of planned tests needs to be incremented when adding a new test. This
57 is done near the top of a test file, in a line that looks like:
553e254 Patch from Simon Glover <scog@amnh.org>:
Nicholas Clark authored Mar 19, 2002
58
59 use Parrot::Test tests => 8;
60
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
61 =head2 Parrot Assembler
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
62
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
63 PASM tests are mostly used for testing ops. Appropriate test files for basic
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
64 ops are F<t/op/*.t>. Perl Magic Cookies are tested in F<t/pmc/*.t>. Add the
65 new test like this:
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
66
b38c8aa @bschmalhofer Some fiddling with test suite:
bschmalhofer authored Mar 4, 2005
67 pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
68 *** a big chunk of assembler, eg:
69 print 1
70 print "\n" # you can even comment it if it's obscure
71 end # don't forget this...!
72 CODE
73 *** what you expect the output of the chunk to be, eg.
74 1
75 OUTPUT
76
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
77 =head2 Parrot Intermediate Representation
78
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored Mar 14, 2006
79 Writing tests in B<PIR> is more convenient. This is done with C<pir_output_is> and
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
80 friends.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
81
82 pir_output_is(<<'CODE',<<'OUT','nothing useful');
eb9e7dd @particle #37520: [TODO] rename library files from .imc to .pir
particle authored Dec 28, 2005
83 .include 'library/config.pir'
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
84
3721ccc [PATCH] @directive -> :directive - part 2 src, docs, tools
Leopold Toetsch authored Oct 11, 2005
85 .sub main :main
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
86 print "hi\n"
87 .end
88 CODE
89 hi
90 OUT
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored Feb 26, 2004
91
92 =head2 C source tests
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
93
ea0630c @jhoblitt reformat all Pod files under docs with podtidy (modified to also remo…
jhoblitt authored Oct 5, 2005
94 C source tests are usually located in F<t/src/*.t>. A simple test looks like:
280b750 @jhoblitt fix verbatim block in docs/tests.pod
jhoblitt authored Oct 5, 2005
95
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
96 c_output_is(<<'CODE', <<'OUTPUT', "name for test");
97 #include <stdio.h>
98 #include "parrot/parrot.h"
99 #include "parrot/embed.h"
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
100
101 static opcode_t *the_test(Parrot_Interp, opcode_t *, opcode_t *);
102
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
103 int main(int argc, char* argv[]) {
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
104 Parrot_Interp interpreter;
01f4e37 Updated doc and some code to reflect new Parrot_new() interface
Jens Rieks authored Sep 16, 2004
105 interpreter = Parrot_new(NULL);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
106
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
107 if (!interpreter)
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
108 return 1;
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
109
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
110 Parrot_init(interpreter);
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
111 Parrot_run_native(interpreter, the_test);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
112 printf("done\n");
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
113 fflush(stdout);
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
114 return 0;
115 }
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
116
117 static opcode_t*
118 the_test(Parrot_Interp interpreter,
9d8cce4 a bit more tidying up
Michael Scott authored Feb 28, 2004
119 opcode_t *cur_op, opcode_t *start)
58c0aa8 update info on src tests
Leopold Toetsch authored Feb 26, 2004
120 {
121 /* Your test goes here. */
122
123 return NULL; /* always return NULL */
124 }
c543d6f @boemmels Add C test info to tests.pod (Courtesy of Michael Scott <michael_scot…
boemmels authored Sep 30, 2003
125 CODE
126 # Anything that might be output prior to "done".
127 done
128 OUTPUT
129
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
130 Note that it's always a good idea to output "done" to confirm that the compiled
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
131 code executed completely. When mixing C<printf> and C<PIO_printf> always append
132 a C<fflush(stdout);> after the former.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
133
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored Mar 14, 2006
134 =head2 Test Perl5 helpers
135
136 Perl5 unit tests are in F<t/perl/*.t>.
137
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
138 =head2 Testing language implementations
139
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
140 Language implementations are usually tested with the test function
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
141 C<language_output_is>.
142
1acea25 update info on pir tests in main; don't generate packfile-perl.pod
Leopold Toetsch authored Feb 26, 2004
143 =head1 Ideal tests:
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
144
145 =over 4
146
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored Dec 4, 2005
147 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
148
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
149 Probe the boundaries (including edge cases, errors thrown etc.) of whatever
150 code they're testing. These should include potentially out of band input
151 unless we decide that compilers should check for this themselves.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
152
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored Dec 4, 2005
153 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
154
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
155 Are small and self contained, so that if the tested feature breaks we can
156 identify where and why quickly.
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
157
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored Dec 4, 2005
158 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
159
19b32a9 Minor edits
Simon Glover authored May 14, 2003
160 Are valid. Essentially, they should conform to the additional documentation
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
161 that accompanies the feature (if any). [If there isn't any documentation, then
162 feel free to add some and/or complain to the mailing list].
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
163
afc981d @chromatic Fixed some POD formatting errors.
chromatic authored Dec 4, 2005
164 =item *
8c4a44c @quidity This is a basic quide to writing tests with assembler.
quidity authored Nov 7, 2001
165
166 Are a chunk of assembler and a chunk of expected output.
167
9b4e5d0 Fixups to make parrotcode.org's autogenerated pages happier
Dan Sugalski authored Aug 27, 2002
168 =back
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
169
170 =head1 TODO tests
171
05a7e7b @bschmalhofer This patch is huge because of all the whitespace reformatting. This was
bschmalhofer authored Sep 7, 2005
172 In test driven development, tests are implemented first. So the tests are
173 initially expected to fail. This can be expressed by marking the tests as
aab5022 @bschmalhofer docs: Going over docs/tests.pod.
bschmalhofer authored Dec 28, 2005
174 TODO. See L<Test::More> on how to do that.
175
176 =head1 SKIP tests
177
178 TODO test actually executed, so that unexpected success can be detected.
179 In the case of missing requirements and in the case of serious breakdowns
180 the execution of tests can be skipped.
181 See L<Test::More> on how to do that.
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
182
183 =head1 SEE ALSO
184
e48bb2e @chromatic Patch #37887 from Alberto Simoes (ambs@cpan.org), plus spelling corre…
chromatic authored Dec 12, 2005
185 L<http://qa.perl.org/>
b9beb9d @bschmalhofer Explain what t/perl is for.
bschmalhofer authored Mar 14, 2006
186 F<t/TESTS.STATUS.pod>
a754c8c @bschmalhofer Mention 'pir_output_is', 'language_output_is' and TODO in docs/tests.…
bschmalhofer authored Apr 6, 2005
187
188 =cut
Something went wrong with that request. Please try again.