/
README
46 lines (34 loc) · 1.83 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
"Perl6 Roast Data"
This project contains tools for running the perl6 spec test suite
(roast) by several different implementations, as well as tracking the
results of those runs.
See "perl6_pass_rates" for the data.
Each implementation's last run is checked in under a corresponding
".out" file. Check your implementation's file for any failures or
unexpected passes.
"Why don't all tests pass?"
Each implementation shares roast with all the other
implementations. If a test is added, it might not be immediately
fudged to work on all implementations - in fact, depending on what
is being tested, the entire file may start failing in some
implementations.
The test numbers are run daily from a limited cron environment. If
the test is expecting things (like an environment variable) these
assumptions need to be added to the implementation's script in
bin/
Finally, these tests are run on a schedule; it's not expected that
commits to spec and to an implementation will be synchronized
with the running of this harness; give new tests a day to settle in.
If you notice a test is continually failing (check the history of the
implementation's .out file), please open a ticket with that implementation.
"How to Help - Pugs"
Since Pugs is in maintenance mode at this point, any new tests are not
likely to get a patch to work; To keep Pugs green, any failing tests
should be fudged. For most tests, this involves prefixing the test
with:
#?pugs todo
If this still doesn't allow the test file to complete, change the todo
to a skip - this requires a reason; either put in the error diagnostic
or the word "parsefail", e.g.:
#?pugs skip 'No compatible multi variant found: "&is"'
If you still can't get a test file to pass, drop by #perl6.