Perl 6 test suite
Perl6 Other
Permalink
Failed to load latest commit information.
S01-perl-5-integration remove niecza implementation fudges Feb 16, 2017
S02-lexical-conventions remove mildew implementation fudges Feb 16, 2017
S02-lists Test ()[0] returns Nil Oct 28, 2016
S02-literals remove niecza implementation fudges Feb 16, 2017
S02-magicals remove niecza implementation fudges Feb 16, 2017
S02-names-vars remove niecza implementation fudges Feb 16, 2017
S02-names remove niecza implementation fudges Feb 16, 2017
S02-one-pass-parsing Fix incorrect exit code in test Aug 19, 2016
S02-packages prefer RT #ticket Nov 1, 2015
S02-types Correct test sensitive to hash ordering. Feb 22, 2017
S03-binding remove niecza implementation fudges Feb 16, 2017
S03-feeds remove niecza implementation fudges Feb 16, 2017
S03-junctions remove niecza implementation fudges Feb 16, 2017
S03-metaops remove niecza implementation fudges Feb 16, 2017
S03-operators remove niecza implementation fudges Feb 16, 2017
S03-sequence remove niecza implementation fudges Feb 16, 2017
S03-smartmatch remove niecza implementation fudges Feb 16, 2017
S04-blocks-and-statements remove niecza implementation fudges Feb 16, 2017
S04-declarations remove niecza implementation fudges Feb 16, 2017
S04-exception-handlers Unfudge some passing tests on JVM Dec 31, 2016
S04-exceptions Test `sink` statement prefix explodes Failures Dec 8, 2016
S04-phasers remove niecza implementation fudges Feb 16, 2017
S04-statement-modifiers remove niecza implementation fudges Feb 16, 2017
S04-statement-parsing s/lives_ok/lives-ok/g May 20, 2015
S04-statements Tests to cover RT #130825. Feb 22, 2017
S05-capture remove niecza implementation fudges Feb 16, 2017
S05-grammar remove niecza implementation fudges Feb 16, 2017
S05-interpolation remove niecza implementation fudges Feb 16, 2017
S05-mass remove niecza implementation fudges Feb 16, 2017
S05-match remove niecza implementation fudges Feb 16, 2017
S05-metachars Test to cover RT #130812. Feb 22, 2017
S05-metasyntax remove niecza implementation fudges Feb 16, 2017
S05-modifier remove kp6 implementation fudges Feb 16, 2017
S05-nonstrings Remove any Pugs fudges Jul 11, 2014
S05-substitution remove niecza implementation fudges Feb 16, 2017
S05-syntactic-categories Expect Exception, not X::AdHoc. Nov 12, 2015
S05-transliteration remove niecza implementation fudges Feb 16, 2017
S06-advanced remove niecza implementation fudges Feb 16, 2017
S06-currying remove niecza implementation fudges Feb 16, 2017
S06-macros Fix test where eval-lives-ok didn't get right context Dec 10, 2015
S06-multi remove niecza implementation fudges Feb 16, 2017
S06-operator-overloading remove niecza implementation fudges Feb 16, 2017
S06-other remove niecza implementation fudges Feb 16, 2017
S06-parameters Expect Exception, not X::AdHoc. Nov 12, 2015
S06-routine-modifiers remove niecza implementation fudges Feb 16, 2017
S06-signature remove niecza implementation fudges Feb 16, 2017
S06-traits Test error when assigning to ro var has the name Nov 30, 2016
S07-hyperrace .hyper preserves order Sep 10, 2016
S07-iterators Test Iterator.skip-at-least-pull-one Nov 10, 2016
S07-slip Test itemized Slips roundtrip .perl.EVAL Dec 14, 2016
S09-autovivification remove niecza implementation fudges Feb 16, 2017
S09-hashes Fix too strict tests for empty hash listification Jan 23, 2017
S09-multidim Fudge some more failing tests for JVM Jan 1, 2017
S09-subscript remove niecza implementation fudges Feb 16, 2017
S09-typed-arrays remove niecza implementation fudges Feb 16, 2017
S10-packages remove niecza implementation fudges Feb 16, 2017
S11-compunit Use $*EXECUTABLE instead of 'perl6', fudge for JVM Dec 17, 2015
S11-modules remove niecza implementation fudges Feb 16, 2017
S11-repository Fix return value of CompUnit::Handle::globalish-package Jan 28, 2017
S12-attributes remove niecza implementation fudges Feb 16, 2017
S12-class remove niecza implementation fudges Feb 16, 2017
S12-coercion Use ticket number RT #126524 for fudging Oct 30, 2016
S12-construction remove niecza implementation fudges Feb 16, 2017
S12-enums remove niecza implementation fudges Feb 16, 2017
S12-introspection remove niecza implementation fudges Feb 16, 2017
S12-meta Test for RT 125135 Jan 3, 2017
S12-methods remove niecza implementation fudges Feb 16, 2017
S12-subset remove niecza implementation fudges Feb 16, 2017
S12-traits Replace remaining eval-dies-ok with throws-like Oct 1, 2015
S13-overloading remove niecza implementation fudges Feb 16, 2017
S13-syntax s/lives_ok/lives-ok/g May 20, 2015
S13-type-casting remove niecza implementation fudges Feb 16, 2017
S14-roles remove niecza implementation fudges Feb 16, 2017
S14-traits Revert "Fudge RT #129375 for rakudo.jvm" Sep 28, 2016
S15-literals Conversion of 'No' chars to numeric is not supported Jul 7, 2016
S15-nfg Test to cover SEGV in RT #129227. Feb 22, 2017
S15-normalization Update S15-normalization tests for Unicode 9.0 Jan 4, 2017
S15-string-types remove niecza implementation fudges Feb 16, 2017
S15-unicode-information remove niecza implementation fudges Feb 16, 2017
S16-filehandles remove niecza implementation fudges Feb 16, 2017
S16-io remove niecza implementation fudges Feb 16, 2017
S16-unfiled fixed appended RT number to be within quoted string May 5, 2015
S17-channel Fudge newly added tests for JVM Nov 3, 2016
S17-lowlevel Correct test that did unprotected array writes. Aug 10, 2016
S17-procasync No need to quote a variable Dec 8, 2016
S17-promise Test for RT #128833. Feb 15, 2017
S17-scheduler Adapt tests because of 2a635cd96279ace1c5d3d6d687 Dec 6, 2015
S17-supply Fudge test for Supplier::Preserving on JVM Feb 16, 2017
S19-command-line-options Add some todo'd tests for RT #129093 Dec 19, 2016
S19-command-line Test `Mu` as line return value isn't an issue in REPL Feb 5, 2017
S22-package-format S22-package-format/local.t: use $*SPEC.catdir() to account for backsl… Dec 21, 2015
S24-testing Relax whitespace requirement before "#TODO" in test output Jan 2, 2017
S26-documentation Fudge newly introduced tests; switch to subtests Jan 28, 2017
S28-named-variables the Quasi and Trans slangs do not, in fact, exist Feb 10, 2017
S29-any [coverage] Add tests for Any.[nl-out|print-nl] Sep 20, 2016
S29-context remove niecza implementation fudges Feb 16, 2017
S29-conversions remove niecza implementation fudges Feb 16, 2017
S29-os Use ticket number RT #126524 for fudging Oct 30, 2016
S32-array remove niecza implementation fudges Feb 16, 2017
S32-basics remove niecza implementation fudges Feb 16, 2017
S32-container remove niecza implementation fudges Feb 16, 2017
S32-exceptions Unfudge now passing test Jan 9, 2017
S32-hash remove niecza implementation fudges Feb 16, 2017
S32-io remove niecza implementation fudges Feb 16, 2017
S32-list remove niecza implementation fudges Feb 16, 2017
S32-num remove niecza implementation fudges Feb 16, 2017
S32-scalar remove niecza implementation fudges Feb 16, 2017
S32-str Test .lines/.words with wrong args do not hang Feb 22, 2017
S32-temporal remove niecza implementation fudges Feb 16, 2017
S32-trig remove niecza implementation fudges Feb 16, 2017
integration remove niecza implementation fudges Feb 16, 2017
packages Add some tests for iterator exhaustion Jan 27, 2017
rosettacode Fix Windows line ending workarounds. Nov 3, 2015
t Add implementation independent fudgeandrun with test driver t/fudgean… Feb 13, 2017
.gitignore ignore .precomp folders Dec 11, 2015
CONTRIBUTING.md Move "contributing" doc to its own file Nov 14, 2016
LICENSE Add Artistic License 2 Nov 5, 2013
README.md Update README.md to mention fudgeandrun (#234) Feb 15, 2017
TODO Add a note about RTs. Jul 20, 2014
fudge Replace deprecated is_approx with is-approx Oct 19, 2016
fudgeall Add implementation independent fudgeandrun with test driver t/fudgean… Feb 13, 2017
fudgeandrun fudgeandrun - remove error report for valid non-zero exit code Feb 20, 2017
super-fudger.p6 Add super-fudger.p6, to fudge ALL failing tests in a .t file Dec 24, 2016
test_summary avoid unnecessary differences in output Mar 4, 2015

README.md

The Official Perl 6 Test Suite

The purpose of the test suite is to validate implementations that wish to be known as a conforming Perl 6 implementation.

Contents

Introduction

Please consider this test suite to be the bleeding edge of Perl 6 development. New tests, tests for experimental new features, etc., will live on this 'master' branch. Once a specification is cut, a branch will be created for that version of the spec, e.g., 6.c for Christmas.

As they develop, different implementations will certainly be in different states of readiness with respect to the test suite, so in order for the various implementations to track their progress independently, we've established a mechanism for fudging the tests in a kind of failsoft fashion. To pass a test officially, an implementation must be able to run a test file unmodified, but an implementation may (temporarily) skip tests or mark them as "todo" via the fudging mechanism, which is implemented via the fudge preprocessor. Individual implementations are not allowed to modify the actual test code, but may insert line comments before each actual test (or block of tests) that changes how those tests are to be treated for this platform. The fudge preprocessor pays attention only to the comments that belong to the current implementation and ignores all the rest. If your implementation is named "rakudo" then your special comment lines look like:

#?rakudo: [NUM] VERB ARGS

(The colon is optional.)

The optional NUM says how many statements or blocks to apply the verb to. (If not supplied, a value of 1 is assumed). A statement is arbitrarily defined as one or more lines starting with a test call and ending in a semicolon (with an optional comment).

VERBs include:

skip "reason"   # skip test entirely
eval "reason"   # eval the test because it doesn't parse yet
try "reason"    # try the test because it throws exception
todo "reason"   # mark "todo" because "not ok" is expected
emit code       # insert code (such as "skip-rest();") inline

All fudged tests return an exit code of 1 by default, so the test harness will mark it as "dubious" even if all the tests supposedly pass.

You may also negate the test:

#!rakudo [NUM] VERB ARGS

This will apply the verb on any system that isn't rakudo.

Sometimes environment variables distinguish syntactic or semantic variants, so you may apply a verb depending on the presence or absence of such a setting:

#?MYSPECIALVAR [NUM] VERB ARGS
#!MYSPECIALVAR [NUM] VERB ARGS

The environment variable must be uppercase.

There is also the following directive which modifies the test count of the next construct:

#?DOES count

The count may be an expression as long as any variables referenced in the expression are in scope at the location fudge eventually inserts a "skip()" call.

When applied to a subsequent sub definition, fudge registers the sub name as doing that many tests when called. Note, however, that any skipping is done at the point of the call, not within the subroutine, so the count may not refer to any parameter of the sub.

When you run the fudge preprocessor, if it decides the test needs fudging, it returns the new fudged filename; otherwise it returns the original filename. (Generally you don't run fudge directly, but your test harness runs the fudgeall program for you; see below.) If there is already a fudged program in the directory that is newer than the unfudged version, fudge just returns the fudged version without regenerating it. If the fudged version is older, it removes it and then decides anew whether to regenerate it based on the internal fudge comments.

The fudgeall program may be called to process all the needed fudging for a particular implementation:

$ fudgeall rakudo */*.t */*/*.t

Program fudgeall will use the fudge program to translate any fudged files to a new file where the extension is not *.t but instead is *.rakudo to indicate the implementation dependency. It also returns the fudged list of filenames to run, where unfudged tests are just passed through unchanged as *.t. Each test comes through as either fudged or not, but never both. The test harness then runs the selected test files as it normally would (it shouldn't care whether they are named *.t or *.rakudo).

In cases where the current working directory makes a difference, the tests assume that the working directory is the root of the test suite, so that the relative path to itself is t/spec/S\d\d-$section/$filename.

To fudge and run prove on a specific file:

./fudgeandrun S05-metasyntax/angle-brackets.t

fudgeandrun does not assume any particular implementation but guesses by running perl6 to look at special variables like $*PERL. See fudgeandrun usage to specify a different implementation and other options.

Environmental Variables

  • ROAST_TIMING_SCALE

Some tests rely on a process to complete in a certain amount of time. If you're running on a slowish computer, try setting ROAST_TIMING_SCALE to a larger value that will be used as a multiplier for the time to wait. We don't wait for too long a time by default so as to make the roast run faster. Defaults to 1.

Contributing

See CONTRIBUTING.md