Perl 6 test suite
Perl 6 Perl
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
3rdparty Add new test S15-nfg/concat-stable.t Jul 11, 2017
APPENDICES Cover rat-to-large-power failurage in SAP Jul 31, 2018
MISC Spec `undefine` is deprecated in 6.d Aug 1, 2018
S01-perl-5-integration Allow tests to be run outside of rakudo dir Feb 23, 2018
S02-lexical-conventions Avoid some warnings about redeclarations May 26, 2018
S02-lists [v6.d REVIEW] Use clearer test description Jul 15, 2018
S02-literals [v6.d REVIEW] Move out impl. specific warning test Jul 29, 2018
S02-magicals Add tests for aliasing, or not, of $_ in various block syntaxes Feb 24, 2018
S02-names-vars [JVM] Unfudge now passing tests May 26, 2018
S02-names Add tests for R#2058 Jul 26, 2018
S02-one-pass-parsing Allow tests to be run outside of rakudo dir Feb 23, 2018
S02-packages prefer RT #ticket Nov 1, 2015
S02-types Adapt to slightly changed (|) semantics Aug 1, 2018
S03-binding Add tests for R#2130 Jul 26, 2018
S03-feeds remove niecza implementation fudges Feb 16, 2017
S03-junctions [v6.d REVIEW] Fix typo in test description Aug 4, 2018
S03-metaops [v6.d REVIEW] Test exact type in arity-0 Z test Jul 28, 2018
S03-operators Add in-range but not elem test for ∈ Range:D Aug 15, 2018
S03-sequence Test … op does not prematurely cause consumtion of Seqs Apr 18, 2018
S03-smartmatch Merge pull request #423 from jsimonet/master Jul 26, 2018
S04-blocks-and-statements [v6.d REVIEW] Use better test for Code.file Jul 15, 2018
S04-declarations [v6.d REVIEW] simplify test matcher Jul 14, 2018
S04-exception-handlers Allow tests to be run outside of rakudo dir Feb 23, 2018
S04-exceptions Rename JSON excp. handler test file to more generic one Jul 30, 2018
S04-phasers [v6.d REVIEW] Fix switched up test descriptions Jul 7, 2018
S04-statement-modifiers Tests for RT #131593. Jun 19, 2017
S04-statement-parsing s/lives_ok/lives-ok/g May 20, 2015
S04-statements [v6.d REVIEW] Test exact types in list comprehension Jul 28, 2018
S05-capture tie back to RT Jun 21, 2018
S05-grammar [v6.d REVIEW] Move grammar error text test out of roast Jul 15, 2018
S05-interpolation Allow tests to be run outside of rakudo dir Feb 23, 2018
S05-mass [JVM] Unfudge passing tests May 8, 2018
S05-match Add test for R#2057 Jul 26, 2018
S05-metachars Test to cover RT #130812. Feb 22, 2017
S05-metasyntax [v6.d REVIEW] Test exact types in .comb/match Jul 28, 2018
S05-modifier [v6.d REVIEW] Add another test for trailing ws in :s Aug 4, 2018
S05-nonstrings Remove any Pugs fudges Jul 11, 2014
S05-substitution [v6.d REVIEW] Move NoMulti .subst test to appendix Jul 28, 2018
S05-syntactic-categories Expect Exception, not X::AdHoc. Nov 12, 2015
S05-transliteration Allow tests to be run outside of rakudo dir Feb 23, 2018
S06-advanced Cover --> Foo:D with containerized Nil Aug 11, 2018
S06-currying Test Slurpy Positional params with .assuming Jun 15, 2018
S06-macros Use same ticket for skipped test Sep 10, 2017
S06-multi [jvm] Unfudge some tests Dec 21, 2017
S06-operator-overloading Put earlier-existing :sym reservation tests together Jul 19, 2018
S06-other [v6.d REVIEW] Loosen text content check in error tests Jul 19, 2018
S06-parameters [v6.d REVIEW] Compactify test code Dec 25, 2017
S06-routine-modifiers Unfudge now-passing Proxy tests Jul 9, 2018
S06-signature Cover crash in a slurpy slippage Aug 10, 2018
S06-traits Remove over-specific and dubious error test Jul 9, 2018
S07-hyperrace Cover hyper.grep perf regression Apr 18, 2018
S07-iterators [v6.d REVIEW] Remove pull test in skip-at-least Jul 28, 2018
S07-slip Test itemized Slips roundtrip .perl.EVAL Dec 14, 2016
S09-autovivification Remove old comment Sep 11, 2017
S09-hashes Avoid some warnings about redeclarations May 26, 2018
S09-multidim [v6.d REVIEW] Test return types in .rotate/.reverse… Jul 28, 2018
S09-subscript Test actual sliced data structures… Aug 6, 2018
S09-typed-arrays [v6.d REVIEW] Extend/improve object Hash cover tests Jul 21, 2018
S10-packages [v6.d REVIEW] Improve priv is rw attr package use test Jul 15, 2018
S11-compunit [JVM] Fix typo in implementation name May 22, 2018
S11-modules [JVM] Unfudge passing tests May 8, 2018
S11-repository Test installation with proper Distribution usage May 12, 2018
S12-attributes Correct tests using return instead of return-rw Jul 9, 2018
S12-class Add tests for R#2083 Jul 26, 2018
S12-coercion Add tests for RT #131414 May 10, 2018
S12-construction Test for R#2174 Aug 4, 2018
S12-enums [v6.d REVIEW] Extend enum cover tests Jul 21, 2018
S12-introspection [v6.d REVIEW] Simplify Attribute.gist test Jul 15, 2018
S12-meta Allow tests to be run outside of rakudo dir Feb 23, 2018
S12-methods Update class-and-instance.t to add tests for class and instance Apr 5, 2018
S12-subset [v6.d REVIEW] Refine capture `where` test Jul 18, 2018
S12-traits Replace remaining eval-dies-ok with throws-like Oct 1, 2015
S13-overloading Allow tests to be run outside of rakudo dir Feb 23, 2018
S13-syntax s/lives_ok/lives-ok/g May 20, 2015
S13-type-casting remove niecza implementation fudges Feb 16, 2017
S14-roles Add test for RT#126724 May 10, 2018
S14-traits Revert "Fudge RT #129375 for rakudo.jvm" Sep 28, 2016
S15-literals Allow tests to be run outside of rakudo dir Feb 23, 2018
S15-nfg Add tests for Regional Indicator concat and join Aug 13, 2018
S15-normalization Commit the changes to the normalization test-gen.p6 Jul 22, 2017
S15-string-types Fix skip reason for nfd/nfc adverb tests Jul 16, 2017
S15-unicode-information [v6.d REVIEW] Use better test routine for Nil uniprop Jul 29, 2018
S16-filehandles [v6.d REVIEW] Improve .printf tests Jul 28, 2018
S16-io [v6.d REVIEW] Remove cruft and Perl-5-isms from .eof test Jul 8, 2018
S16-unfiled Allow tests to be run outside of rakudo dir Feb 23, 2018
S17-channel Make a stress test a little less stressful. Feb 28, 2017
S17-lowlevel [v6.d REVIEW] Harden Semaphore array increment test Jul 8, 2018
S17-procasync [v6.d REVIEW] Reword Proc::Async empty output test Jul 28, 2018
S17-promise [JVM] Unfudge some passing tests Mar 8, 2018
S17-scheduler Adapt tests because of 2a635cd96279ace1c5d3d6d687 Dec 6, 2015
S17-supply [JVM] Some more fudges for Proc::Async May 29, 2018
S19-command-line-options [JVM] Unfudge passing tests for eof on stdin Mar 5, 2018
S19-command-line Allow tests to be run outside of rakudo dir Feb 23, 2018
S22-package-format S22-package-format/local.t: use $*SPEC.catdir() to account for backsl… Dec 21, 2015
S24-testing [JVM] Unfudge now passing tests May 8, 2018
S26-documentation add tests for =defn implementation (#457) Aug 2, 2018
S28-named-variables Fix typo Nov 5, 2017
S29-any [v6.d REVIEW] Remove useless Any:U.*pend/*shift/push/pop tests Jul 15, 2018
S29-context [JVM] Another case of equal Bufs not recognized May 17, 2018
S29-conversions Use better ticket for fudge message Mar 19, 2017
S29-os [v6.d REVIEW] Don't rely on throws-like() sinking Jul 15, 2018
S32-array [v6.d REVIEW] Polish Array.splice tests Jul 28, 2018
S32-basics [v6.d REVIEW] Fix up .pairup throwage test Jul 28, 2018
S32-container [v6.d REVIEW] Don't spec stringification of Blob.WHICH Jul 15, 2018
S32-encoding Unfudge now-passing test Aug 8, 2018
S32-exceptions Add test for R#2111 Jul 23, 2018
S32-hash Test .Map doesn't introduce bogus itemization Aug 13, 2018
S32-io [v6.d REVIEW] Use more future-safe test values for family Aug 6, 2018
S32-list [v6.d REVIEW] Also test :partial in empty-list rotor Jul 28, 2018
S32-num Cover largish Int math on hex, bin, and octals Aug 1, 2018
S32-scalar Allow tests to be run outside of rakudo dir Feb 23, 2018
S32-str Document the assumptions in translate-nl test Aug 8, 2018
S32-temporal [v6.d REVIEW] Improve DateTime.Date test desc Jul 21, 2018
S32-trig remove niecza implementation fudges Feb 16, 2017
docs Add file for documenting Unicode generated test files Oct 1, 2017
integration [v6.d REVIEW] Clarify race-y SEGV tests Jul 15, 2018
packages Make `doesn't-hang` wait longer Jul 30, 2018
rosettacode Fix Windows line ending workarounds. Nov 3, 2015
t Fix space-safety of path usage Feb 24, 2018
.gitignore ignore .precomp folders Dec 11, 2015
CONTRIBUTING.md Use implementation-neutral language Apr 8, 2018
LICENSE Add Artistic License 2 Nov 5, 2013
README.md Introduce specification APPENDICES Jul 15, 2018
TODO Add a note about RTs. Jul 20, 2014
VERSION “nom” → “master” Oct 26, 2017
fudge Toss is_run_repl helper routine Oct 7, 2017
fudgeall Fix globbing on windows Nov 18, 2017
fudgeandrun Propagate exit code Oct 11, 2017
super-fudger.p6 Add super-fudger.p6, to fudge ALL failing tests in a .t file Dec 24, 2016
test_summary avoid unnecessary differences in output Mar 4, 2015

README.md

The Official Perl 6 Test Suite

The purpose of the test suite is to validate implementations that wish to be known as a conforming Perl 6 implementation.

Contents

Introduction

Please consider this test suite to be the bleeding edge of Perl 6 development. New tests, tests for proposed new features, etc., will live on this 'master' branch. Once a specification is cut, a branch will be created for that version of the spec, e.g., 6.c for Christmas language release. If errata becomes available, it will be kept in -errata branch for that specific language version, e.g. 6.c-errata for 6.c.

As they develop, different implementations will certainly be in different states of readiness with respect to the test suite, so in order for the various implementations to track their progress independently, we've established a mechanism for fudging the tests in a kind of failsoft fashion. To pass a test officially, an implementation must be able to run a test file unmodified, but an implementation may (temporarily) skip tests or mark them as "todo" via the fudging mechanism, which is implemented via the fudge preprocessor. Individual implementations are not allowed to modify the actual test code, but may insert line comments before each actual test (or block of tests) that changes how those tests are to be treated for this platform. The fudge preprocessor pays attention only to the comments that belong to the current implementation and ignores all the rest. If your implementation is named "rakudo" then your special comment lines look like:

#?rakudo: [NUM] VERB ARGS

(The colon is optional.)

The optional NUM says how many statements or blocks to apply the verb to. (If not supplied, a value of 1 is assumed). A statement is arbitrarily defined as one or more lines starting with a test call and ending in a semicolon (with an optional comment).

VERBs include:

skip "reason"	# skip test entirely
eval "reason"	# eval the test because it doesn't parse yet
try "reason"	# try the test because it throws exception
todo "reason"	# mark "todo" because "not ok" is expected
emit code		# insert code (such as "skip-rest();") inline

All fudged tests return an exit code of 1 by default, so the test harness will mark it as "dubious" even if all the tests supposedly pass.

You may also negate the test:

#!rakudo [NUM] VERB ARGS

This will apply the verb on any system that isn't rakudo.

Sometimes environment variables distinguish syntactic or semantic variants, so you may apply a verb depending on the presence or absence of such a setting:

#?MYSPECIALVAR [NUM] VERB ARGS
#!MYSPECIALVAR [NUM] VERB ARGS

The environment variable must be uppercase.

There is also the following directive which modifies the test count of the next construct:

#?DOES count

The count may be an expression as long as any variables referenced in the expression are in scope at the location fudge eventually inserts a "skip()" call.

When applied to a subsequent sub definition, fudge registers the sub name as doing that many tests when called. Note, however, that any skipping is done at the point of the call, not within the subroutine, so the count may not refer to any parameter of the sub.

When you run the fudge preprocessor, if it decides the test needs fudging, it returns the new fudged filename; otherwise it returns the original filename. (Generally you don't run fudge directly, but your test harness runs the fudgeall program for you; see below.) If there is already a fudged program in the directory that is newer than the unfudged version, fudge just returns the fudged version without regenerating it. If the fudged version is older, it removes it and then decides anew whether to regenerate it based on the internal fudge comments.

The fudgeall program may be called to process all the needed fudging for a particular implementation:

$ fudgeall rakudo */*.t */*/*.t

Program fudgeall will use the fudge program to translate any fudged files to a new file where the extension is not *.t but instead is *.rakudo to indicate the implementation dependency. It also returns the fudged list of filenames to run, where unfudged tests are just passed through unchanged as *.t. Each test comes through as either fudged or not, but never both. The test harness then runs the selected test files as it normally would (it shouldn't care whether they are named *.t or *.rakudo).

In cases where the current working directory makes a difference, the tests assume that the working directory is the root of the test suite, so that the relative path to itself is t/spec/S\d\d-$section/$filename.

To fudge and run prove on a specific file:

./fudgeandrun S05-metasyntax/angle-brackets.t

fudgeandrun does not assume any particular implementation but guesses by running perl6 to look at special variables like $*PERL. See fudgeandrun usage to specify a different implementation and other options.

Test Utilities

This repository contains Test::Util module with helper routines you can use when writing tests. See POD documentation included at the end of the module's source code. To include the module, in your test file, you need to add use lib line to your test file.

use lib $?FILE.IO.parent(2).add: 'packages';
use Test::Util;
use Test;

Depending on the location of your test file, the number inside .parent(2) may need to be adjusted to go up the correct number of times from the test files's location to the root of the repository.

Appendices

The APPENDICES directory contains advisory tests implentations may optionally choose to follow. See the README included in that directory for more information.

Environmental Variables

  • ROAST_TIMING_SCALE

Some tests rely on a process to complete in a certain amount of time. If you're running on a slowish computer, try setting ROAST_TIMING_SCALE to a larger value that will be used as a multiplier for the time to wait. We don't wait for too long a time by default so as to make the roast run faster. Defaults to 1.

Contributing

See CONTRIBUTING.md