Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Tree: b4f577d0d6
Fetching contributors…

Cannot retrieve contributors at this time

209 lines (120 sloc) 4.485 kB

Parrot Testing Status

The primary goal of the Parrot testing effort is to provide complete coverage of the critical Parrot core subsystems on all targeted platforms. Secondary goals include testing of non-critical core subsystems, and non-core code. This document tracks the status of our effort.

Until we have an automated way of quantitatively tracking test coverage, coverage will be tracked quantitatively, using the scale described below:

Poor

Very little, or no coverage of a component. This status is completely unacceptable and should be remedied immediately.

Fair

Some coverage of basic component operation. Missing some subtests of basic operation, failure cases, and invalid input. Component functionality may only be tested on a subset of platforms that provide that functionality.

Good

Full or nearly-full coverage of basic component operation, failure cases, and invalid input. Tests are run on all platforms which allow component operation. Hard-to-test code is not yet well-covered. Some missing functionality is partially tested with failing TODO and SKIP tests.

Excellent

Full coverage of basic component operation, failure cases, and invalid input on all platforms. Hard-to-test code is well-covered. Missing component functionality, including basic operation, failure cases, and invalid input is tested with failing TODO and SKIP tests.

Parrot Core Subsystems

Below is a breakdown of the Parrot core subsystems, and a qualitative measure of testing status.

Parser

This module tokenizes the input.

Status

Fair. There are tests for basic operation, however these tests are spread throughout the test suite, and are not comprehensive. We can assume basic operation is well-tested, otherwise the compiler, optimizer, and interpreter probably wouldn't work very well. Tests under t/compilers/imcc/syn/, and others.

Compilers: IMCC, PGE, TGE

Status

Good.

Optimizer

This rearranges PASM code to make it run faster.

Status

Untested. IMCC no longer has the ability to output PASM code, only PBC.

Interpreter

This is the core of Parrot. It comprises the various runcores, input/output, threading, events, exceptions, objects, bytecode loader, etc. Test coverage of the Parrot Interpreter is critical.

Status

Fair. A breakdown of Interpreter subsystems below lists individual status.

PMCs

Fair. Each PMC file has a corresponding test file. Some PMCs are heavily undertested. Tests under t/pmc/.

Operators

Good. Most operators are well-tested. Tests under t/op/.

IO

Fair. Tests exist, but do not run on all platforms, and do not test some important cases. Tests in t/pmc/io.t and t/src/io.t.

Dynamic PMCs

Poor. Some dynamic PMCs do not have test files. Coverage in existing test files is far from complete. Tests under t/dynpmc/.

Dynamic ops

Poor. Only a few tests exist, and coverage of important test cases is weak. Tests under t/dynops/.

Packfiles

Poor. For more exotic platforms tests under t/native_pbc are rarely updated.

Runtime Libraries and Includes

Status

Poor. Some runtime libraries are tested, while others are not. No tests exist for include files. Tests under t/library/.

Configure Libraries

Status

Good. Tests exist, and the coverage is decent and improving. Tests under t/configure/.

Test Libraries

Status

Poor. Few tests exist, and coverage is poor. However, the test suite seems to run fine, giving a false sense of security.

Documentation

Status

Fair. Opcode documentation is well-tested, and Pod files are checked for standards-conformance, but many documentation tests are missing. Tests under t/doc.

Coding standard

Status

Good. Tests under t/codingstd/*.t.

Examples and Benchmarks

Status

Fair. Tests exist, and cover basic functionality of nearly all examples and benchmarks. Extensive testing of failures is not a priority. Tests under t/examples/ and t/benchmark/.

Tools

Status

Poor. Few tests exist, and coverage is sparse. Tests under t/tools/.

Jump to Line
Something went wrong with that request. Please try again.