NAME

Test::X1 - A lightweight test manager

SYNOPSIS

use Test::X1;
use Test::More;

test {
    my $c = shift;
    ok 1;
    ok 2;
    $c->done;
};

test {
    my $c = shift;
    my $timer; $timer = AnyEvent->timer(
        after => 2,
        cb => sub {
            test {
                ok 2;
                is 3, 0;
                undef $timer;
                $c->done;
                undef $c;
            } $c;
        },
    );
    ok 1;
} n => 3, name => ['anyevent', 'callback'];

test {
    my $c = shift;
    ok 1;
    $c->done;
} n => 1;

run_tests;

ok 1 - [1] - [1]
ok 2 - [1] - [2]
ok 3 - [2] anyevent.callback - [1]
ok 4 - [3] - [1]
ok 5 - [2] anyevent.callback - [2]
not ok 6 - [2] anyevent.callback - [3]
#   Failed test '[2] anyevent.callback - [3]'
#   at lib/Test/X1.pod line 24.
#          got: '3'
#     expected: '0'
# [2] anyevent.callback: 1 test failed
1..6
# Looks like you failed 1 test of 6.

DESCRIPTION

The Test::X1 module defines a simple, lightweight, and AnyEvent-compatible test management framework for Test::Builder-based tests. It supports automatic naming of tests, partial execution, concurrent execution, and other useful features.

USAGE

By useing the Test::X1 module, functions for defining and running tests are exported to the caller package. Note that it only exports test management functions. You have to import or define your favorite test assertion sets such as Test::More or Test::Differences by yourself. The Test::X1 test framework only assumes that you are writing a Test::Builder-based test script. As long as test assertion functions are build upon Test::Builder's framework, you can choose any test module.

Note that some test modules are incompatible with AnyEvent-based asynchronous invocation of callbacks. For example, subtests of Test::More does not work well.

Tests

A Test::X1-based test script consists of one or more tests. A test is a group of subtests, which are Test::Builder-based assertion functions such as is and ok. A test can be defined by enclosing subtests and related codes by the test{} function:

test {
    my $c = shift;
    
    is 2 * 4, 8;
    is 40 - 30, 10;
    
    $c->done;
};

This usage of the test{} function is sometimes referred to as test definition. The code block in the test definition is invoked with an argument, the context object for the test, $c. The $c->done method has to be called when all subtests in the test are done. (See "Context objects" for more information on $c.)

You can specify the number of subtests in a test by specifying the n option to the test definition. If the n option is specified, it is verified that the expected number of subtests are done before the $c->done metohd is invoked. It is considered as good practice to specify the number of subtests, especially when there are loops in the test, such that it can be confirmed that all expected subtests are really executed. Since the number of subtests can be frequently changed in some cases, or it could even be unknown a priori, the n option is not required.

The test can be named by specifying the name option to the test definition. The name of the test is [n], where n is the sequential number of the test, followed by space and name value if any, e.g. [1] hoge or [2].

The name of the test is used to construct the name of the subtests in the test.

The TEST_METHOD environment variable can be used to specify the regular expression used to filter the tests to run by their names. If the environment variable is not set, all tests are executed. Otherwise, only the tests whose name matches the pattern are executed.

The TEST_METHOD_EXCLUDED environment variable can be used to specify the regular expression used to filter the tests to not run by their names. If the environment variable is not set, no test is excluded.

See also "Naming" for how names are handled and used by the module.

For example, consider the following test script:

# t/foo.t
use Test::X1;
use Test::More;

my $x = 10;

test {
    my $c = shift;
    is $x, 10;
    $c->done;
} name => 'abc2';

test {
    my $c = shift;
    is $x * 2, 20;
    $c->done;
};

run_tests;

The first test is named as [1] abc2, while the second test is named as [2]. If the script is executed with no TEST_METHOD, both of them are executed. If the script is executed with TEST_METHOD=abc, only the first test is executed. If the script is executed with TEST_METHOD=2, both of them are executed.

The run_tests function runs the tests defined by test{} functions. It must be invoked exactly once in the test script, after all tests are defined. You can run any preparation code before the invocation, or cleanup code after the invocation. The run_tests function does not return until all tests are done.

Context objects

The code block in the test definition is invoked with an argument, i.e. the context object, in the @_ array. The context object provides several utility methods for the test and is created specifically for the test. In other word, different context objects are created for different test definitions. In this document, the context object is sometimes referred to as $c.

The most significant method of the test context object is the done method. The $c->done method must be invoked when and only when all the subtests in the test has been done, successfully or not. Once the method has been invoked, no subtest can be performed in the test. If the method is not invoked in the test, an error would be reported.

The most basic usage would be invoking the $c->done method just before the end of the test code:

test {
    my $c = shift;
    
    ... tests ...
    
    $c->done;
};

If there are callbacks, the $c->done method should be invoked at the end of the last callback (the one invoked finally):

test {
    my $c = shift;
    my $timer = AE::timer 10, 0, sub {
        test {
            ...;
            $c->done;
            undef $c;
        } $c;
    };
};

The inner test{} statement is a test block; See "Test blocks" for details. Also note that undef $c is executed after $c->done is invoked. Though this is not required for this particular case, it might be a good practice to undef the context object reference when it is done'ed within a callback function as it would delete a possible loop reference when the context object has the reference to some object and the object then contains the reference to the callback function which contains the reference to the context object.

More complex example:

test {
    my $c = shift;
    
    my $cv = AE::cv;
    
    # This callback will be executed after all of
    # following callbacks are invoked.
    $cv->begin (sub { test { $c->done } $c });
    
    $cv->begin;
    my $timer1 = AE::timer 10, 0, sub { $cv->end };
    
    $cv->begin;
    my $timer2 = AE::timer 4, 0, sub { $cv->end };
    
    ...
    
    $cv->end;
};

Application test framework built on top of the module might define additional methods to context objects. See "Subclassing" on the guideline for extending the context object interface.

Test blocks

Another usage of the test{} function is defining a test block. They are different from the test definitions in that they are used within some test definition and takes the test context object $c as the first argument.

Test blocks are typically used within callback functions in a test; in fact there has to be a test block within an asynchronously invoked callback function:

test {
    my $c = shift;
    AE::io *STDIN, 0, sub {
        test {
            is scalar <STDIN>, "hoge";
            $c->done;
        } $c;
    };
};

A test block gives the test context as encapsulated by $c to the subtests within the block. You have to enclose subtests within callback functions by yourself, unfortunately, otherwise the test manager losts the association of subtests and their "parent" test, due to the asynchronousness of the invocation of the callback.

Test block can be named by the name option:

test {
    test {
        ok 1, 'Test X';
    } $c, name => 'hoge';
};

# ok 1 - [1] - [1] hoge Test X

The TEST_BLOCK_SKIP environment variable can be used to specify the regular expression to skip the test blocks with names matching to the pattern. For example, TEST_BLOCK_SKIP=og.$ would prevent the ok statement in the example above from executed. See also "Naming".

Naming

Tests, test blocks, and subtests can be named. Instead of a scalar value, representing the literal string, an array reference containing string components can be specified.

test {
    my $c = shift;
    test {
        is $hoge, $fuga, 'Subtest 1';
    } $c, name => 'Test block 1';
    $c->done;
} name => 'Test 1';

test {
    my $c = shift;
    test {
        is $hoge, $fuga, ['Subtest', 2];
    } $c, name => ['Test block', 2];
    $c->done;
} name => ['Test', 2];

Naming by array reference would be particularly useful when defining multiple tests by iteration:

for my $value (1, 2, 30, 120) {
    test {
        my $c = shift;
        like $c, qr{^\d+$};
        $c->done;
    } name => ['Test', $value];
}

If the name is represented as an array reference, its items are joined by . (period) before actually used to output results, or filter tests by environment variables, i.e. TEST_METHOD, TEST_METHOD_EXCLUDED, and TEST_BLOCK_SKIP. Any empty string is replaced by (empty) and any undef value is replaced by (undef).

Waiting for a condvar

An AnyEvent condvar can be specified as the wait parameter to a test definition (not test block!) to wait for the condvar to receive a value. The received value can be accessed from the $c->received_data method of the context object.

my $cv = AE::cv;
test {
    my $c = shift;
    is $c->received_data, 123;
    $c->done;
} wait => $cv;

Instead of a codevar object, a code reference which, when invoked, returns a condvar object or the undef value can be specified as the wait value. This practice is rather recommended as it would prevend the condvar object from being instantiated when the target tests are filtered by TEST_METHOD environment variable. For example,

test {
    my $c = shift;
    is $c->received_data, 123;
    $c->done;
} wait => sub { start_server_and_return_cv () };

... does not start the server for the test when the test is excluded from the execution.

The default wait value, used when no wait parameter is explicitly specified to test definitions, can be provided by subclassing (see "Subclassing" for details) and defining $tm->default_test_wait_cv method returning a condvar (or undef) in the test manager subclass. In this case, by explicitly setting undef value for the wait parameter of test definitions, this default can be cleared. (See t/cv-wait-default.t test script for examples.)

The wait value can also be a hash reference (or a code reference which returns a hash reference). The cv key of the hash reference can have the condvar object as the value. The destroy_as_cv key can contain the code reference, which will be invoked after relevant tests have been run. The code should be useful to stop the server started by the cv condvar's preparation, for example. The code must return a condvar object, whose callback will be invoked after the destroy process has been done. The same destroy_as_cv code is invoked only once. If the code is specified as part of the wait value of multiple tests, it is only invoked after all of them has been executed. Example:

test {
    ...
} wait => {cv => sub {
    return $server->start_as_cv;
}, destroy_as_cv => sub {
    return $server->stop_as_cv;
}};

Additionally, the timeout value can also be specified in the wait hash reference. Its default value is 60. After the seconds of the timeout elapse, if the wait condvar's callback is not invoked, the associated test fails. The timeout value is also applied to the context_begin and context_end methods of the $c->received_data object; These methods have to invoke the callback before the timeout. Please note that they should not block the entire script, otherwise the timeout will not work.

Concurrent execution of tests

Thanks to AnyEvent framework, tests (as defined by outermost test{} functions) can be concurrenrly executed when they are written in non-blocking way using AnyEvent.

Consider the following test script fragment:

test {
    my $c = shift;
    ok 'Subtest #1.1';
    AnyEvent::Example->something(cb => sub {
        test {
            is $_[0], 'hoge', 'Subtest #1.2';
            $c->done;
        } $c;
    });
};

test {
    my $c = shift;
    ok 'Subtest #2.1';
    AnyEvent::Example->something(cb => sub {
        test {
            is $_[0], 'hoge', 'Subtest #2.2';
            $c->done;
            undef $c;
        } $c;
    });
};

run_tests;

In this case, execution order of Subtests #1.2 and #2.2 is unclear at all, depending on how long AnyEvent::Example->something defers the execution of the callbacks. (Please also note that, although in the current implementation Subtest #2.1 is always executed after Subtest #1.1, as that test is defined by test{} after the other test, this is not guaranteed and you should not reply on this exact order. Future version of the module could introduce shuffling execution mode, for instance.)

Anyway, we can describe this situation that multiple tests are concurrently executed. By default, at most five tests are concurrently executed by Test::X1. Setting a number to the TEST_MAX_CONCUR environment variable can override this default, if desired. TEST_MAX_CONCUR=1 disables this concurrency, which will be useful for debugging purposes in particular.

More test option

The timeout option of the test definition (not a test block) specifies the timeout in seconds. The test must end within the seconds after the test is started (not inlcuding any wait, context_before, and context_after processing). The default value is 60. Note that the timeout might be applied as intended if the test blocks the script, by, e.g., blocking I/O access, system, or sleep.

Subclassing

Test scripts often need application-specific factory functions and/or utility functions to create expected precondition or to manage states of tested environment. For example, a test for Web application would need to start a Web server before any test and stop the server after tests. A test for database operation would want to insert a number of records into the database at some points in the test code. Although they can be implemented orthogonally from the Test::X1's framework, subclassing of the Test::X1 class should be a good candidate if you want to control the lifetime of such temporary objects by relating them with lexical scopes of tests.

If you'd like to extend Test::X1 for your application My, the subclass module, My/Test/X1.pm, would look like:

package My::Test::X1;
use Test::X1 ();
Test::X1::define_functions(__PACKAGE__);

package My::Test::X1::Manager;

sub my_create_database { ... }
sub my_drop_database { ... }

sub stop_test_manager { shift->my_drop_database }

package My::Test::X1::Context;

sub my_insert_data { ... }

1;

The Test::X1 module is used without importing any function, then the Test::X1::define_functions function is invoked with the package, i.e. My::Test::X1. Additional methods are defined in subclasses of test manager and context objects. Then, in your test script, instead of directly useing Test::X1 module, load your module:

use My::Test::X1;
my $tm = get_test_manager;
$tm->my_create_database;

test {
    my $c = shift;
    $c->my_insert_data;
    ...
};

run_tests;

It is considered as good practice to prepend a short prefix taken from the subclass name (my_ in this example) to the method names defined by subclasses such that future additions to base classes will not conflict with them.

Both test manager and context objects are blessed hash references. Subclasses can use hashes to save their data associated with objects. Such hash keys should be prefixed by subclass names as well.

EXPORTED FUNCTIONS

By useing the Test::X1 module, your test script imports following functions:

$tm = get_test_manager

Obtain the instance of the test manager for the test script. The test manager object is singleton; the function always returns the same object.

test { CODE } NAME => VALUE, ...; (Test definition)

Define a test. It must be invoked outside of any other test{} function call. It must be invoked before run_tests function call.

The code block is expected to run one or more subtests. The number of subtests is expected to be equal to the n parameter value, if specified. The code is expected to not throw any exception.

The code block, when invoked, receives the text context object $c for the test as the argument. The $c->done method is expected to be invoked after all subtests are run.

TEST MANAGER OBJECT

The test manager object is the object created for the test script, holding references to tests in the test script and monitoring their results. The test manager object is singleton; there is at most one test manager object at one time.

The Text::X1 class (and its subclasses) exports get_test_manager function, which takes no argument, returning the current test manager object.

If you are writing simple test scripts, you don't have to directly access test manager usually. The exported functions explained in the previous section are in fact invoking appropriate methods of the test manager object.

Following methods can be invoked or defined when you are subclassing the test manager object:

$cv = $tm->default_test_wait_cv

This method can be overridden by subclasses, if desired. This method is expected to return an AnyEvent condvar or undef. The value returned by this method is used as the wait parameter value of test definitions, when it is not explicitly specified.

$hashref = $tm->context_args

This method can be overridden by subclasses, if desired. This method is invoked when test context objects are created. It is expected to return a hash reference containing name/value pairs passed as arguments to the new method of the test context class. By default it returns an empty hash reference.

Name/value pairs specified here can be accessed from the test context object's blessed hash reference. See "Subclassing" for their usage.

$tm->stop_test_manager

This method can be overridden by subclasses, if desired. This method is invoked before the test manager object is destructed. It is expected to be used to close anything opened by the test manager, if necessary. This method can be invoked more than once for an test manager object. The Test::X1 module does it's best effort to invoke the method for the test manager object before Perl goes into the global destruction phase.

$tm->diag($color, $message)

$tm->note($color, $message)

Print a diagnostic message or a note, through Test::Builder's diag or note method. For their usage, see Test::Builder and Test::More documentations.

The first argument must be a color specification for Term::ANSIColor, e.g. "red" or "".

The second argument must be a diagnostic or note text, possibly utf8-flagged.

CONTEXT OBJECT

The context object is created for each test. It provides several information on the test, which can be used within test for debugging purpose. The context object is passed to the test as the first argument.

$name = $c->test_name

The compound name of the test, in a single character string. However the name is specified (or not), the method returns the single string as used in TAP test name part. See "Naming".

$data = $c->received_data

Return the data received from the AnyEvent condvar specified to the wait parameter of the test definition for the current test. See also "Waiting for a condvar".

The data can be any value, including the undef value. If the data is an object which has context_begin and context_end methods, they are invoked just after and just before the data is associated with a context object. (Please note that the data can be associated with multiple context objects when the condvar is specified for multiple tests.) Both methods will receive a code reference as an argument. The methods are expected to invoke the code once the object is ready for start or termination of the test. Typical use case of these methods is preparation and termination of a server process used within the test.

Example of class for such an object:

package My::Data;

sub context_begin {
  my ($self, $code) = @_;
  $self->{rc}++;
  $code->();
}

sub context_end {
  my ($self, $code) = @_;
  $self->{rc}--;
  $self->stop_server unless $self->{rc};
  $code->();
}

$c->diag($color, $message)

Print a diagnostic message, through Test::Builder's diag method. For their usage, see Test::Builder and Test::More documentations.

The first argument must be a color specification for Term::ANSIColor, e.g. "red" or "". (Please note that this argument is ignored in this version.)

The second argument must be a diagnostic text, possibly utf8-flagged.

$c->done

Notify that the substests in the test is done. This method must be invoked exactly once for a test. See also "Context objects".

LIMITATIONS

Subtests as implemented by Test::More / Test::Builder cannot be used in the context of this module as they are globally stateful such that concurrent execution of multiple different test introduced by this module is incompatibile with them. Please note that this module's concept of subtests is different from those subtests.

EXAMPLES

See t/*.t for more examples.

DEPENDENCY

This module requires Perl 5.10 or later. In addition to core modules, this module depends on Exporter::Lite and AnyEvent.

AUTHOR

Wakaba <wakaba@suikawiki.org>.

HISTORY

This module is inspired by following modules: Test::Builder, Test::Class, Test::More.

This repository was located at <https://github.com/wakaba/perl-test-x1> until 19 April 2023, then transferred to <https://github.com/pawjy/perl-test-x1>.

LICENSE

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

Name		Name	Last commit message	Last commit date
Latest commit History 687 Commits
.github		.github
config		config
doc		doc
lib/Test		lib/Test
t		t
.gitignore		.gitignore
Makefile		Makefile
README.pod		README.pod

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

NAME

SYNOPSIS

DESCRIPTION

USAGE

Tests

Context objects

Test blocks

Naming

Waiting for a condvar

Concurrent execution of tests

More test option

Subclassing

EXPORTED FUNCTIONS

TEST MANAGER OBJECT

CONTEXT OBJECT

LIMITATIONS

EXAMPLES

DEPENDENCY

AUTHOR

HISTORY

LICENSE

About

Releases

Packages

Contributors 2

Languages

pawjy/perl-test-x1

Folders and files

Latest commit

History

Repository files navigation

NAME

SYNOPSIS

DESCRIPTION

USAGE

Tests

Context objects

Test blocks

Naming

Waiting for a condvar

Concurrent execution of tests

More test option

Subclassing

EXPORTED FUNCTIONS

TEST MANAGER OBJECT

CONTEXT OBJECT

LIMITATIONS

EXAMPLES

DEPENDENCY

AUTHOR

HISTORY

LICENSE

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Languages

Packages