Add custom bootstrap code for examples

[malkusch]

Consider adding a custom bootstrap code (similar to phpunit's --bootstrap option) which would allow defining needed resources in the scope of the test (e.g. files, variables, class imports).

To emphasize on a single aspect, the documentation might show just an excerpt of runnable code (e.g. variable definitions are missing). The bootstrap script would then define those missing variables and imports. Here's a real world documentation example:

$throttle->setRate(100, BandwidthThrottle::KIBIBYTES);

The bootstrap script would need to provide $throttle and use bandwidthThrottle\BandwidthThrottle.
That means the bootstrap code needs to be evald together with the example code as one string.

I guess a next problem would arise how to refer to specific examples in the bootstrap code, as one might setup different scenarios for different examples. I suggest not including the bootstrap in the documentation itself, because the purpose of documentation is being readable for the user in the first place.

[cundd]

I added a --bootstrap option. But the file will be included once before the tests are run.

I understand this will not solve your problem. So I'll keep this issue open try to find a good idea :)

[malkusch]

Can't you just change the semantic of --bootstrap and run that file for each test case within the scope of the test case? I guess this can easily be done by simply putting a require right before the eval() of the example. This would already help for a lot of cases.

[cundd]

My idea is to create kind of a event-listener system, where a function can be defined that will be called before the code is ran. Than one could register a listener in the bootstrap file.

The listener will then be invoked with the Test Definition as argument. This allows to check which test is run at the moment and respond to individual tests.

In addition this prevents errors like "class already exists" if one is defined in the bootstrap file.

What do you think?

[malkusch]

The listener will then be invoked with the Test Definition as argument.

How exactly does that argument look like?

[cundd]

The argument will be a \Cundd\TestFlight\Definition\DefinitionInterface

[malkusch]

I fear that it might become not so user friendly also I don't see how variables could be defined within the execution context of the test. Let me present another approach, just from the user perspective:

Let's first define an annotation schema which can locate any tests (also more than one) in a project:
E.g. @testflight foo\Bar::foo() would reference all tests in the docblock of foo\Bar::foo().

Now the user can implement those event listeners which would more look like this:

/**
 * @testflight foo\Bar::foo() 
 **/
class FooTestSetUp implements TestFlightSetUp
{
    public function setUp() {
        $bar = new Bar();
        return new TestContext(["bar" => $bar]);
    }
}

Testflight would now use that context for the referenced test and put the variables ($bar) into the test's execution context.

And there could be many interfaces for different tasks: E.g.

• TestFlightTearDown to perform a clean up task

How do you like that idea. This sounds probably like a lot of work, but I'm willing to help you on that.

[cundd]

The new annotation (@testflight) would be added to the test setup class, which than would be called before the corresponding test (in your example the test "around" foo\Bar::foo()). Correct?

I yesterday started to implement a simple event dispatcher. You can see an example at https://github.com/cundd/test-flight/tree/feature/event-dispatcher/src/Event.

I like your idea of a context object, it e.g. could also be added as second argument to the event listener callback. Then in the event listener the TestContext can be modified and will finally be extracted in the code runner.

[cundd]

Of course your idea isn't bad. It just seems very "heavy" at the moment.

[cundd]

At it's core it may would look something like this:

$dispatcher = new \Cundd\TestFlight\Event\EventDispatcher();

// Collect all methods with @testflight
$allMethods = [...];

foreach ($allMethods as $class => $method) {
    $dispatcher->register('test.will_invoke', [$class, $method]);
}

So the event-workflow and the annotation workflow could work hand-in-hand?

[malkusch]

The new annotation (@TestFlight) would be added to the test setup class, which than would be called before the corresponding test (in your example the test "around" foo\Bar::foo()). Correct?

Correct.

// Collect all methods with @TestFlight
$allMethods = [...];

Actually it's meant "collect all methods which match the argument of @TestFlight". I see this part as a bit hard depending on how powerful that reference schema will be designed.

But if you think that's too heavy for now, then keep it simple. I'm happy when I get any opportunity to define some variable for my tests ;)

[cundd]

$dispatcher = new \Cundd\TestFlight\Event\EventDispatcher();

// Collect all methods which match the argument of @testflight
$allMethods = [...];

foreach ($allMethods as $class => $method) {
    $dispatcher->register('test.will_invoke', [$class, $method]);
}

Let's start with the event-based system and keep the annotations in mind. 😄

Another question is how the Context should be handled in method-based tests? Maybe just pass the Context as argument. For Code-string-based tests one could use extract(). Do you have a good idea?

[malkusch]

how the Context should be handled in method-based tests?

You could continue adopting PHPUnit's vocabular by adding methods annotated with @before. Actually those method-based tests don't need that. They can setup whatever context they want. They are not part of the documentation, are they? Maybe I'm not the right person to ask that, as I don't know why one would put test methods into the implementation in the first place.

For Code-string-based tests one could use extract().

Yes extract() right before the eval(). I guess here.

[cundd]

As a side note: Test-Flight can run different types of tests. Those based on strings (or blocks) of code and those that are regular PHP methods

[cundd]

My last comment became superfluous.

[malkusch]

Yeah, it took me a quick read of your documentation to understand what you were talking about. So I edited my answer ;)

[cundd]

Yes you're right. Method-based tests are not part of the documentation and have a context of their own anyway.

I'll look into @before. I haven't heard about that yet

[cundd]

I prepared the bootstrap and context features. I tried to describe the feature in https://github.com/cundd/test-flight/tree/master/src/Context

[malkusch]

Thank you. This will do the job.

[cundd]

I will have to make the backport to PHP 5.

[cundd]

I backported the changes in https://github.com/cundd/test-flight/tree/php56