Pooling tests

Pooling tests

It is common when designing a test to end up with a very large test. The best way to resolve this is to break the test up into smaller sub-tests to be presented to the end user. The toolbox handles this in two ways: pool of tests or pool of pages.

Pool of pages

This is the simplest way to resolve the issue. This is client side and requires the least extra work from the test creator. In this mode, the client receives all of the pages and then selects the appropriate pages based on random numbers.

Consider the following page excerpt:

<waet>
    <setup>
    ...
    </setup>
    <page id="test-1">...</page>
    <page id="test-2">...</page>
    <page id="test-3">...</page>
    <page id="test-4">...</page>
    <page id="test-5">...</page>
    <page id="test-6">...</page>
</waet>

In this case there are 6 test pages to complete, which may take far too long for a single user to complete. In this case we can use the poolSize attribute of the setup node:

<setup poolSize="2">...</setup>

By setting it to a size of 2, the system will select 2 of the six given pages at random and present them to the user. For example, the first three invocations could pick pages [1, 3, 4], [1, 4, 6] and [1, 2, 3]. The danger of using it this way is the random nature, as you can see from our example the 5th page was never tested but the first page may have been picked 3 times! If there is a large number of test participants this may be solved simply by the law of averages. But if your test is targetting a small set of users then be aware of this problem.

Server-Side Pool of pages

This uses the same approach as the above, but the Python / PHP server will manage the page randomisation to ensure an as-close to uniform test environment as possible.

To use this feature, do the following steps:

1. Save your XML as '/tests/pool.xml'.
2. Make sure to set the poolSize attribute in the <setup> node.
3. Enter the correct URL: /test.html?url=php/pool.php

Pool of tests

This requires more work to achieve but can guarantee test uniformity and only works on PHP. This follows a similar concept to the pool of pages approach, except the test creator must create several test XML files. For instance, following on from the six test example above, we may break these up into two page tests:

Test 1:

<waet>
    <setup>
    ...
    </setup>
    <page id="test-1">...</page>
    <page id="test-2">...</page>
</waet>

Test 2:

<waet>
    <setup>
    ...
    </setup>
    <page id="test-3">...</page>
    <page id="test-4">...</page>
</waet>

Test 3:

<waet>
    <setup>
    ...
    </setup>
    <page id="test-5">...</page>
    <page id="test-6">...</page>
</waet>

Each of these XMLs must then be placed within the folder /tests/'. All XMLs within this directory are added to the pool, it doesn't scan sub-directories. For instance our directory would now hold [test1.xml, test2.xml, test3.xml]`.

Set the URL argument of the test link to the pseudo service: test.html?url=php/pesudo.php