Add distinction between a test group and a test series

docs/api-adding-tests.md

@@ -1,18 +1,36 @@
-These `CanaryTest` methods are needed to write tests that can be run with Canary.
+These [`CanaryTest`](api-introduction.md) methods are needed to write tests that can be run with Canary.
 
 Tests and test groups can optionally be assigned names. It is strongly recommended that test names always be provided, since descriptive names will make it easier to understand where errors occur, when they occur.
 
+There are three different kinds of tests:
+
+- Individual tests, created using the [`test`](api-adding-tests.md#test) method.
+- Test groups, created using the [`group`](api-adding-tests.md#group) method.
+- Test series, created using the [`series`](api-adding-tests.md#series) method.
+
+An individual test can be synchronous or asynchronous. It represents a single test prodecude, and it is the most basic building block of a test suite.
+
+A test group is essentially a special case of an individual test. It is a test which may contain a list of child or subordinate tests. It may declare callbacks to run at the beginning and end of the test group's execution, and at the beginning and end of each child test's execution. In a test group, the order in which the child tests are run should not affect the outcome, and when any child test fails the rest of the tests will still be attempted.
+
+A test series is a special case of a test group. A test series is distinguished from a test group in that the order of execution of child tests is significant, and that if one child test fails the following tests should not be attempted.
+
+**Use a test group when:** It contains tests without side effects, or whose side effects are certain not to affect each other.
+
+**Use a test series when:** It contains tests that have side effects that may impact the other tests in the series, such as when testing a stateful API.
+
+Using these structures, a test tree, or hierarchy, is created. The individual tests are the leaves on this test tree and the test groups and series represent their own subtrees. Normally, the global `canary` instance will always be at the root of the test tree.
+
 # test
 
-Add an individual test. Tests are added to test groups. This results in a test tree, or hierarchy, where the test created in this way is a child of the test group it was added to.
+Add an individual test to a test group or series, such as the global `canary` instance.
 
 **Arguments:** `({string} name, {function} body)` _or_ `({function} body)`
 
-If the body function returns a `Promise`, then when the test is run Canary will wait to see if the promise is resolved or rejected before completing the test and progressing to the next one. (A resolved promise indicates that this part of the test ran without errors and a rejected promise is treated the same as an unhandled thrown error.)
+If the body function returns a [`Promise`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise), then when the test is run Canary will wait to see if the promise is resolved or rejected before completing the test and progressing to the next one. (A resolved promise indicates that this part of the test ran without errors and a rejected promise is treated the same as an unhandled thrown error.)
 
 When the body function is called, both `this` and the first argument will refer to the test to which the function belongs.
 
-**Returns:** The newly-created `CanaryTest` instance.
+**Returns:** The newly-created [`CanaryTest`](api-introduction.md) instance.
 
 **Examples:**
 
@@ -23,31 +41,33 @@ canary.test("Example test", function(){
 ```
 
 ``` js
-canary.group("Example test group", function(){
-    this.test("Example test", function(){
-        assert(1 < 3);
-    });
+canary.test("Example asynchronous test", async function(){
+    const result = await someAsyncFunction();
+    assert(result === 10);
 });
 ```
 
 ``` js
-canary.test("Example asynchronous test", async function(){
-    const result = await someAsyncFunction();
-    assert(result === 10);
+canary.group("Example test group", function(){
+    this.test("Example test", function(){
+        assert(1 < 3);
+    });
 });
 ```
 
 # group
 
-Add a test group. A test group is a special kind of test that may have child tests and callbacks such as `onBegin` and `onEnd`, but must not itself contain test logic.
+Add a test group to a parent test or series. A test group is a special kind of test that may have child tests and callbacks such as [`onBegin`](api-group-callbacks.md#onbegin) and [`onEnd`](api-group-callbacks.md#onend), but must not itself contain test logic.
+
+Unlike a test series, the children of a test group are not guaranteed to run in the order that they were added to the group. This means that the tests added to a group should be fully independent of one another. If they are not independent, then a series should be used instead.
 
 **Arguments:** `({string} name, {function} body)` _or_ `({function} body)`
 
-The body function should be synchronous. If it happens to return a `Promise` then, unlike a test instantiated with the `test` method, Canary will not wait for that promise to be resolved or rejected.
+The body function should be synchronous. If it happens to return a [`Promise`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise) then, unlike a test instantiated with the `test` method, Canary will not wait for that promise to be resolved or rejected.
 
 When the body function is called, both `this` and the first argument will refer to the test to which the function belongs.
 
-**Returns:** The newly-created `CanaryTest` instance.
+**Returns:** The newly-created [`CanaryTest`](api-introduction.md) instance.
 
 **Examples:**
 
@@ -66,3 +86,29 @@ canary.group("Example test group", function(){
 });
 ```
 
+# series
+
+Add a test series to a parent test or series. A test series is a special kind of test group that will always run its child tests in the order they were added, and that will abort at the first failure of any child test.
+
+Like a normal test group, a test series should have a synchronous body function without itself containing any test logic, and it may use callbacks like [`onBegin`](api-group-callbacks.md#onbegin) and [`onEnd`](api-group-callbacks.md#onend).
+
+**Arguments:** `({string} name, {function} body)` _or_ `({function} body)`
+
+The body function should be synchronous. If it happens to return a [`Promise`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise) then, unlike a test instantiated with the `test` method, Canary will not wait for that promise to be resolved or rejected.
+
+When the body function is called, both `this` and the first argument will refer to the test to which the function belongs.
+
+**Returns:** The newly-created [`CanaryTest`](api-introduction.md) instance.
+
+**Examples:**
+
+``` js
+canary.series("Example test series", function(){
+    this.test("First example test", function(){
+        assert("hello" === "hello");
+    });
+    this.test("Second example test", function(){
+        assert("world" === "world");
+    });
+});
+```

docs/api-advanced-usage.md

@@ -1,10 +1,26 @@
-These `CanaryTest` class methods are for those who need more complex or customized behavior from Canary. They are not relevant to the majority of users.
+These [`CanaryTest`](api-introduction.md) class methods are for those who need more complex or customized behavior from Canary. They are not relevant to the majority of users.
+
+# constructor
+
+The [`CanaryTest`](api-introduction.md) class constructor can be used to instantiate a test object with a given name and body function. Usually, test objects should be instantiated indirectly using the [`test`](api-adding-tests.md#test), [`group`](api-adding-tests.md#group), and [`series`](api-adding-tests.md#series) methods.
+
+**Arguments:** `({string} name, {function} body)`
+
+**Returns:** The new [`CanaryTest`](api-introduction.md) instance.
+
+**Examples:**
+
+``` js
+const test = new canary.Test("Example test", function(){
+    assert(true);
+});
+```
 
 # silent
 
 Set a test and all its children to run silently. This means that they will not output any log messages.
 
-The `log` method of a `CanaryTest` instance can be used to log a message only if the test has not been set to run silently.
+The `log` method of a [`CanaryTest`](api-introduction.md) instance can be used to log a message only if the test has not been set to run silently.
 
 Note that when the `concise` flag of the `doReport` function is set, it will cause all tests to run silently.
 
@@ -20,7 +36,7 @@ canary.test("Example silent test", function(){
 
 Set a test and all its children to run verbosely. This means they will output even more detailed logs than usual.
 
-The `logVerbose` method of a `CanaryTest` instance can be used to log a message only when the test is set to run verbosely.
+The `logVerbose` method of a [`CanaryTest`](api-introduction.md) instance can be used to log a message only when the test is set to run verbosely.
 
 Note that when the `verbose` flag of the `doReport` function is set, it will cause all tests to run verbosely that were not otherwise set to run silently.
 
@@ -79,9 +95,9 @@ Get a list of tags that have been added to this test.
 
 # getTestTotal
 
-Recursively get the number of `CanaryTest` instances in a test tree.
+Recursively get the number of [`CanaryTest`](api-introduction.md) instances in a test tree.
 
-Note that if this method is called for a test group, and the group has not already been expanded, then this method will cause it to be expanded. (As though the `expandGroups` method was called.)
+Note that if this method is called for a test group or series, and the group has not already been expanded, then this method will cause it to be expanded. (As though the `expandGroups` method was called.)
 
 **Returns:** The total number of tests in this test tree.
 
@@ -107,17 +123,29 @@ Get the length of time taken to run this test, in milliseconds.
 
 Add an error to the test's list of recorded errors.
 
-The method requires an Error object and optionally accepts another argument indicating the test or callback where the error occurred.
+The method requires an [`Error`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) object and optionally accepts another argument indicating the test or callback where the error occurred.
 
 **Arguments:** `({Error} error, {CanaryTest|CanaryTestCalback} location)`
 
-**Returns:** The newly-created `CanaryTestError` instance.
+**Returns:** The newly-created [`CanaryTestError`](api-error-class.md) instance.
 
 # abort
 
 Abort the test and mark it as failed.
 
-The method optionally accepts information about the error that resulted in the test being aborted. The first argument, if provided, must be an Error object, and the second argument can be used to indicate the test or callback where the error occured. (This is the same as with the `error` method.)
+[`onFailure`](api-group-callbacks.md#onfailure), [`onEachFailure`](api-group-callbacks.md#oneachfailure), [`onEnd`](api-group-callbacks.md#onend), and [`onEachEnd`](api-group-callbacks.md#oneachend) callbacks will be executed upon calling this method.
+
+The method optionally accepts information about the error that resulted in the test being aborted. The first argument, if provided, must be an [`Error`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) object, and the second argument can be used to indicate the test or callback where the error occured. (This is the same as with the [`addError`](api-advanced-usage.md#adderror) method.)
+
+**Arguments:** `({Error} error, {CanaryTest|CanaryTestCalback} location)`
+
+# fail
+
+Abort the test as failed, though not as aborted.
+
+This method should only be called as the test is completed, since it immediately executes any [`onFailure`](api-group-callbacks.md#onfailure), [`onEachFailure`](api-group-callbacks.md#oneachfailure), [`onEnd`](api-group-callbacks.md#onend), and [`onEachEnd`](api-group-callbacks.md#oneachend) callbacks.
+
+The method optionally accepts information about the error that caused the test to fail. The first argument, if provided, must be an [`Error`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) object, and the second argument can be used to indicate the test or callback where the error occured. (This is the same as with the [`addError`](api-advanced-usage.md#adderror) method.)
 
 **Arguments:** `({Error} error, {CanaryTest|CanaryTestCalback} location)`
 
@@ -137,7 +165,31 @@ Get whether this test was run without encountering any errors.
 
 Get a list of errors that were encountered while attempting this test, if any.
 
-**Returns:** An array of `CanaryTestError` objects encountered while running this test.
+**Returns:** An array of [`CanaryTestError`](api-error-class.md) objects encountered while running this test.
+
+# anyFailedChildren
+
+Get whether any child tests of a test group or series failed.
+
+**Returns:** `true` when any child tests have failed and `false` otherwise.
+
+If this method is called for an individual test rather than for a test group, then the return value will always be `false`.
+
+# noFailedChildren
+
+Get whether none of the child tests of a test group or series have been marked as failed.
+
+**Returns:** `true` when no child tests have failed and `false` when any have failed.
+
+If this method is called for an individual test rather than for a test group, then the return value will always be `true`.
+
+# getFailedChildren
+
+Get a list of child tests that have failed, if any.
+
+**Returns:** An array of [`CanaryTest`](api-introduction.md) objects representing the failed child tests.
+
+If this method is called for an individual test rather than for a test group, then the returned array will always be empty.
 
 # addTest
 
@@ -161,6 +213,19 @@ Remove all child tests from a parent group.
 
 **Arguments:** `({CanaryTest} test)`
 
+**Examples:**
+
+``` js
+const someGroup = canary.group("Example test group", function(){
+    this.test("Example test", function(){
+        assert(true);
+    });
+});
+assert(someGroup.getChildren().length === 1);
+someGroup.removeAllTests();
+assert(someGroup.getChildren().length === 0);
+```
+
 # orphan
 
 Remove a test from its parent group.
@@ -169,15 +234,17 @@ Remove a test from its parent group.
 
 # getParent
 
-Get the `CanaryTest` instance which is the parent of this one.
+Get the [`CanaryTest`](api-introduction.md) instance which is the parent of this one.
 
-**Returns:** The `CanaryTest` instance to which this test has been added, or `undefined` if the test does not have a parent.
+**Returns:** The [`CanaryTest`](api-introduction.md) instance to which this test has been added, or `undefined` if the test does not have a parent.
 
 # getChildren
 
 Get a list of the tests that are children of this test group.
 
-**Returns:** An array of `CanaryTest` instances which are children of this test group.
+Note that if this method is called for a test group or series, and the group has not already been expanded, then this method will cause it to be expanded. (As though the `expandGroups` method was called.)
+
+**Returns:** An array of [`CanaryTest`](api-introduction.md) instances which are children of this test group.
 
 # applyFilter
 
@@ -195,6 +262,23 @@ Recursively run all the body functions assigned to test groups, but not to ordin
 
 Group expansion is put off until tests are actually needed in order to make the startup performance impact of including tests in an application source file close to nonexistent, even in the case of testing code errors that could potentially cause hangups, since extremely little work is done at the time of declaration.
 
+**Examples:**
+
+``` js
+const someGroup = canary.group("Example test group", function(){
+    canary.test("Example test", function(){
+        assert("hello" === "hello");
+    });
+});
+// Test groups are not expanded upon declaration, meaning that Canary
+// doesn't know about someGroup's children yet.
+assert(someGroup.getChildren().length === 0);
+// After a call to expandGroups, Canary will have knowledge of the
+// structure of the test tree.
+canary.expandGroups();
+assert(someGroup.getChildren().length === 1);
+```
+
 # reset
 
 Resets the test's state so that it is safe to run it again. This method resets tests recursively, so all tests that belong to a group will also be reset.

docs/api-callback-class.md

@@ -1,10 +1,10 @@
-The `CanaryTestCallback` class is used to represent callbacks added to a test using methods such as `onBegin` and `onEnd`. It has a constructor and no methods. It can be accessed via `canary.Callback`.
+The [`CanaryTestCallback`](api-callback-class.md) class is used to represent callbacks added to a test using methods such as [`onBegin`](api-group-callbacks.md#onbegin) and [`onEnd`](api-group-callbacks.md#onend). It has a constructor and no methods. It can be accessed via `canary.Callback`.
 
 # getOwner
 
 Get the test object to which this callback belongs.
 
-**Returns:** The `CanaryTest` instance to which this callback was added.
+**Returns:** The [`CanaryTest`](api-introduction.md) instance to which this callback was added.
 
 # getName
 
@@ -17,4 +17,3 @@ Get a string naming this callback.
 Get a string containing an identifying title for this callback.
 
 **Returns:** A string representing the title of this callback.
-

docs/api-error-class.md

@@ -1,14 +1,14 @@
-The `CanaryTestError` class is used to record errors encountered while attempting tests using Canary. It can be accessed via `canary.Error`.
+The [`CanaryTestError`](api-error-class.md) class is used to record errors encountered while attempting tests using Canary. It can be accessed via `canary.Error`.
 
 # stack
 
-A property to get the stack trace attribute of the error object that this `CanaryTestError` instance was instantiated with, provided such an attribute exists.
+A property to get the stack trace attribute of the error object that this [`CanaryTestError`](api-error-class.md) instance was instantiated with, provided such an attribute exists.
 
 **Value:** A stack trace, or `undefined` if none was found.
 
 # message
 
-A property to get the message attribute of the error object that this `CanaryTestError` instance was instantiated with, provided such an attribute exists.
+A property to get the message attribute of the error object that this [`CanaryTestError`](api-error-class.md) instance was instantiated with, provided such an attribute exists.
 
 **Value:** An error message, or `undefined` if none was found.