Web Inspector: add TestSuite hooks for per-testcase setup and teardow…

…n actions

https://bugs.webkit.org/show_bug.cgi?id=148641

Reviewed by Timothy Hatcher.

Source/WebInspectorUI:

It should be possible to run setup and teardown actions for each test case.
The use case here is to add listeners, set up testcase-specific state, etc.
To match how test functions work in each suite, the sync and async implementations
take functions with no arguments and(resolve, reject), respectively.

* UserInterface/Test/TestSuite.js:
(TestSuite.prototype.addTestCase):
(TestSuite.messageFromThrownObject):
(TestSuite):
(AsyncTestSuite.prototype.runTestCases): Stick the extra functions in the chain.
(AsyncTestSuite):
(SyncTestSuite.prototype.runTestCases):
(SyncTestSuite):

LayoutTests:

Extend existing unit tests to also exercise setup and teardown actions.

* inspector/unit-tests/async-test-suite.html:
* inspector/unit-tests/async-test-suite-expected.txt:
* inspector/unit-tests/sync-test-suite.html:
* inspector/unit-tests/sync-test-suite-expected.txt:


Canonical link: https://commits.webkit.org/166791@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@189213 268f45cc-cd09-0410-ab3c-d52691b4dbfc

LayoutTests/ChangeLog

@@ -1,3 +1,17 @@
+2015-09-01  Brian Burg  <bburg@apple.com>
+
+        Web Inspector: add TestSuite hooks for per-testcase setup and teardown actions
+        https://bugs.webkit.org/show_bug.cgi?id=148641
+
+        Reviewed by Timothy Hatcher.
+
+        Extend existing unit tests to also exercise setup and teardown actions.
+
+        * inspector/unit-tests/async-test-suite.html:
+        * inspector/unit-tests/async-test-suite-expected.txt:
+        * inspector/unit-tests/sync-test-suite.html:
+        * inspector/unit-tests/sync-test-suite-expected.txt:
+
 2015-09-01  Zalan Bujtas  <zalan@apple.com>
 
         Repaint cleanup:

LayoutTests/inspector/unit-tests/async-test-suite-expected.txt

@@ -7,6 +7,12 @@ PASS: should not be able to add non-object test case.
 PASS: test case should require string name.
 PASS: test case should require non-whitespace name.
 PASS: test case should require test function.
+PASS: should not be able to specify non-Function `setup` parameter.
+PASS: should not be able to specify non-Function `setup` parameter.
+PASS: should not be able to specify non-Function `setup` parameter.
+PASS: should not be able to specify non-Function `teardown` parameter.
+PASS: should not be able to specify non-Function `teardown` parameter.
+PASS: should not be able to specify non-Function `teardown` parameter.
 PASS: should not be able to run empty test suite.
 
 == Running test suite: AsyncTestSuite.RunTwiceSuite
@@ -38,3 +44,36 @@ PASS: abortOnFailureSuite should have passed one test.
 PASS: abortOnFailureSuite should have failed one test.
 PASS: abortOnFailureSuite should have skipped one test.
 
+== Running test suite: AsyncTestSuite.SetupAndTeardown
+-- Running test setup.
+-- Running test case: TestWithSetupAndTeardown
+PASS: Test should see side effects of running setup() action.
+-- Running test teardown.
+PASS: Teardown should see side effects of running setup() action.
+
+-- Running test case: TestRunningAfterTeardown
+PASS: Test should see side effects of previous test's teardown() action.
+PASS: Promise from setupAndTeardownTestSuite.runTestCases() should resolve.
+
+== Running test suite: AsyncTestSuite.SetupException
+-- Running test setup.
+!! EXCEPTION: 
+PASS: Promise from setupExceptionTestSuite.runTestCases() should reject.
+
+== Running test suite: AsyncTestSuite.SetupFailure
+-- Running test setup.
+!! EXCEPTION: undefined
+PASS: Promise from setupFailureTestSuite.runTestCases() should reject.
+
+== Running test suite: SyncTestSuite.TeardownException
+-- Running test case: TestWithExceptionDuringTeardown
+-- Running test teardown.
+!! EXCEPTION: 
+PASS: Promise from teardownExceptionTestSuite.runTestCases() should reject.
+
+== Running test suite: SyncTestSuite.TeardownFailure
+-- Running test case: TestWithExceptionDuringTeardown
+-- Running test teardown.
+!! EXCEPTION: undefined
+PASS: Promise from teardownFailureTestSuite.runTestCases() should reject.
+

LayoutTests/inspector/unit-tests/async-test-suite.html

@@ -73,6 +73,66 @@
     } catch (e) {
         ProtocolTest.log("PASS: test case should require test function.");
     }
+    try {
+        badArgsSuite.addTestCase({
+            name: "foo",
+            test: () => {},
+            setup: "astd"
+        });
+        ProtocolTest.log("FAIL: should not be able to specify non-Function `setup` parameter.");
+    } catch (e) {
+        ProtocolTest.log("PASS: should not be able to specify non-Function `setup` parameter.");
+    }
+    try {
+        badArgsSuite.addTestCase({
+            name: "foo",
+            test: () => {},
+            setup: 123
+        });
+        ProtocolTest.log("FAIL: should not be able to specify non-Function `setup` parameter.");
+    } catch (e) {
+        ProtocolTest.log("PASS: should not be able to specify non-Function `setup` parameter.");
+    }
+    try {
+        badArgsSuite.addTestCase({
+            name: "foo",
+            test: () => {},
+            setup: {}
+        });
+        ProtocolTest.log("FAIL: should not be able to specify non-Function `setup` parameter.");
+    } catch (e) {
+        ProtocolTest.log("PASS: should not be able to specify non-Function `setup` parameter.");
+    }
+    try {
+        badArgsSuite.addTestCase({
+            name: "foo",
+            test: () => {},
+            teardown: "astd"
+        });
+        ProtocolTest.log("FAIL: should not be able to specify non-Function `teardown` parameter.");
+    } catch (e) {
+        ProtocolTest.log("PASS: should not be able to specify non-Function `teardown` parameter.");
+    }
+    try {
+        badArgsSuite.addTestCase({
+            name: "foo",
+            test: () => {},
+            teardown: 123
+        });
+        ProtocolTest.log("FAIL: should not be able to specify non-Function `teardown` parameter.");
+    } catch (e) {
+        ProtocolTest.log("PASS: should not be able to specify non-Function `teardown` parameter.");
+    }
+    try {
+        badArgsSuite.addTestCase({
+            name: "foo",
+            test: () => {},
+            teardown: {}
+        });
+        ProtocolTest.log("FAIL: should not be able to specify non-Function `teardown` parameter.");
+    } catch (e) {
+        ProtocolTest.log("PASS: should not be able to specify non-Function `teardown` parameter.");
+    }
 
     let runEmptySuite = ProtocolTest.createAsyncSuite("AsyncTestSuite.RunEmptySuite");
     try {
@@ -176,6 +236,157 @@
         return Promise.resolve(); // Continue this test.
     });
 
+    var setupAndTeardownSymbol = Symbol("async-suite-setup-and-teardown-token");
+    window[setupAndTeardownSymbol] = 0;
+
+    let setupAndTeardownTestSuite = ProtocolTest.createAsyncSuite("AsyncTestSuite.SetupAndTeardown");
+    setupAndTeardownTestSuite.addTestCase({
+        name: "TestWithSetupAndTeardown",
+        description: "Check execution order for setup and teardown actions.",
+        setup: (resolve, reject) => {
+            window[setupAndTeardownSymbol] = 1;
+            resolve();
+        },
+        test: (resolve, reject) => {
+            ProtocolTest.expectThat(window[setupAndTeardownSymbol] === 1, "Test should see side effects of running setup() action.");
+            window[setupAndTeardownSymbol] = 2;
+            resolve();
+        },
+        teardown: (resolve, reject) => {
+            ProtocolTest.expectThat(window[setupAndTeardownSymbol] === 2, "Teardown should see side effects of running setup() action.");
+            window[setupAndTeardownSymbol] = 3;
+            resolve();
+        }
+    });
+    setupAndTeardownTestSuite.addTestCase({
+        name: "TestRunningAfterTeardown",
+        description: "Check execution order for test after a teardown action.",
+        test: (resolve, reject) => {
+            ProtocolTest.expectThat(window[setupAndTeardownSymbol] === 3, "Test should see side effects of previous test's teardown() action.");
+            resolve();
+        },
+    });
+
+    result = result.then(() => {
+        return setupAndTeardownTestSuite.runTestCases();
+    }).then(function resolved() {
+        ProtocolTest.log("PASS: Promise from setupAndTeardownTestSuite.runTestCases() should resolve.");
+        return Promise.resolve(); // Continue this test.
+    }, function rejected(e) {
+        ProtocolTest.log("FAIL: Promise from setupAndTeardownTestSuite.runTestCases() should resolve.");
+        return Promise.resolve(); // Continue this test.
+    });
+
+    let setupExceptionTestSuite = ProtocolTest.createAsyncSuite("AsyncTestSuite.SetupException");
+    setupExceptionTestSuite.addTestCase({
+        name: "TestWithExceptionDuringSetup",
+        description: "Check execution order for setup action that throws an exception.",
+        setup: (resolve, reject) => { throw new Error() },
+        test: (resolve, reject) => {
+            ProtocolTest.assert(false, "Test should not execute if its setup action threw an exception.");
+            reject();
+        },
+        teardown: (resolve, reject) => {
+            ProtocolTest.assert(false, "Teardown action should not execute if its setup action threw an exception.");
+            reject();           
+        }
+    });
+
+    result = result.then(() => {
+        return setupExceptionTestSuite.runTestCases();
+    }).then(function resolved() {
+        ProtocolTest.log("FAIL: Promise from setupExceptionTestSuite.runTestCases() should reject.");
+        return Promise.resolve(); // Continue this test.
+    }, function rejected(e) {
+        ProtocolTest.log("PASS: Promise from setupExceptionTestSuite.runTestCases() should reject.");
+        return Promise.resolve(); // Continue this test.
+    });
+
+    let setupFailureTestSuite = ProtocolTest.createAsyncSuite("AsyncTestSuite.SetupFailure");
+    setupFailureTestSuite.addTestCase({
+        name: "TestWithFailureDuringSetup",
+        description: "Check execution order for setup action that has failed.",
+        setup: (resolve, reject) => { reject(); },
+        test: (resolve, reject) => {
+            ProtocolTest.assert(false, "Test should not execute if its setup action failed.")
+            reject();
+        },
+        teardown: (resolve, reject) => {
+            ProtocolTest.assert(false, "Teardown action should not execute if its setup action failed.")
+            reject();
+        }
+    });
+
+    result = result.then(() => {
+        return setupFailureTestSuite.runTestCases();
+    }).then(function resolved() {
+        ProtocolTest.log("FAIL: Promise from setupFailureTestSuite.runTestCases() should reject.");
+        return Promise.resolve(); // Continue this test.
+    }, function rejected(e) {
+        ProtocolTest.log("PASS: Promise from setupFailureTestSuite.runTestCases() should reject.");
+        return Promise.resolve(); // Continue this test.
+    });
+
+    let teardownExceptionTestSuite = ProtocolTest.createAsyncSuite("SyncTestSuite.TeardownException");
+    teardownExceptionTestSuite.addTestCase({
+        name: "TestWithExceptionDuringTeardown",
+        description: "Check execution order for teardown action that throws an exception.",
+        test: (resolve, reject) => { resolve(); },
+        teardown: (resolve, reject) => { throw new Error() }
+    });
+    teardownExceptionTestSuite.addTestCase({
+        name: "TestAfterTeardownException",
+        descrption: "Check execution order for test after previous test's teardown throws an exception.",
+        setup: (resolve, reject) => {
+            ProtocolTest.assert(false, "Setup action should not execute if previous test's teardown action threw an exception.");
+            reject();
+        },
+        test: (resolve, reject) => {
+            ProtocolTest.assert(false, "Test should not execute if previous test's teardown action threw an exception.");
+            reject();
+        }
+    });
+
+    result = result.then(() => {
+        return teardownExceptionTestSuite.runTestCases();
+    }).then(function resolved() {
+        ProtocolTest.log("FAIL: Promise from teardownExceptionTestSuite.runTestCases() should reject.");
+        return Promise.resolve(); // Continue this test.
+    }, function rejected(e) {
+        ProtocolTest.log("PASS: Promise from teardownExceptionTestSuite.runTestCases() should reject.");
+        return Promise.resolve(); // Continue this test.
+    });
+
+    let teardownFailureTestSuite = ProtocolTest.createAsyncSuite("SyncTestSuite.TeardownFailure");
+    teardownFailureTestSuite.addTestCase({
+        name: "TestWithExceptionDuringTeardown",
+        description: "Check execution order for teardown action that has failed.",
+        test: (resolve, reject) => { resolve(); },
+        teardown: (resolve, reject) => { reject(); },
+    });
+    teardownFailureTestSuite.addTestCase({
+        name: "TestAfterTeardownException",
+        description: "Check execution order for test after previous test's teardown throws an exception",
+        setup: (resolve, reject) => {
+            ProtocolTest.assert(false, "Setup action should not execute if previous test's teardown action failed.");
+            reject();
+        },
+        test: (resolve, reject) => {
+            ProtocolTest.assert(false, "Test should not execute if previous test's teardown action failed.");
+            reject();
+        }
+    });
+
+    result = result.then(() => {
+        return teardownFailureTestSuite.runTestCases();
+    }).then(function resolved() {
+        ProtocolTest.log("FAIL: Promise from teardownFailureTestSuite.runTestCases() should reject.");
+        return Promise.resolve(); // Continue this test.
+    }, function rejected(e) {
+        ProtocolTest.log("PASS: Promise from teardownFailureTestSuite.runTestCases() should reject.");
+        return Promise.resolve(); // Continue this test.
+    });
+
     // This will finish the test whether the chain was resolved or rejected.
     result = result.then(() => { ProtocolTest.completeTest(); });
 }

LayoutTests/inspector/unit-tests/sync-test-suite-expected.txt

@@ -7,6 +7,12 @@ PASS: should not be able to add non-object test case.
 PASS: test case should require string name.
 PASS: test case should require non-whitespace name.
 PASS: test case should require test function.
+PASS: should not be able to specify non-Function `setup` parameter.
+PASS: should not be able to specify non-Function `setup` parameter.
+PASS: should not be able to specify non-Function `setup` parameter.
+PASS: should not be able to specify non-Function `teardown` parameter.
+PASS: should not be able to specify non-Function `teardown` parameter.
+PASS: should not be able to specify non-Function `teardown` parameter.
 PASS: should not be able to run empty test suite.
 
 == Running test suite: SyncTestSuite.RunTwiceSuite
@@ -35,3 +41,31 @@ PASS: abortOnFailureSuite should have passed one test.
 PASS: abortOnFailureSuite should have failed one test.
 PASS: abortOnFailureSuite should have skipped one test.
 
+== Running test suite: SyncTestSuite.SetupAndTeardown
+-- Running test setup.
+-- Running test case: TestWithSetupAndTeardown
+PASS: Test should see side effects of running setup() action.
+-- Running test teardown.
+PASS: Teardown should see side effects of running setup() action.
+
+-- Running test case: TestRunningAfterTeardown
+PASS: Test should see side effects of previous test's teardown() action.
+
+== Running test suite: SyncTestSuite.SetupException
+-- Running test setup.
+!! EXCEPTION: 
+
+== Running test suite: SyncTestSuite.SetupFailure
+-- Running test setup.
+!! EXCEPTION
+
+== Running test suite: SyncTestSuite.TeardownException
+-- Running test case: TestWithExceptionDuringTeardown
+-- Running test teardown.
+!! EXCEPTION: 
+
+== Running test suite: SyncTestSuite.TeardownFailure
+-- Running test case: TestWithExceptionDuringTeardown
+-- Running test teardown.
+!! EXCEPTION:
+