Skip to content
Permalink
Browse files
Web Inspector: add TestSuite hooks for per-testcase setup and teardow…
…n actions

https://bugs.webkit.org/show_bug.cgi?id=148641

Reviewed by Timothy Hatcher.

Source/WebInspectorUI:

It should be possible to run setup and teardown actions for each test case.
The use case here is to add listeners, set up testcase-specific state, etc.
To match how test functions work in each suite, the sync and async implementations
take functions with no arguments and(resolve, reject), respectively.

* UserInterface/Test/TestSuite.js:
(TestSuite.prototype.addTestCase):
(TestSuite.messageFromThrownObject):
(TestSuite):
(AsyncTestSuite.prototype.runTestCases): Stick the extra functions in the chain.
(AsyncTestSuite):
(SyncTestSuite.prototype.runTestCases):
(SyncTestSuite):

LayoutTests:

Extend existing unit tests to also exercise setup and teardown actions.

* inspector/unit-tests/async-test-suite.html:
* inspector/unit-tests/async-test-suite-expected.txt:
* inspector/unit-tests/sync-test-suite.html:
* inspector/unit-tests/sync-test-suite-expected.txt:


Canonical link: https://commits.webkit.org/166791@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@189213 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
burg committed Sep 1, 2015
1 parent 2b0e3bc commit 4d07116974f3deca6df7b5ef02b82cc16ce372d4
Showing 7 changed files with 559 additions and 22 deletions.
@@ -1,3 +1,17 @@
2015-09-01 Brian Burg <bburg@apple.com>

Web Inspector: add TestSuite hooks for per-testcase setup and teardown actions
https://bugs.webkit.org/show_bug.cgi?id=148641

Reviewed by Timothy Hatcher.

Extend existing unit tests to also exercise setup and teardown actions.

* inspector/unit-tests/async-test-suite.html:
* inspector/unit-tests/async-test-suite-expected.txt:
* inspector/unit-tests/sync-test-suite.html:
* inspector/unit-tests/sync-test-suite-expected.txt:

2015-09-01 Zalan Bujtas <zalan@apple.com>

Repaint cleanup:
@@ -7,6 +7,12 @@ PASS: should not be able to add non-object test case.
PASS: test case should require string name.
PASS: test case should require non-whitespace name.
PASS: test case should require test function.
PASS: should not be able to specify non-Function `setup` parameter.
PASS: should not be able to specify non-Function `setup` parameter.
PASS: should not be able to specify non-Function `setup` parameter.
PASS: should not be able to specify non-Function `teardown` parameter.
PASS: should not be able to specify non-Function `teardown` parameter.
PASS: should not be able to specify non-Function `teardown` parameter.
PASS: should not be able to run empty test suite.

== Running test suite: AsyncTestSuite.RunTwiceSuite
@@ -38,3 +44,36 @@ PASS: abortOnFailureSuite should have passed one test.
PASS: abortOnFailureSuite should have failed one test.
PASS: abortOnFailureSuite should have skipped one test.

== Running test suite: AsyncTestSuite.SetupAndTeardown
-- Running test setup.
-- Running test case: TestWithSetupAndTeardown
PASS: Test should see side effects of running setup() action.
-- Running test teardown.
PASS: Teardown should see side effects of running setup() action.

-- Running test case: TestRunningAfterTeardown
PASS: Test should see side effects of previous test's teardown() action.
PASS: Promise from setupAndTeardownTestSuite.runTestCases() should resolve.

== Running test suite: AsyncTestSuite.SetupException
-- Running test setup.
!! EXCEPTION:
PASS: Promise from setupExceptionTestSuite.runTestCases() should reject.

== Running test suite: AsyncTestSuite.SetupFailure
-- Running test setup.
!! EXCEPTION: undefined
PASS: Promise from setupFailureTestSuite.runTestCases() should reject.

== Running test suite: SyncTestSuite.TeardownException
-- Running test case: TestWithExceptionDuringTeardown
-- Running test teardown.
!! EXCEPTION:
PASS: Promise from teardownExceptionTestSuite.runTestCases() should reject.

== Running test suite: SyncTestSuite.TeardownFailure
-- Running test case: TestWithExceptionDuringTeardown
-- Running test teardown.
!! EXCEPTION: undefined
PASS: Promise from teardownFailureTestSuite.runTestCases() should reject.

@@ -73,6 +73,66 @@
} catch (e) {
ProtocolTest.log("PASS: test case should require test function.");
}
try {
badArgsSuite.addTestCase({
name: "foo",
test: () => {},
setup: "astd"
});
ProtocolTest.log("FAIL: should not be able to specify non-Function `setup` parameter.");
} catch (e) {
ProtocolTest.log("PASS: should not be able to specify non-Function `setup` parameter.");
}
try {
badArgsSuite.addTestCase({
name: "foo",
test: () => {},
setup: 123
});
ProtocolTest.log("FAIL: should not be able to specify non-Function `setup` parameter.");
} catch (e) {
ProtocolTest.log("PASS: should not be able to specify non-Function `setup` parameter.");
}
try {
badArgsSuite.addTestCase({
name: "foo",
test: () => {},
setup: {}
});
ProtocolTest.log("FAIL: should not be able to specify non-Function `setup` parameter.");
} catch (e) {
ProtocolTest.log("PASS: should not be able to specify non-Function `setup` parameter.");
}
try {
badArgsSuite.addTestCase({
name: "foo",
test: () => {},
teardown: "astd"
});
ProtocolTest.log("FAIL: should not be able to specify non-Function `teardown` parameter.");
} catch (e) {
ProtocolTest.log("PASS: should not be able to specify non-Function `teardown` parameter.");
}
try {
badArgsSuite.addTestCase({
name: "foo",
test: () => {},
teardown: 123
});
ProtocolTest.log("FAIL: should not be able to specify non-Function `teardown` parameter.");
} catch (e) {
ProtocolTest.log("PASS: should not be able to specify non-Function `teardown` parameter.");
}
try {
badArgsSuite.addTestCase({
name: "foo",
test: () => {},
teardown: {}
});
ProtocolTest.log("FAIL: should not be able to specify non-Function `teardown` parameter.");
} catch (e) {
ProtocolTest.log("PASS: should not be able to specify non-Function `teardown` parameter.");
}

let runEmptySuite = ProtocolTest.createAsyncSuite("AsyncTestSuite.RunEmptySuite");
try {
@@ -176,6 +236,157 @@
return Promise.resolve(); // Continue this test.
});

var setupAndTeardownSymbol = Symbol("async-suite-setup-and-teardown-token");
window[setupAndTeardownSymbol] = 0;

let setupAndTeardownTestSuite = ProtocolTest.createAsyncSuite("AsyncTestSuite.SetupAndTeardown");
setupAndTeardownTestSuite.addTestCase({
name: "TestWithSetupAndTeardown",
description: "Check execution order for setup and teardown actions.",
setup: (resolve, reject) => {
window[setupAndTeardownSymbol] = 1;
resolve();
},
test: (resolve, reject) => {
ProtocolTest.expectThat(window[setupAndTeardownSymbol] === 1, "Test should see side effects of running setup() action.");
window[setupAndTeardownSymbol] = 2;
resolve();
},
teardown: (resolve, reject) => {
ProtocolTest.expectThat(window[setupAndTeardownSymbol] === 2, "Teardown should see side effects of running setup() action.");
window[setupAndTeardownSymbol] = 3;
resolve();
}
});
setupAndTeardownTestSuite.addTestCase({
name: "TestRunningAfterTeardown",
description: "Check execution order for test after a teardown action.",
test: (resolve, reject) => {
ProtocolTest.expectThat(window[setupAndTeardownSymbol] === 3, "Test should see side effects of previous test's teardown() action.");
resolve();
},
});

result = result.then(() => {
return setupAndTeardownTestSuite.runTestCases();
}).then(function resolved() {
ProtocolTest.log("PASS: Promise from setupAndTeardownTestSuite.runTestCases() should resolve.");
return Promise.resolve(); // Continue this test.
}, function rejected(e) {
ProtocolTest.log("FAIL: Promise from setupAndTeardownTestSuite.runTestCases() should resolve.");
return Promise.resolve(); // Continue this test.
});

let setupExceptionTestSuite = ProtocolTest.createAsyncSuite("AsyncTestSuite.SetupException");
setupExceptionTestSuite.addTestCase({
name: "TestWithExceptionDuringSetup",
description: "Check execution order for setup action that throws an exception.",
setup: (resolve, reject) => { throw new Error() },
test: (resolve, reject) => {
ProtocolTest.assert(false, "Test should not execute if its setup action threw an exception.");
reject();
},
teardown: (resolve, reject) => {
ProtocolTest.assert(false, "Teardown action should not execute if its setup action threw an exception.");
reject();
}
});

result = result.then(() => {
return setupExceptionTestSuite.runTestCases();
}).then(function resolved() {
ProtocolTest.log("FAIL: Promise from setupExceptionTestSuite.runTestCases() should reject.");
return Promise.resolve(); // Continue this test.
}, function rejected(e) {
ProtocolTest.log("PASS: Promise from setupExceptionTestSuite.runTestCases() should reject.");
return Promise.resolve(); // Continue this test.
});

let setupFailureTestSuite = ProtocolTest.createAsyncSuite("AsyncTestSuite.SetupFailure");
setupFailureTestSuite.addTestCase({
name: "TestWithFailureDuringSetup",
description: "Check execution order for setup action that has failed.",
setup: (resolve, reject) => { reject(); },
test: (resolve, reject) => {
ProtocolTest.assert(false, "Test should not execute if its setup action failed.")
reject();
},
teardown: (resolve, reject) => {
ProtocolTest.assert(false, "Teardown action should not execute if its setup action failed.")
reject();
}
});

result = result.then(() => {
return setupFailureTestSuite.runTestCases();
}).then(function resolved() {
ProtocolTest.log("FAIL: Promise from setupFailureTestSuite.runTestCases() should reject.");
return Promise.resolve(); // Continue this test.
}, function rejected(e) {
ProtocolTest.log("PASS: Promise from setupFailureTestSuite.runTestCases() should reject.");
return Promise.resolve(); // Continue this test.
});

let teardownExceptionTestSuite = ProtocolTest.createAsyncSuite("SyncTestSuite.TeardownException");
teardownExceptionTestSuite.addTestCase({
name: "TestWithExceptionDuringTeardown",
description: "Check execution order for teardown action that throws an exception.",
test: (resolve, reject) => { resolve(); },
teardown: (resolve, reject) => { throw new Error() }
});
teardownExceptionTestSuite.addTestCase({
name: "TestAfterTeardownException",
descrption: "Check execution order for test after previous test's teardown throws an exception.",
setup: (resolve, reject) => {
ProtocolTest.assert(false, "Setup action should not execute if previous test's teardown action threw an exception.");
reject();
},
test: (resolve, reject) => {
ProtocolTest.assert(false, "Test should not execute if previous test's teardown action threw an exception.");
reject();
}
});

result = result.then(() => {
return teardownExceptionTestSuite.runTestCases();
}).then(function resolved() {
ProtocolTest.log("FAIL: Promise from teardownExceptionTestSuite.runTestCases() should reject.");
return Promise.resolve(); // Continue this test.
}, function rejected(e) {
ProtocolTest.log("PASS: Promise from teardownExceptionTestSuite.runTestCases() should reject.");
return Promise.resolve(); // Continue this test.
});

let teardownFailureTestSuite = ProtocolTest.createAsyncSuite("SyncTestSuite.TeardownFailure");
teardownFailureTestSuite.addTestCase({
name: "TestWithExceptionDuringTeardown",
description: "Check execution order for teardown action that has failed.",
test: (resolve, reject) => { resolve(); },
teardown: (resolve, reject) => { reject(); },
});
teardownFailureTestSuite.addTestCase({
name: "TestAfterTeardownException",
description: "Check execution order for test after previous test's teardown throws an exception",
setup: (resolve, reject) => {
ProtocolTest.assert(false, "Setup action should not execute if previous test's teardown action failed.");
reject();
},
test: (resolve, reject) => {
ProtocolTest.assert(false, "Test should not execute if previous test's teardown action failed.");
reject();
}
});

result = result.then(() => {
return teardownFailureTestSuite.runTestCases();
}).then(function resolved() {
ProtocolTest.log("FAIL: Promise from teardownFailureTestSuite.runTestCases() should reject.");
return Promise.resolve(); // Continue this test.
}, function rejected(e) {
ProtocolTest.log("PASS: Promise from teardownFailureTestSuite.runTestCases() should reject.");
return Promise.resolve(); // Continue this test.
});

// This will finish the test whether the chain was resolved or rejected.
result = result.then(() => { ProtocolTest.completeTest(); });
}
@@ -7,6 +7,12 @@ PASS: should not be able to add non-object test case.
PASS: test case should require string name.
PASS: test case should require non-whitespace name.
PASS: test case should require test function.
PASS: should not be able to specify non-Function `setup` parameter.
PASS: should not be able to specify non-Function `setup` parameter.
PASS: should not be able to specify non-Function `setup` parameter.
PASS: should not be able to specify non-Function `teardown` parameter.
PASS: should not be able to specify non-Function `teardown` parameter.
PASS: should not be able to specify non-Function `teardown` parameter.
PASS: should not be able to run empty test suite.

== Running test suite: SyncTestSuite.RunTwiceSuite
@@ -35,3 +41,31 @@ PASS: abortOnFailureSuite should have passed one test.
PASS: abortOnFailureSuite should have failed one test.
PASS: abortOnFailureSuite should have skipped one test.

== Running test suite: SyncTestSuite.SetupAndTeardown
-- Running test setup.
-- Running test case: TestWithSetupAndTeardown
PASS: Test should see side effects of running setup() action.
-- Running test teardown.
PASS: Teardown should see side effects of running setup() action.

-- Running test case: TestRunningAfterTeardown
PASS: Test should see side effects of previous test's teardown() action.

== Running test suite: SyncTestSuite.SetupException
-- Running test setup.
!! EXCEPTION:

== Running test suite: SyncTestSuite.SetupFailure
-- Running test setup.
!! EXCEPTION

== Running test suite: SyncTestSuite.TeardownException
-- Running test case: TestWithExceptionDuringTeardown
-- Running test teardown.
!! EXCEPTION:

== Running test suite: SyncTestSuite.TeardownFailure
-- Running test case: TestWithExceptionDuringTeardown
-- Running test teardown.
!! EXCEPTION:

0 comments on commit 4d07116

Please sign in to comment.