Better pending test case output

[shx7]

Hello guys, I’m working in AppCode, and we are very excited to add support for Quick framework to our IDE soon. But we encountered several problems during implementation of pending test cases support. Essentially it’s not very convenient for us to parse ‘pending’ test cases because pending() produces pretty ambiguous output. Here is an example:

class PendingTestsEx : QuickSpec {
  func spec() {
    it("it") {}
    context(“context”) {
      it(“it”) { /* .. */ }
      pending(“it”) { /* .. */ }
      pending(“it”) { /* .. */ }
      it(“it”) { /* .. */ }
     }   
  }
}

Xcode console output:

Pending: it
Pending: it
Pending: it
Test Suite 'Selected tests' started at 2018-02-06 17:18:08.129
Test Suite 'Quick - macOSTests.xctest' started at 2018-02-06 17:18:08.135
Test Suite 'PendingTestsEx' started at 2018-02-06 17:18:08.136
Test Case '-[Quick___macOSTests.PendingTestsEx context__it]' started.
Test Case '-[Quick___macOSTests.PendingTestsEx context__it]' passed (0.985 seconds).
Test Case '-[Quick___macOSTests.PendingTestsEx context__it_2]' started.
Test Case '-[Quick___macOSTests.PendingTestsEx context__it_2]' passed (0.005 seconds).
Test Suite 'PendingTestsEx' passed at 2018-02-06 17:18:09.129.
Executed 2 tests, with 0 failures (0 unexpected) in 0.990 (0.993) seconds
Test Suite 'Quick - macOSTests.xctest' passed at 2018-02-06 17:18:09.131.
Executed 2 tests, with 0 failures (0 unexpected) in 0.990 (0.996) seconds
Test Suite 'Selected tests' passed at 2018-02-06 17:18:09.132.
Executed 2 tests, with 0 failures (0 unexpected) in 0.990 (1.003) seconds
Program ended with exit code: 0

As you can see, It's unclear, which one of 'it' is pending. For 'pending' cases I'd love to see test id's as for usual tests. Probably output may be something like (now we can identify where pending tests are, and put them into test tree in right place):

Pending: Quick___macOSTests.PendingTestsEx it
Pending: Quick___macOSTests.PendingTestsEx context__it
Pending: Quick___macOSTests.PendingTestsEx context__it_3
Test Suite 'Selected tests' started at 2018-02-06 17:18:08.129
Test Suite 'Quick - macOSTests.xctest' started at 2018-02-06 17:18:08.135
Test Suite 'PendingTestsEx' started at 2018-02-06 17:18:08.136
Test Case '-[Quick___macOSTests.PendingTestsEx context__it_1]' started.
Test Case '-[Quick___macOSTests.PendingTestsEx context__it_1]' passed (0.985 seconds).
Test Case '-[Quick___macOSTests.PendingTestsEx context__it_2]' started.
Test Case '-[Quick___macOSTests.PendingTestsEx context__it_2]' passed (0.005 seconds).
Test Suite 'PendingTestsEx' passed at 2018-02-06 17:18:09.129.
Executed 2 tests, with 0 failures (0 unexpected) in 0.990 (0.993) seconds
Test Suite 'Quick - macOSTests.xctest' passed at 2018-02-06 17:18:09.131.
Executed 2 tests, with 0 failures (0 unexpected) in 0.990 (0.996) seconds
Test Suite 'Selected tests' passed at 2018-02-06 17:18:09.132.
Executed 2 tests, with 0 failures (0 unexpected) in 0.990 (1.003) seconds
Program ended with exit code: 0

Also if it's possible, we'd love to see similar behaviour for xit, xdescribe, xcontext, xitBehavesLike (probably enabled by additional knob like environmental var). To sum up: all we need is to have an opportunity to identify pending test cases in the test tree by unique id.
Thank you in advance!

Environment:

• Quick: 1.2.0
• Xcode Version: 9.0 (9A235)
• Swift Version: 4.0

[eduardbosch]

Hi,

Should xit, xdescribe, ... appear as "ignored" in the test results in AppCode?

When I set xit, xdescribe, ... they never appear to the test results. I think will be good that they appear as ignored so we know we have pending tests to implement.

They don't appear either in Xcode.

[ikesyo]

This is related to #490, specifically to #490 (comment).

[jwfriese]

Does anyone still desire this feature? I have a PR related to it (#771), but resolving conflicts and merging it might be some significant work, and I'd rather focus my free time's energy on other issues if this one is not a high priority.

[fizker]

@jwfriese I would love for pending tests to be visible. I always spec out the expected behaviour with pending its first, and then start implementing the tests one by one, while also implementing the code necessary to the new tests go green.

For me, pending tests are sort-of a todo list written in test files. Also, a CI environment cannot fail because of ignored tests if it does not know about the ignored tests.