XCTestAssertions do not work within a QuickSpec written in Objective-C

[krzysztofzablocki]

When using XCTest assertions or 3rd Party libraries(OCMock/Mockito), failures are not handled properly on QuickSpec.
the proper method is getting called

- (void)recordFailureWithDescription:(NSString *)description inFile:(NSString *)filePath atLine:(NSUInteger)lineNumber expected:(BOOL)expected;

but Xcode still doesn't show test as failed.

Running through debugger I noticed that when running through nimble expectation example is set correctly, but if run from any other lib or XCTest we end up with empty example, might be something related to that?

To reproduce you could just add

XCTFail(@"Fail!");

[modocache]

@krzysztofzablocki This is a great bug, thank you! 🐻 💓

I've been very interested in confirming OCMock works correctly. Could you please show me some sample use cases? In the meantime I'll try looking at XCTFail in particular.

[krzysztofzablocki]

@modocache just adding any of those (or all of them)

qck_it(@"fails",^{
  XCTAssertEqual(@NO, @YES);
  XCTFail(@"fail");
  [self recordFailureWithDescription:@"test" inFile:@__FILE__ atLine:__LINE__ expected:YES];
});  

should fail test, yet it doesn't while using quick, thats why I've issue with OCMock/Mockito and I've had to switch their source code to throw exceptions instead of using standard XCTest reporting

[modocache]

@krzysztofzablocki I'm looking into mocking/stubbing in Swift. Seems like an interesting problem: OCMock is pessimistic as to whether it will ever fully support Swift, and NSHipster claims mocking/stubbing libraries aren't needed in Swift in the first place--although I disagree with this claim.

That being said, I'm still not 100% sure what the issue you are experiencing is. Are you saying that linking OCMock and importing it's header causes the assertions in your code sample to not fail? That is, the following is the issue?

#import <OCMock/OCMock.h>

QuickSpecBegin(MySpec)

// should fail, but does not?
qck_it(@"fails",^{
  XCTAssertEqual(@NO, @YES);
  XCTFail(@"fail");
  [self recordFailureWithDescription:@"test" inFile:@__FILE__ atLine:__LINE__ expected:YES];
}); 

QuickSpecEnd

[jeffh]

Currently, it's just not possible in Swift. Swift behaves more like C++, so there isn't necessarily a dynamic dispatch table to replace at runtime (which mocking libraries rely on).

For now, recommending to use OCMock for objective-c works for now.

[modocache]

I've confirmed an issue with using Quick in Objective-C with OCMock. The following test succeeds, but it should fail:

#import <Quick/Quick.h>
#import <OCMock/OCMock.h>

@interface Whale : NSObject
- (NSString *)sing;
@end

@implementation Whale
- (NSString *)sing {
    return @"Quooo!!";
}
@end

QuickSpecBegin(WhaleSpec)

qck_it(@"sings", ^{
  id mock = OCMClassMock([Whale class]);
  OCMStub([mock sing]).andReturn(@"Under the sea!");
  XCTAssertEqualObjects([mock sing], @"Hakuna matata!", @"expected one song, got another");
});

QuickSpecEnd

I can't imagine what the problem is... maybe it has something to do with the -ObjC linker flag OCMock requires? Still investigating.

[krzysztofzablocki]

@jeffh @modocache the issue isn't with how mocking libraries works at all (they will work fine for any objected that bases on ObjC or uses @objc annotation).

The real problem is that in quick spec XCTest assertions doesn't seem to work, XCTAssert / XCTFail / recordFailure should all be reported as failed test, they don't, instead tests with them are marked as success.

[modocache]

@krzysztofzablocki Ah, OK, gotcha. Sorry for the confusion! 😖

It's actually pretty weird this is an issue: for a while, Quick had a ton of logic around reporting failures, but now, outside of checking whether the failure is in a shared example, it pretty much behaves exactly like a normal XCTestCase would. Definitely worth looking in to. 🎩 💡

[krzysztofzablocki]

From debugging it seems that the issue might be due to a fact that when using XCT reportFailure directly (or via XCT assertions) the example group is nil, if you run Nimble matcher then it's set correctly, ideas why this might be happening?

Note for anyone looking for a workaround to be able to use mocking lib or similar. this works best for now:
change mocking lib expectations to use Exceptions instead of XCT reporting, then use nimble matchers to have exception not thrown. If you don't use nimble matcher for exceptions the exception will still be reported as failed test, but it will be very SLOW. OCMock change looks like this https://github.com/krzysztofzablocki/ocmock/commit/ac833a023bf8e1adda2b1e24953d985df21cd196https://github.com/krzysztofzablocki/ocmock/commit/ac833a023bf8e1adda2b1e24953d985df21cd196

[jeffh]

An interesting idea is for Quick to hook into test failures using NSAssertionHandlers. I'm thinking XCTest assertions might be calling XCTestCase's test failure mechanism directly, but it might be forwarded through some other handler inside NSAssertionHandler which would make it probably not invoke the Quick-overridden one.

[modocache]

This is definitely an issue. Ideally, an XCTFail inside an Objective-C it block would cause a spec to fail, but currently it doesn't.

Which is really quite strange, actually: even if you remove Quick's override of -[XCTestCase recordFailureWithDescription:inFile:atLine:expected:], XCTFail just doesn't work. What's going on here? 😖 💦

[modocache]

I believe the core of the issue is that, when an example closure is executed from within a QuickSpec written in Objective-C, self.invocation is nil. When used in Objective-C, the XCTAssert family of macros rely on self.invocation being non-nil.

To demonstrate, the following is correctly reported as a test failure in Swift:

class MySpec: QuickSpec {
  override func spec() {
    it("fails") { XCTFail() }
  }
}

But the same thing written in Objective-C is not correctly reported as a failure:

QuickSpecBegin(MySpec)
it(@"fails", ^{ XCTFail(); });
QuickSpecEnd

The Swift implementations of the XCTAssert family of macros must be less brittle than their Objective-C counterparts, and correctly deal with self.invocation being nil.

It might be possible to fix this issue by setting the invocation somehow. Or just add a note to the README that you can only use XCTAssert macros in Swift.

So @krzysztofzablocki, the short answer to your problem is: write your tests in Swift! 💔

[jeffh]

@modocache, this bug is nuanced. See this line. That line builds the test graph. self refers the instance created in +initialize and not the ones XCTest will create and run. The instance in +initialize will not have any invocation set on it.

To get around this, it does seem like Swift's XCTest is using some other sideline communication.

PS - As a side note, having breakpoints inside a Quick spec would be nice, it's a bit too macro heavy for lldb right now.

[modocache]

@jeffh Absolutely right. The invocation isn't set until later, and its on a different test case instance: https://github.com/Quick/Quick/blob/master/Quick/QuickSpec.m#L73

It may be possible to have each example maintain a reference to the test case, then set the invocation on that test case once the invocation is called on that line above... but that's definitely not very elegant. Perhaps it's time to re-think how we build example groups...

[modocache]

Now that #304 has been merged, I think this issue is the biggest blocker for Quick v1.0.0 at the moment. I'd love to get to the bottom of this...!

[orta]

For additional info, this in Specta fails correctly:

  it(@"Fails via XCTFail", ^{
    XCTFail(@"fail");
  });

[modocache]

I believe I've isolated the difference between how Quick and Specta behave to this line, which routes the failure recording to the current spec's test run. Commenting out that line exhibits the same behavior as Quick.

We may need to store a global pointing to the current spec being run, like Specta does. Still looking into it...