False positive test results when using recordFailure

[smaljaar]

Hello,

I noticed an issue with getting false positive when using the XCTest method recordFailure.

extension XCTestCase {

    func waitForElementToAppear(element: XCUIElement, file: String = #file, line: UInt = #line) {
        let existsPredicate = NSPredicate(format: "exists == true")
        expectation(for: existsPredicate, evaluatedWith: element, handler: nil)

        waitForExpectations(timeout: 2) { (error) -> Void in
            if (error != nil) {
                let message = "Failed to find \(element) after 2 seconds."

                //Issue: self.recordFailure does not result in a failed test case the second time it is called. 
                //This issue does not occur while using regular subclasses of XCTestCase, so something in XCTest-Gherkin causes this not to work properly.
                self.recordFailure(withDescription: message, inFile: file, atLine: UInt(arc4random()), expected: true)

            }
        }
    }
}

The issue occurs if you call the method waitForElementToAppear for a second time from a StepDefiner subclass.

[nap-sam-dean]

Hi, I've tried this test case (in the UI test target) and it behaves as expected (i.e. 1 passing test, and then 2 failing ones).

extension XCTestCase {

    func waitForElementToAppear(element: XCUIElement, file: String = #file, line: UInt = #line) {
        let existsPredicate = NSPredicate(format: "exists == true")
        expectation(for: existsPredicate, evaluatedWith: element, handler: nil)

        waitForExpectations(timeout: 2) { (error) -> Void in
            if (error != nil) {
                let message = "Failed to find \(element) after 2 seconds."

                //Issue: self.recordFailure does not result in a failed test case the second time it is called.
                //This issue does not occur while using regular subclasses of XCTestCase, so something in XCTest-Gherkin causes this not to work properly.
                self.recordFailure(withDescription: message, inFile: file, atLine: line, expected: true)
                
            }
        }
    }
}

class FailingSecondTest: XCTestCase {

    func test0() {
        return
    }

    func test1() {
        let element = XCUIApplication().buttons["hunter1"]
        self.waitForElementToAppear(element: element)
    }

    func test2() {
        let element = XCUIApplication().buttons["hunter2"]
        self.waitForElementToAppear(element: element)
    }
}

Can you modify this so that it shows your issue please?

[smaljaar]

@nap-sam-dean
The false positive only happens when you use NativeTestCase to run .feature files. Here is a zip that contains the files to show the issue. We stopped using self.recordFailure and just use XCTFail(message:) and pass the current step and test name as failure message, which works fine.

DemoFalsePositive.zip

I'm pretty sure this is the same issue as just again reported in #54

[SpeedyNassir]

@smaljaar @nap-sam-dean The issue even occurs when you use swift feature files(Need not be native one). Please refer #54 for code snippet for same.