Swift Async/Await Support

[younata]

This PR provides basic swift concurrency support. Now, you can await functions inside of beforeEach, afterEach, aroundEach, it, fit, and xit blocks.

Note: The various itBehavesLike closures will not take in async closures. These closures should not have any logic in them and should instead just be simple assignment/passing in pre-assigned variables.

For example, the following tests are now possible:

describe("my awesome tests") {
    beforeEach {
        await someAsyncSetupFunction()
    }

    afterEach {
        await teardownButAsync()
    }

    it("can run tests asynchronously!") {
        let result = await methodBeingTested()
        expect(result).to(equal(expectedValue))
    }
}

Breaking Changes

Where possible, I took efforts to try to ease this transition as much as possible. However, this will almost certainly require you to make changes to your test suite.

• Swift tests (it, beforeEach, afterEach and aroundEach blocks) no longer run on the main thread by default.
  • You must specify @MainActor in the relevant closure to get the code to run on the main thread, like so:

    it("runs on the main thread") { @MainActor in
        // contents of this closure will run in the main actor.
    }

  • Objective-C tests still run on the main thread.
• All Swift tests now run in an async context.
  • This will almost certainly be a source of breakage, as Apple shipped many APIs that run on the main actor which work just fine when used in a synchronous context.
  • Objective-C tests still run in a synchronous context, because Swift Concurrency isn't supported in Objective-C.
• aroundEach must now await the runExample closure, like so:

  aroundEach { runExample in
      await runExample()
  }

• aroundEach is no longer available in Objective-C tests.
  • After spending a couple months trying to get aroundEach in Objective-C to work, I decided that it wasn't worth the effort and dropped it. My apologies to those who use aroundEach in Objective-C.

Todo

• Documentation. This will change the model of how tests execute (previously, it was synchronous, on the main thread. Now it's going to be async, off the main thread), and we'll need to prepare users for that change.

Future Work

• Nimble also will need to add Async support, which XCTest's assertion functions don't even support as of this writing.

[github-actions]

	1 Warning
⚠️	Big PR

Generated by 🚫 Danger

[younata]

For those following along, changes that I just pushed include:

• Rebased on top of current main. Includes support for the new justBeforeEach feature.
• Fully backwards compatible with current Quick. That is, this is no longer a breaking change!
• Fixed issue with declaring & assigning variables that are then picked up in the beforeEach/afterEach/aroundEach/justBeforeEach/it blocks.

[younata]

Whoops. Putting this back to a draft while I figure out the issues with 13.3.1. Not quite ready to drop that version of Xcode yet.

[shinichy]

@younata I'd like to use async/await in Quick. Is there any plan to merge this PR soon?

[younata]

@younata I'd like to use async/await in Quick. Is there any plan to merge this PR soon?

This PR still has a number of issues with it. Notably getting aroundEach to work in Objective-C (where we can't use async). That's the main blocker to this, and something I'm still working on resolving. Assuming I don't come across other blockers, then this can be merged and released.

Afterwards, I plan to close out the other Quick 6.0 tasks and then release.

[shinichy]

@younata Thanks for letting me know the status. I look forward to this new async support!

[stefanrenne]

I just tried this in our codebase, new async tests work like a charm! but some of our old RxSwift tests are failing now, this seems to be related to that this pr executes as much as possible on the main thread icm with an RxSwift observable also executing code on the main thread

Here is an example failing rxswift test, where this one would succeed before this pr, and it also succeeds after removing the dispatching result on the main thread .observe(on: MainScheduler.instance)

context("sample rxswift dispatch result on main thread test") {

    var result: Result<Bool, Error>!

    beforeEach {

        let blockingSingle = Single<Bool>.create { observer in
            DispatchQueue.global().asyncAfter(deadline: .now() + .seconds(1)) {
                observer(.success(true))
            }
            return Disposables.create()
        }
        .observe(on: MainScheduler.instance)
        .toBlocking(timeout: 2)

        result = Result(catching: { try blockingSingle.single() })
    }

    it("equals true") {
        expect(try result.get()).to(beTrue())
    }
}

Edit1:
I also just discovered that an simple DispatchQueue.main.after test now also fails

context("sample dispatchQueue asyncAfter on main thread test") {

    var result: Bool = false

    beforeEach {
        DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(500)) {
            result = true
        }
    }

    it("eventually equals true") {
        expect(result).toEventually(beTrue(), timeout: .seconds(1))
    }
}

[younata]

I just tried this in our codebase, new async tests work like a charm! but some of our old RxSwift tests are failing now, this seems to be related to that this pr executes as much as possible on the main thread icm with an RxSwift observable also executing code on the main thread

Here is an example failing rxswift test, where this one would succeed before this pr, and it also succeeds after removing the dispatching result on the main thread .observe(on: MainScheduler.instance)

context("sample rxswift dispatch result on main thread test") {

    var result: Result<Bool, Error>!

    beforeEach {

        let blockingSingle = Single<Bool>.create { observer in
            DispatchQueue.global().asyncAfter(deadline: .now() + .seconds(1)) {
                observer(.success(true))
            }
            return Disposables.create()
        }
        .observe(on: MainScheduler.instance)
        .toBlocking(timeout: 2)

        result = Result(catching: { try blockingSingle.single() })
    }

    it("equals true") {
        expect(try result.get()).to(beTrue())
    }
}

Edit1: I also just discovered that an simple DispatchQueue.main.after test now also fails

context("sample dispatchQueue asyncAfter on main thread test") {

    var result: Bool = false

    beforeEach {
        DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(500)) {
            result = true
        }
    }

    it("eventually equals true") {
        expect(result).toEventually(beTrue(), timeout: .seconds(1))
    }
}

Thanks for the feedback @stefanrenne!

You'll note that in the changes I just pushed, I got rid of most of the @MainActor annotations. Especially given that developers can manually specify @MainActor, it's unnecessary (and as you noticed, can cause test failures) to force all tests to run on the main actor.

[younata]

Ok, this has been fun. Apparently, while I fixed the aroundEach objc issue for Swift 5.6 in macOS, tvOS, and Linux, I haven't fixed it in iOS. Doubly fun that I'm now seeing the issue in Xcode 14/iOS 16 simulators. I'm still figuring out the cause of this particular issue.

[stefanrenne]

Cool I just discovered that you can mark beforeEach / it to be run on the mainActor

it("supports XCTest expectations") { @MainActor in

I like it but for me this introduces some rework for our testsuite, so I would suggest to still make this a major release.

Also I think nimble needs an upgrade because now I'm getting this crash Thread 8: "expect(...).toEventually(...) can only run on the main thread."