Test Discovery

Emcee allows you to query your test bundle and extract discovered tests from it. There are different ways of doing it.

parseFunctionSymbols test discovery mode

This is the most easy way of discovering tests, but it is also the most limited and the most error prone way. Emcee will run

nm -j -U bundle.xctest/TestBinary

to extract all symbols from the test bundle. Then Emcee will load libswiftDemangle.dylib from the provided developer dir, demangle all Swift symbols, and extract all method names that have a signature in form of @objc ModuleName.ClassName.testMethodName() -> ().

There could be false positive and false negative results. This method does not provide any additional information about tests, except test class and method names. Good side: no source code modification is required.

runtimeXCTest test discovery mode

Availability: 20.0.0

A more reliable but bit slower method of discovering tests. It uses native XCTest framework and thus may find tests that get registered in runtime via overriding +(NSArray*)testInvocations. This mode is useful if you use test frameworks like Quick or Kiwi.

In order to work properly it requires installed Xcode and simulator runtime that you use to run tests against.

runtimeXCTest mode uses EmceeTestsInspector binary that is provided as prebuilt artifact in archive with emcee main binary. Be sure to keep it in the same directory as emcee binary on machines where you're going to run emcee client.

Runtime Dump

If you want to have a richer test discovery experience, you can annotate your tests and inject a runtime dump support.

Using this method requires test source code modification, but it allows you to:

• Discover all tests that are truly exist in a test bundle

• Extract additional test information, e.g. path to source file where the test is coded

• Annotate your tests with additional information using tag system

This method allows you to get a completely filled DiscoveredTestEntry object. You can read a bit more about runtime dump here.

Again, there are different ways of performing test discovery using runtime dump technique.

Emcee has a sample code for runtime dump implementation that can be adopted for your needs.

runtimeExecutableLaunch test discovery mode

This technique launches your app, which in turn will load your xctest bundle, and perform runtime dump. Good side: Emcee will launch your app without booting up Simulator, which is fast, and enough to hit your app's main() entry point. Inside your main() look up for the following environment variables:

• EMCEE_RUNTIME_TESTS_EXPORT_PATH: this indicates where you are expected to write out runtime dump JSON file
• EMCEE_XCTEST_BUNDLE_PATH: this is an absolute path to a test bundle which you are expected to load, and then dump all available tests from it into EMCEE_RUNTIME_TESTS_EXPORT_PATH file.

Example implementation that we use in Avito:

int main(int argc, char* argv[]) {
    #if TEST
        [[TestsBundleLoader new] exportTestsAndExitProcessIfNeeded];
    #endif
    // normal execution flow

TestsBundleLoader class:

#if TEST
import Foundation

@objc public class TestsBundleLoader: NSObject {
    @objc public func exportTestsAndExitProcessIfNeeded() {
        let environment = ProcessInfo.processInfo.environment
        guard
            let emceeOutputPath = environment["EMCEE_RUNTIME_TESTS_EXPORT_PATH"],
            let xctestBundlePath = environment["EMCEE_XCTEST_BUNDLE_PATH"],
            emceeOutputPath.isNotEmpty,
            xctestBundlePath.isNotEmpty else {
                return
        }
        
        guard let bundle = Bundle(path: xctestBundlePath) else {
            fatalError("Failed to find loadable bundle in \(xctestBundlePath)")
        }
        
        bundle.load()
        
        guard let exporter = bundle.principalClass as? TestsExporter.Type else {
            fatalError("Principal class \(String(describing: bundle.principalClass)) is not a TestExporter")
        }
        
        exporter.exportTests(path: emceeOutputPath)
        
        exit(0)
    }
}

runtimeLogicTest test discovery mode

This method will launch your test bundle in logic test mode, instantiating your test bundle's NSPrincipalClass, from which you can dump all test methods available in runtime. This is faster than runtimeAppTest, but slower than runtimeExecutableLaunch. Only tests source code is modified.

runtimeAppTest test discovery mode

This method will launch your test bundle in application test mode, launching your app first and then instantiating your NSPrincipalClass of your test bundle, from which you can dump all test methods. This is the slowest way of test discovery. Only tests source code is modified.

Enriching your test discovery

Since all runtime* test discovery methods are happening in runtime, you can return richer information about test. This will require you to annotate your tests though. We use the following annotation of each test which we find handy, especially for UI tests:

final class SomeUITest: TestCase {
    override var info: TestCaseInfo? {
        TestCaseInfo(
            id: 42,
            name: "Test logout",
            tags: ["basic", "regression"]
        )
    }
    
    func test() {
        // ...
    }
}

TestCaseInfo can be as simple as:

public final class TestCaseInfo {
    public let id: Int?
    public let name: String?
    public let tags: [String]
    public let file: String
    public let line: Int

    public init(
        id: Int? = nil,
        name: String? = nil,
        tags: [String] = [],
        file: String = #file,
        line: Int = #line
    ) { ... }
}

Since in runtime dump we instantiate all XCTestCase objects, we can try to cast them to TestCase and extract the value for info: TestCaseInfo field.