A new testing harness

[orta]

Rack for Testing, using XPC to communicate between each part of the system.

With the MVP goal to be able to run a test harness inside a hosted app and to be able to pass tests in via XPC to be ran without restarting the sim.

Ideally can split into three parts [ UI ] <---> [ Test Compiler ] <---> [Test Runner ]

Wherein the same UI could exist for many types of languages and test compilers, runners don't have to be some async running process. Could just be the same thing ran instantly but provides a separation against crashes etc.

Initially letting @modocache take a stab at this, but we can take a shot anytime @alloy feels he has some space. Assigning to him, as it is would be largely his work I expect.

[orta]

Expanded out of #22

[orta]

XCTest is now OSS, could make life easier.

[modocache]

The open-source implementation of XCTest is a simple program that prints to stdout. Last we spoke, this project involved IPC, some reverse-engineering of iOS simulator internals, and more--open-source XCTest isn't going to be of much help there.

[orta]

^ This is a legit point

[orta]

Alright so, based on what I learned in code spelunking in Injection for Xcode, here's what I think should happen.

Let's just call it instatests for now. Create a gem called cocoapods-instatests.

Why a CocoaPods plugin? Well we need to add code to the app to start listening out for , people shouldn't need to have it on every computer and ideally it shouldn't get shipped with your app's source, sounds exactly like cocoapods-keys.

The code inside the pod needs to:

• Wait until an XCTest session is about to close, check for any instatests servers.
• If it doesn't find one, close like normal.
• If one exists, create a connection ( Injection uses sockets, could we use NSXPC?)
• Receive NSBundles, run the code as tests in XCTest
• The code can be added entire as a #ifdef DEBUG so it won't go to prod

This plugin has a command pod instatests which starts up a guard-like server. This detects saves, and creates a bundle representing the changes. These get passed to the pod's server, ran, then it gets test results back, these can use NSNotifications to let you know if it passed/failed.

Once a working build is done, then I can look at standardizing all the communication points.

[alloy]

could we use NSXPC?

Depends. How will the bundles with new code be compiled? By Xcode or do you want to do a stand-alone thing, possibly with a Clang compilation database?

[orta]

Simplest feels to be the same as injection - read the build log, find the section for that file, and reuse that. In injection that .o is linked into a blank bundle, which gets passed across the connection.

Though I do want to consult Diamond's reloaded before making the first approach

[orta]

Current feels on a name, xctestjam

[orta]

Don't think XPC is probably the right communication method, weirdly enough, JSCore should have a working websocket, could use that as the intermediary for the client/server to bi-directionally communicate. Meta.

[orta]

Got a web socket to connect between a ruby webserver and a WKWebView websocket - looks like the simplest way to start

[orta]

Now that I've started this, https://github.com/orta/cocoapods-xcautotest/

I'd like to try scope 1.0 as "runs tests, reliably" and 2.0 as "runs tests reliably, in the background, on a separate sim."

This looks a bit tough, but definitely doable, as it looks like I'll have to create ruby bindings for the frameworks generated by https://github.com/facebook/FBSimulatorControl - they have a command-line tool but it looks like it depends on having Carthage, which I'd rather not have as a dependency.

[alloy]

but it looks like it depends on having Carthage

Seems to only be for testing (OCMock).

[orta]

Nah, it is used for the the command-line tool itself - the frameworks themselves are dependecy-less - https://github.com/facebook/FBSimulatorControl/tree/master/fbsimctl

[lawrencelomax]

@orta @alloy As far as Carthage goes we need it for two reasons:

• For OCMock tests for the Frameworks. I'm all for ripping out this dependency by replacing with hand-written doubles.
• ForSwifter with fbsimctl, which can run with an embedded HTTP Server.

If you need to build the Frameworks (XCTestBoostrap/FBSimulatorControl/FBDeviceControl), then you don't need any use of Carthage whatsoever. If you want to use the fbsimctl command-line you'll need it for building the fbsimctl executable. If you have some suggestions here, please let me know.

If you're interested in linking against the XCTestBootstrap framework from Ruby with a FFI and avoiding the fbsimctl executable, you should look at FBTestManagerReporter. These are the Delegate methods that Xcode/xcodebuild would get if launched via these tools. The XCTest protocols themselves are really quite good and should be more than sufficient for making a custom reporter. And for Simulators, XCTestBootstrap enables you to run against multiple booted Simulators on the same host.

[orta]

Awesome - thanks @lawrencelomax 👍