Cedar spec bundles don't execute in Xcode 7 Beta

[alexbasson]

The ios-school project uses a spec bundle to run tests. This works fine in Xcode 6.3. In Xcode 7 Beta, the project builds, but no tests are run. The console output is simply Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds.

[tjarratt]

Interesting, I wouldn't have thought that test bundles would have regressed in the beta. If anything, I would have anticipated that spec suites would have stopped working.

Were you able to find out anything else? Were there any exceptions raised and recovered from when you ran the tests?

[akitchen]

It is probably the result of the test case discovery mechanism changing. I don't assume it is an Xcode regression.

Suites probably still work fine, though I haven't had a chance to test it yet. This may reignite the conversation about inheriting directly from XCTestCase and friends.

On Tue, Jun 9, 2015 at 7:34 AM, Tim Jarratt notifications@github.com
wrote:

Interesting, I wouldn't have thought that test bundles would have regressed in the beta. If anything, I would have anticipated that spec suites would have stopped working.

Were you able to find out anything else? Were there any exceptions raised and recovered from when you ran the tests?

Reply to this email directly or view it on GitHub:
#333 (comment)

[tooluser]

There are some other lovely test output functionalities in Xcode 7.

On Jun 9, 2015, at 07:56, Andrew Kitchen notifications@github.com wrote:

It is probably the result of the test case discovery mechanism changing. I don't assume it is an Xcode regression.

Suites probably still work fine, though I haven't had a chance to test it yet. This may reignite the conversation about inheriting directly from XCTestCase and friends.

On Tue, Jun 9, 2015 at 7:34 AM, Tim Jarratt notifications@github.com
wrote:

Interesting, I wouldn't have thought that test bundles would have regressed in the beta. If anything, I would have anticipated that spec suites would have stopped working.

Were you able to find out anything else? Were there any exceptions raised and recovered from when you ran the tests?

Reply to this email directly or view it on GitHub:
#333 (comment)
—
Reply to this email directly or view it on GitHub #333 (comment).

[codeman9]

I can confirm that test bundles don't run out of the box and that the xctest discovery process is different. Right now Cedar checks for an argument that is either -XCTest or -XCTestScopeFile. These arguments are no longer passed when running the test bundle. I have changed the discovery method to a different approach that does detect xctest runs, but the bundle still doesn't run any of the tests. It seems like the swizzle of defaultTestSuite is not actually getting called by the xctest system.

[alexbasson]

I just had an opportunity to speak with one of the Apple engineers responsible for testing framework development. After looking at my code, he suspects that whatever Cedar is doing to bootstrap tests doesn't depend on publicly available APIs and so is vulnerable to this kind of problem with each new Xcode release.

He pointed me to some new APIs available in Xcode 7, specifically the XCTestObservation protocol which declares methods such as -testBundleWillStart and -testSuiteWillStart, providing hooks that Cedar could potentially use in its bootstrapping stage.

[akitchen]

Outstanding. Thanks for following up!

On Wed, Jun 10, 2015 at 4:03 PM, Alex Basson notifications@github.com
wrote:

I just had an opportunity to speak with one of the Apple engineers responsible for testing framework development. After looking at my code, he suspects that whatever Cedar is doing to bootstrap tests doesn't depend on publicly available APIs and so is vulnerable to this kind of problem with each new Xcode release.

He pointed me to some new APIs available in Xcode 7, specifically the XCTestObservation protocol which declares methods such as -testBundleWillStart and -testSuiteWillStart, providing hooks that Cedar could potentially use in its bootstrapping stage.

Reply to this email directly or view it on GitHub:
#333 (comment)

[mjstallard]

Hey all,

We discovered a few more issues when trying out Xcode 7 on our current project which uses test bundles.

• we're no longer seeing a summary of test failures at the end of a run - perhaps there's an issue with symbolication?
• focussing specs does not appear to work (although marking specs as pending does work)
• rivets do not appear next to specs in the editor window
• executing an individual spec by tapping on its rivet in the Test Navigator runs the entire test suite
• calling addObserver: on a Cedar fake raises an exception for us. It looks we end up calling through to NSObject's actual implementation. This worked fine for us in XCode 6.3. Note that the production implementation of the fake objects overrode -addObserver: to something entirely related to KVO. Renaming them to addBeholder: fixed those failures.

Hope this helps!

[tjarratt]

I was investigating the symbolication failures with @elanakoren yesterday and we discovered that it DOES work. Cedar reaches into the app bundle for Xcode to run atos, when we changed this to refer to the Xcode 7 beta version of atos, the specs for symbolication passed.

[akitchen]

I'm not sure this is the same issue. Is it?

On Tue, Jul 21, 2015 at 10:03 AM, Tim Jarratt notifications@github.com
wrote:

I was investigating the symbolication failures with @elanakoren
https://github.com/elanakoren yesterday and we discovered that it DOES
work. Cedar reaches into the app bundle for Xcode
https://github.com/pivotal/cedar/blob/master/Source/CDRSymbolicator.m#L211
to run atos, when we changed this to refer to the Xcode 7 beta version of
atos, the specs for symbolication passed.

—
Reply to this email directly or view it on GitHub
#333 (comment).

[tjarratt]

Well the symbolication tests start passing when we change @"/Applications/Xcode.app/Contents/Developer/usr/bin/atos"

to

@"/Applications/Xcode-beta.app/Contents/Developer/usr/bin/atos"

...so it very much feels to be the same issue to me.

Granted, I know very little about symbolication and why atos might be segfaulting when the tests are run from Xcode 7.

[akitchen]

Yes Tim, you have found the issue with Cedar’s own symbolication tests, however I don’t think this is related to the headline of this email thread.  Please correct me if I’m wrong.

On Tue, Jul 21, 2015 at 2:26 PM, Tim Jarratt notifications@github.com
wrote:

Well the symbolication tests start passing when we change @"/Applications/Xcode.app/Contents/Developer/usr/bin/atos"
to
@"/Applications/Xcode-beta.app/Contents/Developer/usr/bin/atos"
...so it very much feels to be the same issue to me.

Granted, I know very little about symbolication and why atos might be segfaulting when the tests are run from Xcode 7.

Reply to this email directly or view it on GitHub:
#333 (comment)

[tjarratt]

Oh, I see. I was responding to some of the additional issues that @mjstallard found related to Xcode 7. Perhaps those should be logged as a separate issue. Sorry for the noise!

[joemasilotti]

Are there any updates on this? I haven't seen activity on the Xcode7 branch but wanted to double check.

[tjarratt]

There's been some slow, steady progress on Xcode 7 support for test bundles @joemasilotti. In the past few weeks we've fixed symbolication and discovered the root cause of why test bundles no longer have reporter output in the console after specs finish running.

To the best of my knowledge, running your specs in Xcode 7 beta 5 with Cedar in a test bundle should work today, using the Xcode7 branch. The only remaining issue to fix that I'm aware of is the missing reporter output.

[tjarratt]

Have you had any other, unrelated issues with Xcode 7 and test bundles lately, @joemasilotti?

[joemasilotti]

Not a single run of my test suite has completed on Xcode 7. Of the fifteen or so times I've attempted to do so, I receive EXC_BAD_ACCESS or EXC_BREAKPOINT exceptions.

These seem to be occurring on or around view lifecycle methods and when instantiating WebKit objects. Not sure if that's a red herring, though.

If it helps, I'm running ~1600 tests but the suite fails before getting to even 100 of them.

[tjarratt]

Could you provide a sanitized backtrace? It's not clear to me if the exception is being caused by Cedar or maybe something else from your description. If the backtrace is pretty bare, you might try enabling NSZombies and see if there's anything there.

I also saw something similar helping a pivot upgrade from Xcode 6 to 7, but the error seemingly went away when we cleaned the build folder. While I hate cargo-culting that "solution", that's also probably a relevant thing to try.

[joemasilotti]

Well this might all be moot now. Running my suite against Beta 6 raises EXC_BAD_ACCESS in main.m. Not much of a stack trace to share, but I'm interested in hearing if anyone else is experiencing this.

I know that a bunch of XCTest stuff changed with this release, specifically XCTestObservationCenter and XCTestProbe.

[tjarratt]

Absolutely right, Joe. The Xcode 7 branch uses XCTestObservationCenter to hook into XCTest's runtime in a somewhat saner way than we had before in the past.

So far, I know this runs Cedar's specs, some specs for my own projects, the iOS school specs, and a few other SF pivots' projects as well, but I wouldn't be too surprised if there weren't more lingering issues to resolve.

[joemasilotti]

That's reassuring. Has anyone had a chance to try this with Beta 6? I understand that that's a huge ask as it was released only yesterday.

[akitchen]

I'm pretty sure the WebKit stuff has internal async going on. I would consider it a red herring...

On Tue, Aug 25, 2015 at 5:04 PM, Joe Masilotti notifications@github.com
wrote:

That's reassuring. Has anyone had a chance to try this with Beta 6? I understand that that's a huge ask as it was released only yesterday.

Reply to this email directly or view it on GitHub:
#333 (comment)

[joemasilotti]

@akitchen meaning I have other issues to fix or Cedar is responsible for the crash? The crash only occurs when testing the app with Xcode 7, not when running the app in Xcode 6 or 7, nor when testing the app with Xcode 6.

[akitchen]

I would be very surprised if cedar is causing your EXC_BAD_ACCESS crash.

Cedar runs your specs synchronously on the main queue. A crash like this is typically due to side effects of something your code under test is doing. Do any tests run before this crash happens? Can you binary search to isolate the test or group of tests which are involved?

It would also be interesting to know if you are faking or spying instances of certain framework classes. This is really the only case where the crash could be related to cedar. Or do you think it is happening right at startup?

On Tue, Sep 1, 2015 at 12:26 PM, Joe Masilotti notifications@github.com
wrote:

@akitchen meaning I have other issues to fix or Cedar is responsible for the crash? The crash only occurs when testing the app with Xcode 7, not when running the app in Xcode 6 or 7, nor when testing the app with Xcode 6.

Reply to this email directly or view it on GitHub:
#333 (comment)

[joemasilotti]

Me too, which is why this issue has me so confused.

Any test that tries to instantiate a WKWebView crashes on -init. Simple as that. I do spy on the web view, but not until later in the test (code that is never reached). I removed all other tests except this simple example yet it still fails.

If it helps, I've attached a screenshot of the stack and exception.

[akitchen]

You might remember that we tend to fully stub out the UIWebView in testing
scenarios, largely to avoid stuff like this but also to keep the tests fast
and synchronous. WKWebView probably wants similar treatment.

Could you share a simple test as a gist which demonstrates this?

On Tue, Sep 1, 2015 at 1:30 PM, Joe Masilotti notifications@github.com
wrote:

Me too, which is why this issue has me so confused.

Any test that tries to instantiate a WKWebView crashes on -init. Simple
as that. I do spy on the web view, but not until later in the test (code
that is never reached). I removed all other tests except this simple
example yet it still fails.

If it helps, I've attached a screenshot of the stack and exception.

[image: exception]
https://cloud.githubusercontent.com/assets/2092156/9615528/b6b34c8c-50c6-11e5-80a0-191110cde7e8.png

—
Reply to this email directly or view it on GitHub
#333 (comment).

[joemasilotti]

Yea, that's a good point. I should probably be doing that anyway. Thanks for the insight.

I'm having trouble reproducing this in a fresh project, which makes me think there is something else I'm not thinking about. I'll post something when I can get it to reliably fail. Thanks for the help.

[joemasilotti]

Finally nailed down the issue. WKUserContentController has a "write-only" property for adding script handlers. Very similar to how UIAlertController has "write-only" actions. I added a spec helper to swizzle -addScriptMessageHandler:, add the name and message it to a dictionary, and then expose the blocks. Turns out this royally screws up WKWebView and related WebKit classes when testing under Xcode 7 on iOS 9.

That being said it doesn't sound like it's an issue related to Cedar. Thanks for walking me through the debugging process though, it was a huge help!

I'm going to open an issue in PCK to see if there are any better ways to accomplish what I am trying to do.

[akitchen]

Thanks for following up with more info @joemasilotti

Where does this leave us on this particular issue then?

[joemasilotti]

It's not related to Cedar at all. I am able to reproduce the issue with a plain old XCTest example. :-)

[tjarratt]

Thanks for all the feedback and testing everyone. Now that this has been merged into master, I'm more comfortable closing this.

Please open new issues for any new regressions that you may find while writing happy little Cedar specs.