Crash when used in test target with asynchronous tests

[mluisbrown]

As mentioned here. We have been trying to introduce the library to our project but are getting crashes in test targets which have asynchronous tests.

For example:

• Using this library in a test target with tests that use Nimble's toEventually() async testing they crash in Nimble's code to manage the runloop
• With those tests disabled, but with other async tests, the test suite completes, but when the app is shutting down after testing (in UIApplicationMain) we get a crash trying which appears to be in a call to objc_msgSend.

I enabled Zombie object diagnostics and in both of the above situations just before the crash I see this in the console:

*** -[CALayer setNeedsLayout]: message sent to deallocated instance 0x600000582da0

I'm sorry I can't provide more useful info right now. I will see if I can create a small project which reproduces the issue.

[stephencelis]

Dang, that's super frustrating 😕

A repro would be great. We haven't encountered this yet, but if we can narrow it down to a bug we can fix or file with Apple, that'd be great 😄

[mluisbrown]

Ok, I have finally managed to narrow down the cause to a single test function and have created a sample project which will repro the issue every time: https://github.com/mluisbrown/SnapshotBug

The README for the repo explains everything, but I'll copy some of it here so the information is all in one place:

When you run the tests (⌘U) they run fine but end in a crash in UIApplicationMain:

Thread 1: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)

I have enabled Zombie Objects diagnostics for the test target and with that enabled you also get an error in the console similar to:

*** -[CALayer setNeedsLayout]: message sent to deallocated instance 0x60000327d100

You can make the bug happen earlier (before the tests end) if you uncomment the autoreleasepool block in SnapshotBugTests.swift

This will cause a slightly different Zombie error to appear in the console:

*** -[UIWindowLayer isHidden]: message sent to deallocated instance 0x600000d4cd60

This bug is somehow related with the async nature of the test and the fact that SnapshotTesting also uses XCWaiter. This kind of test works fine when using ios-snapshot-test-case which is not async.

Somehow the combination of the two async code paths is probably causing the run loop to release objects which otherwise would not be released.

The root cause of the crashes is async tests similar to the once in the sample repo. Other types of async tests, eg with Nimble's toEventually don't cause the problem, but they can surface the crash earlier in the test suite.

[mluisbrown]

Just to clarify, if you do your async testing using .asyncPullback this issue does not exist. It's an issue if you use SnapshotTesting together with your own async code.

So, I wouldn't categorize it as a bug necessarily, but it's something users should be aware of.

[daehn]

I'm experiencing the same issue very frequently now and was able to catch it using the address sanitizer SUMMARY: AddressSanitizer: heap-use-after-free (libclang_rt.asan_iossim_dynamic.dylib:x86_64+0x1a547) in wrap_memcpy.

Please let me know if there's more information that I can provide to help to isolate and fix this issue.

I also noticed it happens in a view controller snapshot test in which the view controller displays a MKMapView, does not happen in any other of our snapshot tests.

[stephencelis]

I'm going to convert this to a discussion, as @mluisbrown notes, it's something folks may want to be aware of. If there are future related issues found that are due to bugs in SnapshotTesting that we can fix, though, please do open an issue!