Question: Does bluepill support running Xcode UI testing? #16

dyang · 2017-01-19T21:09:43Z

I managed to run bluepill against one of my apps with a bunch of UI tests based on Xcode UI testing. However I'm getting the following error. A bit googling suggests that this might be due to UI testing having different test template/config than "regular" XCTests. Is Xcode UI testing supported by bluepill? :)

Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'No target application path specified via test configuration: <XCTestConfiguration: 0x60800011a9d0>

Thanks!

The text was updated successfully, but these errors were encountered:

oliverhu · 2017-01-19T21:30:14Z

hey @dyang , we haven't actually tested this with Xcode UI testing. Our UI tests are all KIF tests.

We collect the test target by searching SampleApp.app/Plugin/*.xctest. You can test by adding a additional-xctests flag in the config file to the UI testing target and try from there.

It would be great if you can tweak the code base a bit and let us know the result.

krze · 2017-01-19T22:18:31Z

When you compile an XCUITest there are no .xctest files generated in the target app's plugin folder. Instead it's placed inside the runner, named TEST_TARGET_NAME-runner.app/Plugin/TEST_TARGET_NAME.xctest.

I gave what you said a whirl and all 5 sims exited with the same error:

[  INFO  ] Started Simulator 5 (PID 42860).
bp: invalid configuration
	iOS 10.1 is not a valid runtime.

When checking valid runtimes:

== Runtimes ==
iOS 9.3 (9.3 - 13E233) (com.apple.CoreSimulator.SimRuntime.iOS-9-3)
iOS 10.2 (10.2 - 14C89) (com.apple.CoreSimulator.SimRuntime.iOS-10-2)
tvOS 10.1 (10.1 - 14U591) (com.apple.CoreSimulator.SimRuntime.tvOS-10-1)
watchOS 3.1 (3.1 - 14S471a) (com.apple.CoreSimulator.SimRuntime.watchOS-3-1)

I'll install a 10.1 sim and give it another shot

oliverhu · 2017-01-19T22:23:43Z

@krze that's another issue mentioned in #8 , you need to download iOS 10.1 simulator Or alternatively, search for BP_DEFAULT_RUNTIME and change the value to 10.2

krze · 2017-01-19T22:51:37Z

Looks like the target application isn't being pulled from the XCUITest .xctest file:

2017-01-19 17:47:04.802 ApplicationNameX[44527:3337770] *** Assertion failure in -[XCUIApplication init], /Library/Caches/com.apple.xbs/Sources/XCTest_Sim/XCTest-11753/XCTestFramework/UI Testing/XCUIApplication.m:72
/Users/krze/develop/ApplicationName/UITests/TestHelpers.swift:0: error: -[UITests.AppRatingPromptTests testShowsAfterTestSetup] : failed: caught "NSInternalInconsistencyException", "No target application path specified via test configuration: &lt;XCTestConfiguration: 0x60000011df40&gt;
	                  testBundleURL:file:///Users/krze/develop/ApplicationName/TestBuild/Build/Products/UITest-iphonesimulator/UITests-Runner.app/PlugIns/UITests.xctest/
	         testBundleRelativePath:(null)
	              productModuleName:UITests
	                    testsToSkip:
	                     testsToRun:(null)
	             reportResultsToIDE:no
	              sessionIdentifier:18C96A25-3055-407C-8CEF-EFDAA68F8BFC
	     pathToXcodeReportingSocket:(null)
	      disablePerformanceMetrics:no
	treatMissingBaselinesAsFailures:no
	                baselineFileURL:(null)
	       baselineFileRelativePath:(null)
	          targetApplicationPath:(null)
	      targetApplicationBundleID:(null)
	               reportActivities:no
	       testsMustRunOnMainThread:no
	         initializeForUITesting:no
	 aggregateStatisticsBeforeCrash:(null)
"
(
	0   CoreFoundation                      0x000000011ab7834b __exceptionPreprocess + 171
	1   libobjc.A.dylib                     0x000000011a5e021e objc_exception_throw + 48
	2   CoreFoundation                      0x000000011ab7c442 +[NSException raise:format:arguments:] + 98
	3   Foundation                          0x00000001146dde4d -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 195
	4   XCTest                              0x000000012ba022a6 -[XCUIApplication init] + 195
	5   UITests                             0x000000012c409f1f _TTSf4n_g_n___TFE7UITestsCSo10XCTestCase11setUpTumblrfT8loggedInSb8withUserVS_11TumblrCreds_T_ + 207
	6   UITests                             0x000000012c3d23e0 _TFC7UITests20AppRatingPromptTests5setUpfT_T_ + 640
	7   UITests                             0x000000012c3d2726 _TToFC7UITests20AppRatingPromptTests5setUpfT_T_ + 22
	8   XCTest                              0x000000012b9e0e43 __24-[XCTestCase invokeTest]_block_invoke_2 + 173
	9   XCTest                              0x000000012ba197df -[XCTestContext performInScope:] + 190
	10  XCTest                              0x000000012b9e0d83 -[XCTestCase invokeTest] + 255
	11  XCTest                              0x000000012b9e159c -[XCTestCase performTest:] + 457
	12  XCTest                              0x000000012b9de664 -[XCTestSuite performTest:] + 491
	13  XCTest                              0x000000012b9de664 -[XCTestSuite performTest:] + 491
	14  XCTest                              0x000000012b9de664 -[XCTestSuite performTest:] + 491
	15  XCTest                              0x000000012b9ebd2b -[XCTestObservationCenter _observeTestExecutionForBlock:] + 602
	16  XCTest                              0x000000012ba1af0d _XCTestMain + 1050
	17  CoreFoundation                      0x000000011ab1d25c __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
	18  CoreFoundation                      0x000000011ab02304 __CFRunLoopDoBlocks + 356
	19  CoreFoundation                      0x000000011ab01a75 __CFRunLoopRun + 901
	20  CoreFoundation                      0x000000011ab01494 CFRunLoopRunSpecific + 420
	21  GraphicsServices                    0x000000011c739a6f GSEventRunModal + 161
	22  UIKit                               0x000000011877a964 UIApplicationMain + 159
	23  ApplicationName                     0x000000010effccdf main + 111
	24  libdyld.dylib                       0x000000011868e68d start + 1
)

oliverhu · 2017-01-19T23:42:56Z

Hmm, this should be doable, we just need to fill some extra fields in the XCTestConfiguration file.

Check XCTestConfiguration.h and SimulatorHelper.m

In SimulatorHelper.m:
Add [xctConfig setTargetApplicationPath:PATH_TO_APP]; to + (NSString *)testEnvironmentWithConfiguration:(BPConfiguration *)config; should do the trick.

Let me know if you that works :D

casshill · 2017-01-20T03:55:38Z

I tried the above solution and also added the target app bundle identifier like this
[xctConfig setTargetApplicationBundleID:BUNDLE_ID];

But I still can't get my tests to run. I don't think the test application is launching properly
<testcase classname="..." name="test..." time="61.096000"> <failure type="Failure" message="-[...Tests test...] : UI Testing Failure - Unable to update application state promptly.">

plu · 2017-01-20T04:00:51Z

To make the UI Tests work, you need to set a few properties on XCTestConfiguration:

testsMustRunOnMainThread
initializeForUITesting
targetApplicationPath
targetApplicationBundleID

You can peek a bit here how Facebook is doing it: https://github.com/facebook/FBSimulatorControl/blob/master/XCTestBootstrap/Bundles/FBTestConfiguration.m#L85-L95

oliverhu · 2017-01-20T06:29:04Z

@plu love it, thanks!

oliverhu · 2017-01-20T22:31:37Z

@plu it turns out to be more complicated than I thought. I managed to set the environment same as the one Xcode is using to run UI test (arguments, environment and XCTestConfiguration)

// XCTUITest configurations
[xctConfig setPathToXcodeReportingSocket:nil];

[xctConfig setTargetApplicationPath:config.appBundlePath];
[xctConfig setTargetApplicationPath:@"PATHTO_BPSampleAppUITests-Runner.app"];
[xctConfig setTargetApplicationBundleID:@"LI.BPSampleApp"];
[xctConfig setTestsToRun:[NSSet setWithObjects:@"BPSampleAppUITests/testExample", nil]];
[xctConfig setTestsMustRunOnMainThread:YES];
[xctConfig setInitializeForUITesting:YES];
[xctConfig setTreatMissingBaselinesAsFailures:NO];
[xctConfig setReportResultsToIDE:YES];
[xctConfig setReportActivities:YES];

And get:

  Jan 20 01:34:23 khu-mn1 CoreSimulatorBridge[15027]: Launch successful for 'com.apple.test.BPSampleAppUITests-Runner'
  Jan 20 01:34:23 khu-mn1 assertiond[15022]: Could not create or rename power assertion for <BKNewProcess: 0x7ff009c10a90; com.apple.mobilecal; pid: 15102; hostpid: -1>: 0xe00002c9
  Jan 20 01:34:23 khu-mn1 logd[15012]: metadata shared cached uuid is null (using logd's shared cache info) XCTRunner (15103)
  Jan 20 01:34:23 khu-mn1 logd[15012]: Failed to harvest strings for pathless uuid '00000000-0000-0000-0000-000000000000'
  Jan 20 01:34:23 khu-mn1 assertiond[15022]: Could not create or rename power assertion for <BKNewProcess: 0x7ff009c10a90; com.apple.mobilecal; pid: 15102; hostpid: -1>: 0xe00002c9
  Jan 20 01:34:23 khu-mn1 XCTRunner[15103]: assertion failed: 15G1217 14C89: libxpc.dylib + 62677 [0339A402-6531-30E9-8BC2-589C463CD6D6]: 0x7d
  Jan 20 01:34:23 khu-mn1 watchlistd[15090]: Now playing app did change to '(null)' (playing: 0) from '(null)'
  Jan 20 01:34:23 khu-mn1 watchlistd[15090]: WLKPlaybackSummary - Parameter failed validation bundleID. It is nil
  Jan 20 01:34:23 khu-mn1 XCTRunner[15103]: Running tests...
  Jan 20 01:34:23 khu-mn1 XCTRunner[15103]: a
  Jan 20 01:34:23 khu-mn1 XCTRunner[15103]: c  // Before the launching app statement

And it got stuck at the launching app statement. Searched some old threads from Xctool, there is extra testmanagerdr to deal with to run XCTUITest?

Does FBSimualatorControl support running XCTUITest?

plu · 2017-01-20T23:28:59Z

Does FBSimualatorControl support running XCTUITest

Yes, it's working there.

oliverhu · 2017-02-16T08:00:30Z

Some good news here. Finally got this working with the help in FBSimulatorControl. Will submit a PR for this.

krisdyZJU · 2017-02-17T03:09:46Z

@oliverhu when will this pr merged into the latest code to support xcode ui testing?

oliverhu · 2017-02-24T07:07:32Z

it is contingent on #55 , hopefully sometime next week :D

oliverhu · 2017-02-28T05:13:05Z

Update: the support is available in #68, please check it out. The branch works locally but can't pass CircleCI, still trying to figure out why 😕

djrenfro · 2017-03-01T18:16:03Z

@oliverhu I ran your branch locally and ran into two issues running UI tests. In BPPacker.packTests(), I was getting a exception any on xctFile any time it tried to access part of it. While debugging, looks like the file type for some reason is an Array so I fixed it locally by changing the data type from BPXCTestFile to NSArray and and used xctFile.firstObject. Not sure if there is something different that should be done. The other issue is a race condition when trying to run more than 1 UI test in parallel. The same test runs across all the simulators. If I put a debug point in BPRunner.run() after a simulator is started, specifically

[BPUtils printInfo:INFO withString:@"Started Simulator %lu (PID %d).", taskNumber, [task processIdentifier]];

and wait a few seconds to continue the program, then it will run different tests on the different simulators, without the breakpoint it runs the same test. Let me know if you need more specifics.

oliverhu · 2017-03-01T20:30:56Z

@djrenfro it would be great if you could get a PR based on that branch to fix the first issue (and a test case to reproduce the second issue)

djrenfro · 2017-03-01T22:14:56Z

@oliverhu PR to fix the first issue in #82
For the 2nd issue, I created a branch on my fork with a second UI test added to the UI test class - djrenfro@ce31797. If you put a debug point at the location mentioned in my previous comment and wait between 8-10 seconds after the first simulator is launched to continue, you'll see the tests run in parallel. Without the debug point, the second test will run on both sims and the first will never run. Let me know if you need more specifics.

oliverhu · 2017-03-01T23:18:58Z

@djrenfro Good catch! Merged #82. I'll take a look at the second issue tomorrow and this Friday.

oliverhu · 2017-03-06T08:21:47Z

@djrenfro found the problem - NSTemporaryDirectory() returns the same temporary path under different Bluepill-ci processes :( And in my branch, I added a constant session identifier for all blupill-cli. Pushed a fix, should work now.

djrenfro · 2017-03-07T00:14:23Z

@oliverhu That fixed the issue I was having, working well now

oliverhu · 2017-03-07T00:28:10Z

Cool. Also for the CI issue, it is only reproducible with CircleCI but not with Travis, we are considering moving back to Travis and merge this branch in.

oliverhu · 2017-03-10T06:56:25Z

The PR is merged, please try it out and report issues :)

jimmyeisenhauer · 2017-03-12T21:18:35Z

i tried running latest master with ui test bundle and just got
ERROR: (null) as output. Still working on debugging further.

nickmshelley · 2017-03-12T22:07:58Z

@jimmyeisenhauer Try looking at #93

jimmyeisenhauer · 2017-03-13T05:45:03Z

making the change in #93 got it working thanks! Has anyone else tried the -i option with UI tests? I just get the error {4364} 20170312.224328 [ ERROR ] Packing failed: (null)

jimmyeisenhauer · 2017-03-13T14:37:09Z

it also looks like it will not run with ONLY a UI test bundle - 2017-03-13 07:24:31.584 bluepill[9250:15794876] There is .xctest file under /Users/mos6y/Documents/workspace/sour-mash/./hack/Build/Products/Debug-iphonesimulator/sour-mash.app/Plugins ERROR: The folder “Plugins” doesn’t exist.

oliverhu · 2017-03-13T17:50:11Z

@jimmyeisenhauer could you open an issue for that?

jimmyeisenhauer · 2017-03-13T19:38:43Z

submitted #98 #99 for the two above issues I posted. thanks!

fabiomassimo · 2017-03-15T21:15:25Z

I'm using the latest and greatest from master branch but I'm not able to run the Xcod UI Tests in parallel. They are all launched on one simulator only while my unit tests are properly parallel-ized.

Is it a well known issue or this should have been fixed on master?

oliverhu · 2017-03-16T03:33:27Z

@fabiomassimo is the number of UI tests >> the number of unit tests?

In current setup, if say you have 100 unit tests and 25 UI tests, and you have 5 simulators to run all of them - we will do the math as (100 + 25) / 5 = 25, so each simulator will take 25 tests. As a result, all UI tests will run in the simulator.

fabiomassimo · 2017-03-16T09:45:28Z

Thank you for explaining me that! It makes sense now.

What if I want to run only UI Tests then? In that case the algorithm should parallelise the UI test cases.

I started to look into this too but I wonder if the --exclude option is the right way to go.
The reason I'm asking is that I find it quite hard to apply to exclude all unit test cases because, as far as I understood, I have to manually specify all test cases and add them along as I add them.

nickmshelley · 2017-03-16T15:52:00Z

@fabiomassimo I think you want the -t option that allows you to pass in a single .xctest bundle.

DNCGraef · 2017-03-31T08:41:06Z

I can see why you've split the tests that way... but in a project where unit tests outweigh UI tests (trending towards optimal testing pyramid https://martinfowler.com/bliki/TestPyramid.html), little time-saving gains will be achieved.

It's more or less guaranteed that the length of any given UI test will exceed that of a given unit test, likely by order(s) of magnitude.

Would it be possible to treat the unit and UI test suites as separate entities? For example,

If I have 100 unit tests and 20 UI tests
And run blue pill with 4 simulators
Unit tests are split into 100/4 = 25
UI tests are split into 20/4 = 5
Unit tests are run first across 4 Sims
As soon as one sim finishes unit tests, UI tests begin on that sim

oliverhu · 2017-03-31T15:27:38Z

@DNCGraef ;-) Read the read.me carefully, you'll find a flag called no-split, simply put your unit test target there.

DNCGraef · 2017-03-31T17:19:46Z

Nice tip - I'll try that :D

Update Checkstyle regex to correctly parse line & char values from output

oliverhu mentioned this issue Jan 19, 2017

No useful error messages when packing fails #17

Closed

oliverhu added the enhancement label Jan 19, 2017

oliverhu added this to the XCTUITest support milestone Jan 19, 2017

oliverhu self-assigned this Jan 20, 2017

lochness42 mentioned this issue Jan 20, 2017

Unable to run my UI tests #21

Closed

oliverhu added the help wanted label Jan 20, 2017

oliverhu added pr coming and removed help wanted labels Feb 16, 2017

djrenfro mentioned this issue Mar 1, 2017

Fix UI test bundles being and array of array instead of array of xctFile #82

Merged

oliverhu closed this as completed Mar 10, 2017

oliverhu mentioned this issue Mar 30, 2017

On a mixed unit/UI test run, unit tests execute on multiple simulators, but UI tests do not #117

Closed

RainNapper pushed a commit to RainNapper/bluepill that referenced this issue Jun 16, 2021

Merge pull request MobileNativeFoundation#16 from eugeneduetto/master

c4e0360

Update Checkstyle regex to correctly parse line & char values from output

Question: Does bluepill support running Xcode UI testing? #16

Question: Does bluepill support running Xcode UI testing? #16

Comments

dyang commented Jan 19, 2017 • edited

oliverhu commented Jan 19, 2017

krze commented Jan 19, 2017 • edited

oliverhu commented Jan 19, 2017

krze commented Jan 19, 2017

oliverhu commented Jan 19, 2017

casshill commented Jan 20, 2017

plu commented Jan 20, 2017

oliverhu commented Jan 20, 2017

oliverhu commented Jan 20, 2017 • edited

plu commented Jan 20, 2017

oliverhu commented Feb 16, 2017

krisdyZJU commented Feb 17, 2017

oliverhu commented Feb 24, 2017

oliverhu commented Feb 28, 2017

djrenfro commented Mar 1, 2017

oliverhu commented Mar 1, 2017

djrenfro commented Mar 1, 2017

oliverhu commented Mar 1, 2017

oliverhu commented Mar 6, 2017

djrenfro commented Mar 7, 2017

oliverhu commented Mar 7, 2017

oliverhu commented Mar 10, 2017

jimmyeisenhauer commented Mar 12, 2017

nickmshelley commented Mar 12, 2017

jimmyeisenhauer commented Mar 13, 2017

jimmyeisenhauer commented Mar 13, 2017

oliverhu commented Mar 13, 2017

jimmyeisenhauer commented Mar 13, 2017

fabiomassimo commented Mar 15, 2017

oliverhu commented Mar 16, 2017

fabiomassimo commented Mar 16, 2017

nickmshelley commented Mar 16, 2017

DNCGraef commented Mar 31, 2017

oliverhu commented Mar 31, 2017

DNCGraef commented Mar 31, 2017

dyang commented Jan 19, 2017 •

edited

krze commented Jan 19, 2017 •

edited

oliverhu commented Jan 20, 2017 •

edited