Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Command Line Tests Fail -- Xcode 4.5 #96

Closed
adamhowardprice opened this Issue · 52 comments
@adamhowardprice

I have followed all the directions to run my test targets from the command line, and it is still failing. In my internet searching, I haven't found anyone else with my particular error message:

GHUNIT_CLI=1 GHUNIT_AUTORUN=1 xcodebuild -target UnitTests -configuration Debug -sdk iphonesimulator build
Build settings from command line:
    SDKROOT = iphonesimulator6.0

=== BUILD NATIVE TARGET UnitTests OF PROJECT [XXXX] WITH CONFIGURATION Debug ===
Check dependencies

PhaseScriptExecution "Run GHUnit UnitTests From Command Line" build/[XXXX].build/Debug-iphonesimulator/UnitTests.build/Script-CFFB208B15F7DA8F008C898F.sh
    cd /Users/adamhowardprice/Documents/iOS/Projects/[XXXX]
    /bin/sh -c /Users/adamhowardprice/Documents/iOS/Projects/[XXXX]/build/[XXXX].build/Debug-iphonesimulator/UnitTests.build/Script-CFFB208B15F7DA8F008C898F.sh
Running: "/Users/adamhowardprice/Documents/iOS/Projects/[XXXX]/build/Debug-iphonesimulator/UnitTests.app/UnitTests" -RegisterForSystemEvents
2012-09-05 16:04:25.908 UnitTests[18660:707] Unknown Device Type. Using UIUserInterfaceIdiomPhone based on screen size
Terminating since there is no workspace.

** BUILD SUCCEEDED **

I have also tried running command line tests in the MyTestable-iOS test project and it also fails, with this console output:

PhaseScriptExecution RunTests.sh build/MyTestable.build/Debug-iphonesimulator/Tests.build/Script-007F25EE13D283960023DEA7.sh
    cd /Users/adamhowardprice/Documents/iOS/Projects/Sharely/Sharely/Libraries/GHUnit/Examples/MyTestable-iOS
    /bin/sh -c /Users/adamhowardprice/Documents/iOS/Projects/Sharely/Sharely/Libraries/GHUnit/Examples/MyTestable-iOS/build/MyTestable.build/Debug-iphonesimulator/Tests.build/Script-007F25EE13D283960023DEA7.sh
Running: "/Users/adamhowardprice/Documents/iOS/Projects/Sharely/Sharely/Libraries/GHUnit/Examples/MyTestable-iOS/build/Debug-iphonesimulator/Tests.app/Tests" -RegisterForSystemEvents
2012-09-05 16:23:48.579 Tests[22005:707] Warning: CFFIXED_USER_HOME is not set!  It should be set to the simulated home directory.
2012-09-05 16:23:48.610 Tests[22005:707] Unknown Device Type. Using UIUserInterfaceIdiomPhone based on screen size
Terminating since there is no workspace.

Is there any reason why I would require an XCode workspace here? I have made sure to close the iPhone Simulator beforehand as well. I'm not using any ViewTests. Any forks that I should look at that have resolved this problem?

@gabriel
Collaborator

It looks like you are using the xcode beta. If you want to use the standard xcodebuild you can use xcode-select to switch back:

sudo xcode-select -switch /Applications/Xcode.app

If you want to work with the beta, it appears it wants to use a workspace.
You can try to change to use a workspace?

GHUNIT_CLI=1 xcodebuild -workspace ___ -scheme ___

Also be careful not to have multiple Tests schemes or targets?

@adamhowardprice

Aha! That was it -- thank you very much! Am loving using GHUnit btw.

@kdunsmore

Hmmm... I'm getting exactly the same thing. Tried using the workspace but I get the same output:

GHUNIT_CLI=1 xcodebuild -workspace VoiceRecognizer.xcworkspace -scheme VoiceRecognizerTests -configuration Debug -sdk iphonesimulator build

=== BUILD NATIVE TARGET VoiceRecognizerTests OF PROJECT VoiceRecognizer WITH CONFIGURATION Debug ===
Check dependencies

PhaseScriptExecution "Run Script" /Users/kevan/Library/Developer/Xcode/DerivedData/VoiceRecognizer-gjqdkayvfemoducfsjlokmazgxgk/Build/Intermediates/VoiceRecognizer.build/Debug-iphonesimulator/VoiceRecognizerTests.build/Script-0E64583716001423002CE6CB.sh
    cd /Users/kevan/github/ios-sdks/VoiceRecognizer
    /bin/sh -c /Users/kevan/Library/Developer/Xcode/DerivedData/VoiceRecognizer-gjqdkayvfemoducfsjlokmazgxgk/Build/Intermediates/VoiceRecognizer.build/Debug-iphonesimulator/VoiceRecognizerTests.build/Script-0E64583716001423002CE6CB.sh
Running: "/Users/kevan/Library/Developer/Xcode/DerivedData/VoiceRecognizer-gjqdkayvfemoducfsjlokmazgxgk/Build/Products/Debug-iphonesimulator/VoiceRecognizerTests.app/VoiceRecognizerTests" -RegisterForSystemEvents
2012-09-12 14:07:43.300 VoiceRecognizerTests[50929:707] Warning: CFFIXED_USER_HOME is not set!  It should be set to the simulated home directory.
2012-09-12 14:07:43.307 VoiceRecognizerTests[50929:707] Unknown Device Type. Using UIUserInterfaceIdiomPhone based on screen size
Terminating since there is no workspace.

** BUILD SUCCEEDED **

Any pointers here?

@gabriel
Collaborator
@kdunsmore

Hi Gabriel -

In my case I actually want to use the beta :-) I thought based on your previous comments on this issue that it would work. Is the beta not supported by GHUnit?

@gabriel
Collaborator
@max-horvath

It's still happening on the GM. I'm actually running this on a workspace ... still it happens. When switching to SDK 5.1 on the GM, everything works fine ... but that cannot be the solution.

BTW, no solution to that came up on Apple's dev forum.

@gabriel
Collaborator

Found a radar here:
http://openradar.appspot.com/12306879

I am currently baffled.

@max-horvath

Well, I opened this radar.

So far no response.

@max-horvath

I also opened a thread in the Dev forums of Apple:
https://devforums.apple.com/thread/167012

Also no response.

@gabriel
Collaborator

Do you think the message before it.. "Unknown Device Type" has anything to do with the failure?

@SteveMunday

Since this just showed up recently it must have something to do with iPhone 5. I wonder if the default screen size when running the simulator is now Retina - 4"?

@max-horvath
@gabriel
Collaborator

I am running with new 4" default png (Default-568h@2x.png), and get the same error.

@alangria

Do we know if there is workaround for this? I can still repro this in release version of XCode 4.5

@whalec

Likewise. We're also suffering from this in XCode 4.5. Any ideas on this? Happy to take a look if someone can point us in a direction.

@alangria

As a workaround, it is possible to run the the app in simulator using third party command line tools like ios-sim and turn autorun on launch and autoexit on complete flags on.

@gabriel
Collaborator
@kevinbryant

We ran in to the same problem. We're using GHUnit alongside GTM, and got things working by using the RuniOSUnitTestsUnderSimulator.sh script from GTM. Our "Run Script" build phase now looks like this:

if [ ! -z "$BUILD_CLI" ]
then
    export GHUNIT_CLI=true
    sh Tests/UnitTesting/RuniOSUnitTestsUnderSimulator.sh
else
    sh Tools/ghruntests.sh
fi

Where ghruntests.sh is our wrapper around the standard GHUnit unit test runner script. Tests run through xcode via normal Run, via command line by setting BUILD_CLI ( export BUILD_CLI=true ), then running:

xcodebuild -target UnitTests -configuration Debug -sdk iphonesimulator build

Where UnitTests is the name of your test target. RuniOSUnitTestsUnderSimulator.sh is pretty well documented. Because it's running using xcodebuild all of the required environment variables for RuniOSUnitTestsUnderSimulator are already set.

http://code.google.com/p/google-toolbox-for-mac/source/browse/trunk/UnitTesting/RunIPhoneUnitTest.sh

@victorlima

http://stackoverflow.com/a/12655840/1494532

Apparently this can be worked around by setting the simulator version, but all other kinds of creepy XIB compilations failures popped up for me, so depending on what application tests you are running, maybe you can be fine for now. ;/

@levigroker

Can this issue be reopened? I'm having the same issue, and everything I've tried in this thread does not solve or workaround it. :/

@gabriel gabriel reopened this
@gabriel gabriel was assigned
@jkennedy1980

I got it working on the latest XCode using the iOS 5.1 simulator and this command:

GHUNIT_CLI=1 WRITE_JUNIT_XML=YES xcodebuild ARCHS=i386 ONLY_ACTIVE_ARCH=NO VALID_ARCHS=i386 -workspace WhatsOn.xcworkspace -scheme Tests -configuration Debug -sdk iphonesimulator5.1 clean build

@max-horvath
@gabriel
Collaborator

I just noticed updated command line tools. Still broken. Heh wtf apple

@markshiz

My workaround solution was to use waxsim for the moment. I replaced the contents of my RunTests.sh with the following:

#!/bin/sh 
BASEDIR=`dirname $0`

if [ "$GHUNIT_CLI" = "" ] && [ "$GHUNIT_AUTORUN" = "" ]; then
    exit 0
fi

osascript -e 'tell app "iPhone Simulator" to quit'
sleep 5
MY_TMPDIR=`/usr/bin/getconf DARWIN_USER_TEMP_DIR`
RESULTS_DIR="${MY_TMPDIR}test-results"
JUNIT_XML_DIR="${RESULTS_DIR}" GHUNIT_AUTORUN=1 GHUNIT_AUTOEXIT=1 waxsim -s 6.0 -f iphone ${BASEDIR}/../build/Debug-iphonesimulator/UnitTests.app
if [ -d "$RESULTS_DIR" ]; then
  `cp -r -v "$RESULTS_DIR" "$BUILD_DIR"` 
  `rm -rf "$RESULTS_DIR"`
fi

Unfortunately, this has an unwanted side-effect of not running inside the xcode project/workspace build as a script. Not sure on why that is, but waxism freezes the build when run this way. I had to modify my scripts to run RunTests.sh after the build had completed.

To execute and output junit xml, I run the following command:

BUILD_DIR=/path/to/build GHUNIT_CLI=1 WRITE_JUNIT_XML=1 RunTests.sh

NOTE: For those who are running jenkins for CI, you have to start jenkins as a LaunchAgent for waxsim to execute.

Hope this helps someone!

@superrunt19

this worked for me. I changed the implementation of my main.m from this

@autoreleasepool {
    return UIApplicationMain(argc, argv, nil, @"GHUnitIOSAppDelegate");
}

to this. I'm duplicating the check that GHUnitIOSAppDelegate is performing in applicationDidFinishLaunching:

@autoreleasepool {
    if (getenv("GHUNIT_CLI")) {
        retVal = [GHTestRunner run];
    } else {
        retVal = UIApplicationMain(argc, argv, nil, @"GHUnitIOSAppDelegate");
    }
}

What I found is that when running the first way, UIApplicationMain is detecting something incorrect in the environment and is bailing before it ever attempts to instantiate GHUnitIOSAppDelegate (I added an init method to GHUnitIOSAppDelegate and it was never invoked). So it appears that UIApplicationMain has changed with the iOS6 sdk.

@levigroker

@superrunt19 You are my hero. This worked perfectly for me, thank you.

To be clear, I had to modify my main.m like this:

#import <UIKit/UIKit.h>
#import <GHUnitIOS/GHUnitIOSViewController.h>

int main(int argc, char *argv[])
{
    int retVal;
    @autoreleasepool {
        if (getenv("GHUNIT_CLI")) {
            retVal = [GHTestRunner run];
        } else {
            retVal = UIApplicationMain(argc, argv, nil, @"GHUnitIOSAppDelegate");
        }
    }
    return retVal;
}
@gabriel
Collaborator
@kdunsmore

@superrunt19: I believe I now want to have your babies. Awesome!

@adriancowham

This is definitely progress!

But yeah, view tests are failing. Even simply attempting to load a controller's view no longer works, for example, the following asserts always fail when I run the tests from command line.

assertThat(testGallery.view, notNilValue() );
assertThat(testGallery.gridView, notNilValue() );

Thoughts?

@ianyh

It seems like UIApplicationMain does a bunch of UIKit initialization, actually. I get SIGTRAP trying to do

[UIFont systemFontOfSize:16.0];

and I have reason to believe

[UIDevice currentDevice];

does the same.

It looks like circumventing UIApplicationMain doesn't actually work. And it doesn't seem to actually return when Terminating since there is no workspace. gets hit. I found a solution that seems to work, which is using the one thing that does get instantiated, which is the UIApplication instance.

I created a custom subclass of UIApplication called WSApplicationTestRunner that has nothing but the init

- (id)init {
    self = [super init];
    if (getenv("GHUNIT_CLI")) {
        if ([GHTestRunner run] > 0)
            [NSException raise:NSGenericException format:@""];
    }
    return self;
}

The exception is so our Jenkins job catches it as a failure in the absence of directly controlling the return value of the process.

Then, change the UIApplicationMain call in main.m to

UIApplicationMain(argc, argv, NSStringFromClass([WSApplicationTestRunner class]), @"GHUnitIPhoneAppDelegate");
@adriancowham

@ianyh I didn't have any luck with your approach, been at it for a few days with no success. Anyone else been successful?

Here's my main.m

int main(int argc, char *argv[]) {
  @autoreleasepool {
    return UIApplicationMain(argc, argv, @"ZOAppTestRunner", @"ZOTestAppDelegate");
  }
}

Here's my ZOAppTestRunner implementation:

#import "ZOAppTestRunner.h"

@implementation ZOAppTestRunner

- (id)init {
    self = [super init];

    if (getenv("GHUNIT_CLI")) {
        [GHTestRunner run];
    }
    return self;
}

@end

Now, when I run the test script I get the following:
./test.sh: line 47: 71649 Abort trap: 6 "/Users/adriancowham/code/apple/src/build/Debug-iphonesimulator/UnitTests.app/UnitTests" -RegisterForSystemEvents

Prior to applying the change by @ianyh, the tests would at least run, but the ones that referenced a UIView crashed.

Thoughts?

@ianyh

@adriancowham is ZOAppTestRunner a subclass of UIApplication?

@adriancowham

Yes it is.

#import <UIKit/UIKit.h>

@interface ZOAppTestRunner : UIApplication

@end
@jmoody
Collaborator

thanks for the great information here.

using the subclassing UIApplication solution above worked for me (thanks @ianyh et. al.)...mostly.

the problem i have now is that if i use xcodebuild -sdk iphonesimulator5.1 , my test suite is run 2x.

example from the test section of my Makefile:

CC= GHUNIT_CLI=1 xcodebuild -target Tests-iOS -configuration "Debug" -sdk iphonesimulator5.1 build

i have tried using -workspace and -scheme with the same result.

if i just use -sdk iphonesimulator or -sdk iphonesimulator6.0, the test suite runs once (as expected).

i'll be the first to admit that my xcodebuild foo is pretty weak.

am i doing something wrong?

anyone else seeing this?

this screenshot on evernote might be illuminating...
cheers

@ianyh

@jmoody Your problem is that when you run with iphonesimulator5.1 the normal execution works. So my hack to run the tests in UIApplication init runs and when that finishes UIApplicationMain runs successfully which causes GHUnit internals to run the tests a second time.

I think - [UIDevice systemVersion] should give you @"5.1" (though I haven't confirmed that), so you can try accessing that and change the conditional in init to

if (getenv("GHUNIT_CLI") && [[[UIDevice currentDevice] systemVersion] isEqualToString:@"6.0"]) {
    [GHTestRunner run];
}

or something of the like.

@adriancowham Not sure what the problem there is. Can you catch it in debugger? Say, remove the conditional and always run [GHTestRunner run] in the test runner init. It's possible that you are accessing something in UIKit that gets set up after the UIApplication instance gets initialized.

@jmoody
Collaborator

@ianyh - ah! of course! thanks!

@drekka

Hi all, thanks fro the hard work, you have saved me. Subclassing UIApplication and launching with workspace and scheme has worked here.

@adriancowham

Hey folks, though I'd update you all with my status, I was able to get the approach described by @ianyh working. I had two things going on in my environment that I thought I'd share just in case anyone else runs into the same thing.

First, as described above, I was getting this error when I tried to run the tests on the command line:
./test.sh: line 47: 71649 Abort trap: 6 "/Users/adriancowham/code/apple/src/build/Debug-iphonesimulator/UnitTests.app/UnitTests" -RegisterForSystemEvents

It turns out I had a hung UnitTests.app in my process list, after killing it and re-running the tests the error went away. However, when re-running the tests, they hung. So I CTRL-C my terminal to get my prompt back, re-run the tests again and sure enough I got the Abort trap: 6 error again due to a hung instance of UnitTests.app.

So at this point my theory is that running the tests using the approach by @ianyh has introduced ( or exploited ) a deadlock situation in my code. Long story short, running the tests from the UIApplication initializer causes all your tests to be run on the main UI thread, where as before they weren't. There are places in my code where I call

dispatch_sync( dispatch_get_main_queue(), ^ { successBlock(); } );

So, if any of the tests that execute this code are run on the main UI thread, we get a deadlock. The easy fix is to change the code to dispatch_async but we have a specific need for the dispatch_sync.

To work around this I modified the solution by @ianyh as follows:

- (id)init {
    self = [super init];
    if (getenv("GHUNIT_CLI")) {
        __block BOOL done = NO;
        NSOperationQueue * queue = [[ NSOperationQueue alloc ] init ];
        [queue addOperationWithBlock:^{
            [GHTestRunner run];
            done = YES;
        }];

        while( !done ) {            
            [[NSRunLoop currentRunLoop] runUntilDate: [NSDate dateWithTimeIntervalSinceNow: 5] ];
        }
    }
    return self;
}

The above code is the initializer for my subclass of UIApplication. All it does is runs the tests on a separate thread, and waits for them to complete. It's not the most elegant solution, but my tests now run on command line and in Jenkins, I couldn't be happier.

A huge thanks to @ianyh and @superrunt19 for contributing to this issue.

@JonB

This is still broken on the latest beta.

@hborders

I'm using adriancowham's solution, and it seems to work for me.

@gfx

My project works with the solution by ianyh & adriancowham.

FYI, here is the code I'm using:

// main.m
#import <UIKit/UIKit.h>
#import "GHUnitIOS/GHUnit.h"

@interface MyUIApp : UIApplication
@end

@implementation MyUIApp

- (id)init
{
    self = [super init];
    if (self && getenv("GHUNIT_CLI") && [[[UIDevice currentDevice] systemVersion] doubleValue] >= 6.0) {
        __block BOOL done = NO;
        NSOperationQueue * queue = [[ NSOperationQueue alloc ] init ];
        [queue addOperationWithBlock:^{
            int status = [GHTestRunner run];
            if (status != 0) {
                NSString *reason = [NSString stringWithFormat:@"failed to test %d", status];
                @throw [NSException exceptionWithName:@"TestFailure" reason:reason userInfo:nil];
            }
            done = YES;
        }];

        while( !done ) {
            [[NSRunLoop currentRunLoop] runUntilDate: [NSDate dateWithTimeIntervalSinceNow: 5] ];
        }
    }

    return self;
}

@end

int main(int argc, char *argv[])
{
    @autoreleasepool {
        return UIApplicationMain(argc, argv, @"MyUIApp", @"GHUnitIPhoneAppDelegate");
    }
}
@johnboiles
Collaborator

Thanks @ianyh, @adriancowham, and @gfx!

It looks like the application delegate class doesn't get init'd until after [UIApplication init]. Some of the classes I test, depend on the app delegate for some global information, so I had to add: self.delegate = [[MyCustomApplicationDelegate alloc] init]; to the UIApplication subclass's init. All together, my code looks something like this (@gfx's solution plus my fix adapted to coding conventions of my codebase):

- (id)init {
  if ((self = [super init])) {
    if (getenv("GHUNIT_CLI") && [[[UIDevice currentDevice] systemVersion] doubleValue] >= 6.0) {

      // Looks like the app delegate doesn't get init'd until after this method, so do it manually.
      self.delegate = [[MyCustomApplicationDelegate alloc] init];

      __block BOOL done = NO;
      NSOperationQueue *queue = [[NSOperationQueue alloc ] init];
      [queue addOperationWithBlock:^{
        NSInteger status = [GHTestRunner run];
        if (status != 0) {
          NSString *reason = [NSString stringWithFormat:@"Failed to test with status: %d", status];
          @throw [NSException exceptionWithName:@"TestFailure" reason:reason userInfo:nil];
        }
        done = YES;
      }];

      while(!done) {
        [[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:5]];
      }
    }
  }
  return self;
}
@barumrho

Has anyone tried this? http://stackoverflow.com/questions/12557935/xcode-4-5-command-line-unit-testing/14395861#14395861

I just tried it and it seems to work. I just wasn't sure why kind of side effect it may have.

@jmoody
Collaborator

in case this happens to someone else, i just update to Xcode 4.6. this brought in the 6.1 sdk.

to get my cli tests to work for 6.0, i was using a custom application delegate and testing the version like this:

if (getenv("GHUNIT_CLI") && [@"6.0" isEqualToString:sysv])

which will obviously fail.

@jmoody
Collaborator

@barumrho thanks for that link! i tried out that technique and it worked. however, the category on UIWindow will inhibit the test target from ever opening in the simulator. below, i have pasted what i think is a complete solution (no custom application delegate needed).

i have tested on the command line and in the simulator for iOS 5.1, 6.0, and 6.1.

https://gist.github.com/467200

thoughts?

#import <objc/runtime.h>
#import <objc/message.h>

@interface UIWindow (Private)
- (void) swizzled_createContext;
@end

@implementation UIWindow (Private)
- (void) swizzled_createContext {
  // nop
}
@end


int main(int argc, char *argv[]) {
  int retVal;
  @autoreleasepool {
    CFMessagePortRef portRef = NULL;
    if (getenv("GHUNIT_CLI")) {
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wundeclared-selector"
      Method originalWindowCreateContext =
      class_getInstanceMethod([UIWindow class],
                              @selector(_createContext));
#pragma clang diagnostic pop

      Method swizzledWindowCreateConext =
      class_getInstanceMethod([UIWindow class],
                              @selector(swizzled_createContext));

       method_exchangeImplementations(originalWindowCreateContext,
                                      swizzledWindowCreateConext);


      portRef = CFMessagePortCreateLocal(NULL,
                                         (CFStringRef) @"PurpleWorkspacePort",
                                         NULL,
                                         NULL,
                                         NULL);
    }

    retVal = UIApplicationMain(argc, argv,
                               NSStringFromClass([UIApplication class]),
                               @"GHUnitIOSAppDelegate");
    if (portRef != NULL) { CFRelease(portRef); }
  }
  return retVal;
}
@drekka

Updating to @jmoody's code, but still using a custom UIApplication seems to be working. Just using the main by itself didn't work. Still testing.

@jmoody
Collaborator

@drekka are you saying that code above did not work? how did it fail?

@gardner

This is still an issue with Xcode 4.6 after following the directions to enable GHUnit from the command line.

@RoCry

@gardner

Only if I make the sdk 5.* will work fine...

like this:

GHUNIT_CLI=1 xcodebuild -scheme GHTest -configuration Debug -sdk iphonesimulator5.1 -workspace THE_NAME.xcworkspace clean build

@paytonrules paytonrules referenced this issue in OCDSpec/OCDSpec2
Closed

Running Logic Tests not Application Tests #1

@smoynes

I have used the code from @jmoody in his comment #96 (comment) and it works for me on Xcode 4.5 with SDK 6.1 in both the simulator and on the command line.

:thumbsup:

@jmoody
Collaborator

i am also using this in Xcode 5 previews. it works in the simulator (although the UI is a bit squirrelly in iOS 7) and on the command line.

@jmoody jmoody referenced this issue
Closed

an attempt to revive GHUnit #132

16 of 17 tasks complete
@x2on x2on closed this in d10f5e6
@mkauppila mkauppila referenced this issue from a commit in mkauppila/Sidekick
@mkauppila mkauppila Fix running tests from command line
Force the main of the test app to run GHTestRunner if CHUNIT_CLI
is specified properly. The solution was proposed in gh-unit issue 96
(gh-unit/gh-unit#96).
e9bbf46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.