Skip to content
This repository has been archived by the owner on Feb 2, 2021. It is now read-only.

Tests for static libraries #5

Closed
drewcrawford opened this issue May 2, 2013 · 10 comments
Closed

Tests for static libraries #5

drewcrawford opened this issue May 2, 2013 · 10 comments

Comments

@drewcrawford
Copy link

It seems that tests for static libraries aren't supported. xctool currently outputs

/Applications/Xcode.app/Contents/Developer/Tools/RunUnitTests:68: note: RunUnitTests exited without running tests because TEST_AFTER_BUILD was set to NO.

If you pass TEST_AFTER_BUILD=1 you get

/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/Tools/RunPlatformUnitTests:81: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).

I currently work around this issue by modifying the "Run Script" phase of the test target in the following way:

set -e
if [ "$RUN_UNIT_TEST_WITH_IOS_SIM" = "YES" ]; then
test_bundle_path="$BUILT_PRODUCTS_DIR/$PRODUCT_NAME.$WRAPPER_EXTENSION"
SOURCE_DIRECTORY=$SRCROOT
echo sim --sdk=6.1  $SDKROOT/Developer/usr/bin/otest -NSTreatUnknownArgumentsAsOpen NO -ApplePersistenceIgnoreState YES -SenTest Self $CODESIGNING_FOLDER_PATH
sim --sdk=6.1  $SDKROOT/Developer/usr/bin/otest -NSTreatUnknownArgumentsAsOpen NO -ApplePersistenceIgnoreState YES -SenTest Self $CODESIGNING_FOLDER_PATH
else
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests"
fi

and then passing the argument RUN_TESTS_WITH_IOS_SIM=YES.

Note that this workaround uses several completely undocumented SDK tools. I have reverse-engineered them enough to find the correct arguments, but I have no idea how stable this is for future SDKs.

@fpotter
Copy link
Contributor

fpotter commented May 2, 2013

Hi Drew,

With xctool, you shouldn't have to pass TEST_AFTER_BUILD=1 anymore. It doesn't use any of the RunPlatformUnitTests or RunUnitTests scripts to start tests.

Instead you should just be able to do something like...

xctool \
  -workspace YourWorkspace.xcworkspace 
  -scheme YourScheme \
  -sdk iphonesimulator \
  -configuration Debug \
  -arch i386 \
  build build-tests run-tests.

Let me know if this doesn't work....

@drewcrawford
Copy link
Author

Let me know if this doesn't work....

That's why I opened the issue :-)

Maybe a formal bug report would clear it up?

Reproduction steps

git clone git@github.com:drewcrawford/DoesThisRun.git
cd DoesThisRun
bash /path/to/xctool.sh -project DoesThisRun.xcodeproj -scheme DoesThisRun -sdk iphonesimulator build test

Expected results

Failure of test: testExample

Actual results

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
/Applications/Xcode.app/Contents/Developer/Tools/RunUnitTests:68: note: RunUnitTests exited without running tests because TEST_AFTER_BUILD was set to NO.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


  run-test DoesThisRunTests.octest (iphonesimulator6.1, application-test, GC OFF)
    [2013-05-02 14:56:50|Info] Installing '/Users/drew/Library/Developer/Xcode/DerivedData/DoesThisRun-ehiqteaoaxcjsdbhmticnnkibckc/Build/Products/Debug-iphonesimulator' ...
    [2013-05-02 14:56:52|Info] Launching test host and running tests...
** TEST SUCCEEDED: 0 of 0 tests passed ** (3848 ms)

...

@fpotter
Copy link
Contributor

fpotter commented May 2, 2013

Thanks for the repro case!

This part looks suspicious...

[2013-05-02 14:56:50|Info] Installing '/Users/drew/Library/Developer/Xcode/DerivedData/DoesThisRun-ehiqteaoaxcjsdbhmticnnkibckc/Build/Products/Debug-iphonesimulator'

It should be installing the TEST_HOST app.

@drewcrawford
Copy link
Author

I think it's because I have TEST_HOST set to BUNDLE_LOADER which is set to library's .a. It's a workaround for another problem that's documented in more detail here.

@jasongregori
Copy link

I was having the same issue. Updating to 0.1.2 fixed it. Thanks!

@timor-raiman
Copy link

I'm still hitting this in 0.1.7

As in #114, this is accompanied by a crash when passing the -freshInstall option.
(the crash persists regardless of having run build-tests prior to run-tests, or using just tests).

I've also tried switching the targets in the project and in the scheme so the test target is first, thinking xctool would pick it up as the install/uninstall entity, but to no avail.

Thanks.

@timor-raiman
Copy link

The workaround suggested by @drewcrawford works for me as well.

However, this really just a work around, since this is not the way xctool is expected to work.

In my case, after applying the workaround, I can trigger the test by forcing a build of the test target (this moves the running of the tests from the test execution phase to the test target build phase):

xctool (...) -sdk iphonesimulator  build-tests RUN_UNIT_TEST_WITH_IOS_SIM=YES

@drewcrawford
Copy link
Author

I just wanted to go on record, since I originally opened this issue, that the vanilla "xcodebuild test" command from Apple works great after upgrading to [REDACTED]. It doesn't have all the logging bells and whistles that xctool does, but I don't really need those.

My advice is that if you actually want to test like the Xcode GUI test button does, upgrade to [REDACTED] and use the official CLI. If xctool's logging output is desired then it might be convenient to write a log parser that prints pretty colors, but I have no real plans to use xctool now.

@fpotter
Copy link
Contributor

fpotter commented Jul 25, 2013

@drewcrawford I went back and tried the DoesThisRun example with the latest xctool, and it worked just fine. Output: https://gist.github.com/fpotter/6419671ab22a43cb5a76

xctool definitely does support testing static library projects. In fact, this was the test case I used most often during development.

I noticed in your example output that xctool was trying to run DoesThisRunTests.octest as an application test instead of a logic test - this isn't right. DoesThisRunTests doesn't define TEST_HOST or BUNDLE_LOADER in its build settings, which is what an application test would do. So, that's odd - I wonder if it was a bug in earlier versions of xctool that we fixed.

You mentioned setting TEST_HOST or BUNDLE_LOADER settings to the lib's .a - I do know that won't work. The TEST_HOST is an app that gets launched and that your test bundle gets loaded into. Since we can't launch or install a .a, it makes sense that this would break in xctool. In Xcode, I noticed it works without error, but I think Xcode is just intelligently skipping over it.

When I did set TEST_HOST and BUNDLE_LOADER to the .a in DoesThisRun, xctool failed exactly how you describe.

We need better error messages here.

Going to close since I think I understand the problem now. Will open a separate issue to add better error messages.

@timor-raiman
Copy link

Confirmed.
Setting TEST_HOST to empty string through xcode target build settings ==> xctool runs tests for a static lib project.

GREAT!

justinseanmartin pushed a commit to justinseanmartin/xctool that referenced this issue Apr 27, 2016
…d-error-to-system-out to master

* commit '5487abb901dabc755161a13d28dc88fc9d76efd0':
  Append the failure error to system out
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants