[fixed] xcpretty hangs with xcode 8 beta (8S128d) #227

Closed
mindows opened this Issue Jun 15, 2016 · 27 comments

Projects

None yet

10 participants

@mindows
mindows commented Jun 15, 2016

When I execute tests with xcodebuild test it exits with an exit code of 65 (some tests failed). However, when I pipe it through xcpretty it hangs indefinitely.

This is the last line I see in my console as it hangs forever
Executed 1801 tests, with 23 failures (3 unexpected) in 218.200 (219.161) seconds

I guess the latest xcodebuild (Beta) has changed how it exits the process? Has anyone else seen this issue?

thanks in advance

@kenji21
kenji21 commented Jun 15, 2016

To manage this, we simply use the PIPESTATUS variable :

${XCODEBUILD_COMMAND} | xcpretty -s 2>&1
XCODEBUILD_RC=${PIPESTATUS[0]}
@mindows
mindows commented Jun 16, 2016

@kenji21 we use PIPESTATUS also, but the problem is that it hangs on the first line actually. Are you using the new xcode beta?

@jorgeazevedo

@mindows I'm seeing the exact same behavior as you are. Did you ever find a workaround?

@kenji21 were you able to reproduce?

@mindows
mindows commented Jul 1, 2016

I filed a radar on this. It's clearly a bug in xcodebuild (xcode 8 beta 1). I also spoke to an apple engineer at WWDC, and he suspected that they forgot to close file descriptor(s) in xcodebuild.

I wrote a hacky python wrapper around xcodebuild to pipe every stdout & stderr and exit when I see the output lines I'm expecting. It's something like Test Suite \'(Selected|All) tests\' (passed|failed)

Hopefully, beta 2 or 3 will fix this issue.

@jorgeazevedo

@mindows nice one! Thanks for that : -)

(sidenote, do you also see the barrage of "The operation requires a selected developer portal team." errors in stderr? I've filed a bug for it, but I wonder if there's something wrong with my config)

@supermarin
Owner

Thanks for reporting! I'll take a look into it.

@JaNd3r
JaNd3r commented Jul 5, 2016

Instead of xcodebuild test I use the new command-line options xcodebuild build-for-testing and xcodebuild test-without-building. Works like a charm. ;)

@jverkoey jverkoey referenced this issue in google/arc-xcode-test-engine Jul 15, 2016
Closed

arc unit hangs on Xcode 8 beta #7

@jverkoey

fyi: build-for-testing appears to terminate fine but test-without-building still hangs in our explorations (google/arc-xcode-test-engine#7).

@KrauseFx
Contributor

Hey @mindows, would you mind sharing the radar number and also cross-post your radar to openradar? 👍

@jverkoey

+sharing the python script would be rad :)

@JaNd3r
JaNd3r commented Jul 18, 2016

Okay, so test-without-building worked on a real device for me. Using the simulator I noticed that the simulator was started on the CI machine including the UI and remained running after the tests finished. Once I manually terminated the simulator, the test-without-building | xcpretty finished with return code 0. It seems, that the simulator is not terminated and keeps the xcodebuild process alive.

@KrauseFx
Contributor

Thanks for sharing @mindows, have you tried yesterday's  release too? 👍

@bestander bestander added a commit to bestander/react-native that referenced this issue Jul 19, 2016
@bestander bestander Removed xcpretty from running objc tests runner
Quite often, ~10%, Travis ObjC tests stall and fail to compile, e.g. https://travis-ci.org/facebook/react-native/jobs/144980707.
When testing locally I noticed that running tests with xcpretty may cause tests to stall and never finish.
Maybe related to supermarin/xcpretty#227

Anyway, we don't need tests to be pretty, we need them to finish.
3dbdde0
@bestander bestander referenced this issue in facebook/react-native Jul 19, 2016
Closed

Removed xcpretty from running objc tests runner #8898

@joshsnelling

FWIW I'm still seeing this in Xcode 8 Beta 3 (8S174q)

@johntmcintosh johntmcintosh referenced this issue in fastlane/fastlane Jul 28, 2016
Closed

Scan hangs until simulator is killed, Xcode 8, iOS 10 simulator #5173

2 of 2 tasks complete
@KrauseFx
Contributor
KrauseFx commented Aug 1, 2016

It seems like Apple has fixed the issue with today's beta, could you check if that works for you?

@KrauseFx KrauseFx referenced this issue in fastlane/fastlane Aug 3, 2016
Closed

Scan hanging when trying to fetch workspaces #5490

2 of 2 tasks complete
@peterstuart

It's working for me now with beta 4.

@supermarin supermarin changed the title from xcpretty hangs with xcode 8 beta (8S128d) to [fixed] xcpretty hangs with xcode 8 beta (8S128d) Aug 18, 2016
@supermarin supermarin closed this Aug 18, 2016
@supermarin
Owner

closing, please reopen if you still see the issue in Xcode 8

@proskd
proskd commented Oct 18, 2016

@supermarin i'm seeing this happen on my tests with Xcode 8 (installed from AppStore) when our project's tests have a failure. If they all pass, this doesn't seem to happen.

Here's a sample command I'm using:

xcodebuild -workspace Sample.xcworkspace -scheme 'Sample Target' test-without-building -destination 'platform=iOS Simulator,name=iPhone 6s Plus,OS=10.0' | xcpretty

Is anyone else still getting this to happen in Xcode 8?

@supermarin
Owner

@proskd which version of xcpretty are you running?

xcpretty --version

@proskd
proskd commented Oct 18, 2016 edited

I was running 0.1.11.

Before updating, I noticed that without changing anything, one or two runs did complete successfully (that is, they exited the process after test failures).

I then updated to 0.2.4 and got it to hang again with the same command.

Even if I pass the set -o pipefail && before hand (as suggested in the readme.md, or use the other suggested workaround) it still will hang.

EDIT: it seems if i simply wait long enough, xcpretty will actually end itself? maybe i'm just not waiting long enough?

@supermarin
Owner

have you tried NSUnbufferedIO=YES xcodebuild ...?

@proskd
proskd commented Oct 18, 2016

That didn't seem to help, but I'd say at this point as my edit above indicated, I think if I wait long enough (several minutes after seeing ** TEST EXECUTE FAILED ** in my console), eventually xcpretty does output a bunch of stuff and exits. I'm just not really clear what it's doing during this period where nothing is being output to the console...

@supermarin
Owner
supermarin commented Oct 18, 2016 edited

I think the problem sounds like Xcode not closing the file handle properly.
Just to test one more case, did you try running it with tee in the middle?

xcodebuild ... | tee xcodebuild.log | xcpretty

@supermarin
Owner

And which exact Xcode 8 version?

@proskd
proskd commented Oct 18, 2016

Version 8.0 (8A218a)

I wasn't using tee, but i did just try that now per your suggestion and it doesn't appear to make a difference one way or the other.

@supermarin
Owner

mind posting the raw xcodebuild.log?

@proskd
proskd commented Oct 18, 2016

Sure, here you go!

xcodebuild.log.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment