Results output with unuseful message on test failure

[matax87]

<failure type="Failure" message="UNKNOWN ERROR - PARSING FAILED: XCTestOutputBarrierTest Case '-[Target.TestSuite TestCase]' started.">
Unknown File:0
</failure>

[ob]

I've seen this when the app crashes and we somehow fail to catch it... what does the main bluepill log say? Do you see any crashes in there? Sometimes there is a .crash file left.

[matax87]

Sometimes there is a .crash file left.

No there isn't any .crash file.

The main TEST-FinalReport.xmlcontains failure with these kind of messages:

<failure type="Failure" message="UNKNOWN ERROR - PARSING FAILED: XCTestOutputBarrierTest Case '-[Target.TestSuite TestCase]' started.">
Unknown File:0
</failure>

Moreover, each simulator folder contains a TEST-TargetSchemeName-results.xml which presents the same problems of the main report and a N-TargetSchemeName-results.txt plain text file with (null) logs, es.:

Test: TestSuiteName
Start: 2017-09-28 11:08:36 +0000
End: 2017-09-28 11:23:34 +0000
Ended: YES
Passed: NO
Total: 57
Fail: 1
Unexpected: 0
Time: 897.602000
Log:
(null)

[PavloBasiuk]

I've got same error, any ideas why it's failing like that?

[oliverhu]

Looks like an edge case in BPTreeParser, @PavloBasiuk @matteomatassoni do you have a way to reproduce this?

[PavloBasiuk]

It fails for each and every failed test-case. And 99% of my tests fails when expected element is not found on view.

[oliverhu]

@PavloBasiuk could you write a sample test to repro (similar to BPTreeParserTests.m's testParsingCrash) and push to a branch?

[krze]

I plan on digging into this too, but just chiming in to say I'm having the same problem. All failed tests have this error in the logs. I confirmed that the failed tests are not the result of the app crashing, it's a legit assertion failure from XCTest

I'm using Xcode 9 and Bluepill 2.0.1 with a project running Swift 4 XCTest UI Tests

[PavloBasiuk]

My setup is practically the same: Xcode9, Bluepill 2.0 with custom changes for jUnit reports, Swift3 and XCUITests. And yeah, I will try to create nice and simple test to reproduce this issue and share with all.

[oliverhu]

Yes, it would be great if you all could provide dummy tests to reproduce this, otherwise it would be super hard for us to understand what is not parsed correctly.

[adamszeremeta]

I have the same issue as described here. In my case this is happening for tests which crashed and for those where assertion failed.

[PavloBasiuk]

It appeared to be easier then I expected. I've just broke one of existing tests for simplicity, so don't merge this PR: #206
But I was able to reproduce same behavior on BPSampleApp from BPSampleAppUITest

To reproduce:

1. Break UITest (BPSampleApp/BPSampleAppUITests/BPSampleAppUITests.m)
2. Build it all and generate .xctestrun file xcodebuild -workspace Bluepill.xcworkspace -scheme BPSampleApp -sdk iphonesimulator -destination 'platform=iOS Simulator,name=iPhone 7,OS=11.0' build-for-testing -derivedDataPath "./XCTest"
3. Run bluepill bluepill -p -f 1 -d "iPhone 7" -T "40" --xctestrun-path "XCTest/Build/Products/BPSampleApp_iphonesimulator11.0-x86_64.xctestrun" -o "./output" -j -i BPSampleAppUITests

[oliverhu]

ah.. i get it. We didn't fully test XCUITest scenarios.

[krze]

If this helps, I'm not specifying any special flags to change the test output format and I get this error. iirc all the switches to change the output format on the command in step 3 are already set to those options by default (plaintext + junit)

[krze]

Is there any progress on this issue? I spent a few hours digging through and couldn't quite figure out what's going on. Has something changed with the Simulator stdout in xcode 9 that's causing this issue?

[oliverhu]

fixed in #226

[PavloBasiuk]

👍