Cannot run tests with scan (error 65)

[vittoriom]

Scanfile:

scheme '[REDACTED]'
skip_slack true
slack_url 'https://whatever.com'  <----- See #1 issue on scan
skip_html_open true
clean true

Output:

** TEST FAILED **
[17:11:58]: Exit status: 65
/Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/lib/scan/error_handler.rb:32:in `handle_build_error': Error building/testing the application - see the log above (RuntimeError)
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/lib/scan/runner.rb:30:in `block in test_app'
    from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.25.2/lib/fastlane_core/command_executor.rb:62:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.25.2/lib/fastlane_core/command_executor.rb:62:in `execute'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/lib/scan/runner.rb:23:in `test_app'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/lib/scan/runner.rb:9:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/lib/scan/manager.rb:10:in `work'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/lib/scan/commands_generator.rb:41:in `block (2 levels) in run'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:153:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/runner.rb:428:in `run_active_command'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/runner.rb:68:in `run!'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/delegates.rb:15:in `run!'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/lib/scan/commands_generator.rb:60:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/lib/scan/commands_generator.rb:14:in `start'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.1.0/bin/scan:6:in `<top (required)>'
    from /usr/bin/scan:23:in `load'
    from /usr/bin/scan:23:in `<main>'

FWIW:
I saw a red line in the console a few seconds before I got this stack trace (that weirdly enough is not there anymore) about "waiting 120 seconds for the simulator to boot"

FWIW/2:
Running tests with CMD+U from Xcode works fine

Possibly related:
Bitrise issue

P.s.
Thanks for scan, I was really waiting for something like this!

[jshier]

I have seen error 65s as well. Strangely though, usually they show up after the final touch step but sometimes they show up after all of the tests have run. It's bizarre.

Executed 158 tests, with 1 failure (0 unexpected) in 1.997 (2.072) seconds [23:44:50]: Exit status: 65

That run also produced crash 00a01d1fb4cfd810e9e0128ebd5ba6de and my after_all block didn't run.

[vittoriom]

Any clue on this?

[adamcohenrose]

I see error 65 when tests fail, but not when they pass... Would it be possible to have an option that allows error 65 to indicate a failed build, but to let the lane continue?

My specific case is that I would like to run tests in Jenkins across multiple simulators and then gather JUnit reports. When the scan task fails, the report gathering never happens and so I don't see test results in Jenkins!

[augustj]

Some information regarding that 120 second timeout issue - I know it's not necessarily the main culprit here, but I want to share what I've learned (painfully).
From what I understand, calling xcodebuild ..... test starts by launching the simulator, followed by compiling/building the app. If the build process takes longer than 120 seconds, you will see that error about waiting for the simulator to boot. I starting seeing this problem on CircleCI and Bitrise as our project grew larger. My solution was to use xctool (within fastlane) to split build and test into a two step process. I first call xctool ...... build-tests, then xctool ...... run-tests. Unfortunately, the fastlane xctest action always starts with a clean, so I was unable to split the process. If scan is where the effort will be focused going forward, I'd like to see support for first building the tests, then executing the tests.

[jshier]

@augustj Thanks for the explanation, that makes so much sense. Given that you can build for testing separately from actually running the tests, it may be possible to do what you want. I don't think the test action requires a clean, as you can pass it separately. Or are you saying it always cleans every time, even when the clean flag isn't passed?

[augustj]

As I dig through the cobwebs in my head - and look at the fastlane source code, I think the issue wasn't the clean was always being added, it was that build was always added. I just wanted to call test without build.

class XctestAction < Action
      def self.run(params)
        Helper.log.info "Have you seen the new 'scan' tool to run tests? https://github.com/fastlane/scan".yellow
        params_hash = params || {}
        params_hash[:build] = true
        params_hash[:test] = true
        XcodebuildAction.run(params_hash)
      end

[augustj]

After some more experimentation, even calling xcodebuild .... test triggers build steps (including running the scripts for Cocoapods). xctool is an entirely different beast that doesn't use xcodebuild to run the tests. While calling xcodebuild build followed by xcodebuild test helps avoid the simulator timeouts, it's not quite as good as running the tests with xctool. I'll try a two step process with scan and see how it compares to testing with xctool.

Update: I am unable to successfully setup a two step build-test process using xcodebuild (which is what scan uses). So far I am only able to accomplish this with xctool's build-tests and run-tests actions. This is a shortcoming of xcodebuild, not fastlane or any of the tools built on top of it. If someone can make this work, let me know.

[vittoriom]

@augustj please also have a look at facebookarchive/xctool#248 regarding the behavior of xctool

[jshier]

@augustj I was able to get a two phase build to work just fine, or am I missing something?

xcodebuild -destination "platform=iOS Simulator,name=iPhone 6s,OS=9.1" -scheme "APP" -workspace "APP.xcworkspace" -configuration "Test" build 
&& xcodebuild -destination "platform=iOS Simulator,name=iPhone 6s,OS=9.1" -scheme "APP" -workspace "APP.xcworkspace" -configuration "Test" test

Running those commands results in the test command using the stored product from the build command just fine.

[augustj]

@jshier - I should clarify a little - I was able to get a two step process to build and test, but the build step basically didn't seem to do much and the test step was stuck with compiling (again) - BUT I just realized what I did wrong. In my scheme, I specific different configurations for Run/Test. Once I specified the same configuration for both build and test, it was fine. Works like a charm. Now I can yank xctool out completely. I've always liked the output/reporting options for the fastlane tools much better.

[jshier]

@augustj We you able to do that using fastlane's xcbuild and xctest? I don't see a way to tell xctest not to build. Or did you just put a custom shell command in there to do the two phase build?

[augustj]

@jshier

I ended up with a 3 step process

1. xcodebuild w/ clean:true
2. xcodebuild w/ build:true
3. scan

I gave up on xctest - seeing an update to scan renewed my interest in abandoning xctool.

Here is what I have in my fastlane script at the moment (subject to cleanup, of course):

  xcodebuild(scheme: scheme, workspace: $workspace, configuration: 'Test', clean: true)
  xcodebuild(scheme: scheme, workspace: $workspace, configuration: 'Test', destination: $simulator_destination, build: true)
  scan(scheme: scheme, workspace: $workspace, destination:$simulator_destination, output_directory: $output_directory)

[lam2558]

I fixed this by updating ruby and ruby gem:
rbenv install 2.2.2
rbenv global 2.2.2
gem update --system

[jshier]

@augustj Hmm, running xcodebuild and then scan doesn't seem to replicate the behavior I saw with the manual xcodebuild commands chained together. This despite the scan command not having the build component. It still does a full rebuild, leading to possible 65s.

@KrauseFx Any ideas? Would a two phase build be possible for scan?

[jshier]

@augustj Ah, apparently it matters if you do a clean as part of the xcodebuild command. Doing it as a separate one and then building and then running scan apparently works fine. Weird.

[jshier]

@KrauseFx 0.2.0 did not fix the return 65 problems I was having. The two phase build system described by @augustj worked fine.

[lemonkey]

👍 Would really like to see scan patched to address this..

[lemonkey]

Latest release 0.3.0 still throws a Ruby runtime exception whenever a unit test fails. Also, it seems the math being used to count the total number of tests and the number of failures is a bit off in the initial summary string. The "Test Results" summary table is correct however.

Executed 103 tests, with 8 failures (0 unexpected) in 119.544 (119.603) seconds

[11:16:50]: Exit status: 65
+--------------------+-------+
|        Test Results        |
+--------------------+-------+
| Number of tests    | 143   |
| Number of failures | 18    |
+--------------------+-------+

[11:17:43]: Successfully generated report at '/test-reports/report.junit'
/Library/Ruby/Gems/2.0.0/gems/scan-0.3.0/lib/scan/runner.rb:69:in `handle_results': Tests failed (RuntimeError)

[viktorbenei]

Hi everyone. Just wanted to add a couple of notes, as we did quite a lot of research when we implemented the Bitrise Xcode Test step.

The issue is that if you call xcodebuild with just test (xcodebuild ... test) the build time will be included in the test boot time. This practically means that if your clean build (xcodebuild ... clean test) takes more than 120 seconds you'll always get a timeout. We have a test project to demonstrate the issue, which has a simple Bash build script with a 120sec sleep, so the build process always takes more than 120 seconds: https://github.com/bitrise-io/simulator-launch-timeout-includes-build-time

The sample project includes links to a related radar ticket and some more explanation.

What's the solution?

1. Either you do an xcodebuild ... build separately and then an xcodebuild ... test without the clean parameter
2. Or - and this is what we use in our bitrise.io Xcode Test step, and also the one we sent a Pull Request to the previous fastlane solution - you can simply add the build parameter before the test parameter. Example: xcodebuild ... clean build test.

Couple of pointers:

• this works: https://github.com/bitrise-io/simulator-launch-timeout-includes-build-time/blob/master/test-ok.sh#L6
• while this does not: https://github.com/bitrise-io/simulator-launch-timeout-includes-build-time/blob/master/test-fails.sh#L6

EDIT: just checked with Xcode 7.1 and there might have been changes, as running bash test-fails.sh did not fail if the iOS Simulator was running, only if you don't have the simulator open (which is usually the case on a CI server/service) ; previously it failed in every case, with both Xcode 6.4 & 7.0

[wolffan]

I can't use scan either....

I ended up with the following (thanks for all great examples):

xcodebuild(
  workspace:"",
  scheme:"",
  destination: "OS=9.1,name=iPhone 6")

xctest( 
  destination: "OS=9.1,name=iPhone 6",
  workspace:"",
  scheme:"")

[scottrhoyt]

@viktorbenei Thanks for your really helpful comments. Any chance to get a PR for scan to either:

1. xcodebuild ... clean build test
2. open up the simulator before running the rest of scan

[scottrhoyt]

This issue keeps popping up for us. The workarounds here offered a partial fix. However, we ran into another problem. We need to run Xcode UI Tests, which require iOS 9 or above. Our CI server (Bitrise) defaults to an iOS 8 device. So we are trying to use the above workarounds while specifying building and testing on an iOS 9 device, but we still run into the double compilation/timeout problem. It looks something like this:

xcodebuild(scheme: "MyScheme", workspace: $workspace, configuration: "Debug", destination: "platform=iOS Simulator,name=iPhone 6,OS=9.1", clean: true)
xcodebuild(scheme: "MyScheme", workspace: $workspace, configuration: "Debug", destination: "platform=iOS Simulator,name=iPhone 6,OS=9.1", build: true)
scan(scheme: "MyScheme", workspace: $workspace, destination:"platform=iOS Simulator,name=iPhone 6,OS=9.1", output_directory: $output_directory, code_coverage: true)

Any help would be appreciated.

[scottrhoyt]

This looks to be the same as #26 and #34

[scottrhoyt]

FWIW, my current work around is to test via xctool.

[gretzki]

Hi there,

since I'm having the same issue here, that I want to further process the tests result, I ended up forking scan. My suggestion is to provide an option to omit the exception. Additionally, the number of failing tests is provided as return value, so that processing this value is possible.

Please review #64 and let me hear what you think.
It sure works for me ;)

Cheers,
Chris

[alliannas]

@KrauseFx is there any work being done to resolve this 65 error in scan?

The alternative of using xcclean, xcbuild and xctest works, but then you are no longer using scan and its awesome features.

Is there any work being done to resolve this with scan?

[jshier]

@alliannas My workaround was to use the xcodebuild action to do a clean and then a build of my test target. Then I run scan and all it has to do is build the tests. That way I don't hit the 120 second timeout from the simulator.

[lam2558]

Try booting up the simulator first before you run scan.
xcrun instruments -t 'Blank' -l 1 -w "${name} (${OS})"

[frobichaud]

It looks like XCode 7.2.1 may solve this problem

Fixed in Xcode 7.2.1:
• Command line tool ‘xcodebuild test’ will no longer time out waiting for Simulator.app to launch

[vittoriom]

@frobichaud that would be amazing. Will check it out and hope for the best!

[vittoriom]

I just checked it out, and I confirm that scan correctly run the tests!
The problem though, is that it still gives error 65 at the end, even though the junit and html reports are correctly generated.
Also, there is a crash at the end of the process:

** TEST FAILED **
[13:08:31]: Exit status: 65

[...]

/Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/error_handler.rb:32:in `handle_build_error': [!] Error building/testing the application - see the log above (RuntimeError)
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/runner.rb:30:in `block in test_app'
    from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.36.1/lib/fastlane_core/command_executor.rb:82:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.36.1/lib/fastlane_core/command_executor.rb:82:in `execute'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/runner.rb:23:in `test_app'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/runner.rb:9:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/manager.rb:10:in `work'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/commands_generator.rb:41:in `block (2 levels) in run'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:153:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/runner.rb:428:in `run_active_command'
    from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.36.1/lib/fastlane_core/ui/fastlane_runner.rb:23:in `run!'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/delegates.rb:15:in `run!'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/commands_generator.rb:60:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/commands_generator.rb:14:in `start'
    from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/bin/scan:6:in `<top (required)>'
    from /usr/bin/scan:23:in `load'
    from /usr/bin/scan:23:in `<main>'

[alliannas]

I just tested this with Xcode 7.2.1 and fastlane 1.57.0 and newest scan
0.4.2 and the problem disappeared.

No more weird timeouts issues.

On Fri, Feb 5, 2016 at 12:09 PM, Vittorio Monaco notifications@github.com
wrote:

I just checked it out, and I confirm that scan correctly run the tests!
The problem though, is that it still gives error 65 at the end, even
though the junit and html reports are correctly generated.
Also, there is a crash at the end of the process:

** TEST FAILED **
[13:08:31]: Exit status: 65

[...]

/Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/error_handler.rb:32:in handle_build_error': [!] Error building/testing the application - see the log above (RuntimeError) from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/runner.rb:30:inblock in test_app'
from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.36.1/lib/fastlane_core/command_executor.rb:82:in call' from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.36.1/lib/fastlane_core/command_executor.rb:82:inexecute'
from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/runner.rb:23:in test_app' from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/runner.rb:9:inrun'
from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/manager.rb:10:in work' from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/commands_generator.rb:41:inblock (2 levels) in run'
from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:178:in call' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:178:incall'
from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:153:in run' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/runner.rb:428:inrun_active_command'
from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.36.1/lib/fastlane_core/ui/fastlane_runner.rb:23:in run!' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/delegates.rb:15:inrun!'
from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/commands_generator.rb:60:in run' from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/lib/scan/commands_generator.rb:14:instart'
from /Library/Ruby/Gems/2.0.0/gems/scan-0.4.2/bin/scan:6:in <top (required)>' from /usr/bin/scan:23:inload'
from /usr/bin/scan:23:in `

'

—
Reply to this email directly or view it on GitHub
#7 (comment).

Alexander Annas Helgason
iOS Engineer
QuizUp / Plain Vanilla Games
e. alliannas@quizup.com
Favorite topics: Game of Thrones
The information contained in this message and any attachments are
confidential and may be privileged. If you are not the intended recipient,
please uphold strict confidentiality and neither read, copy, nor otherwise
make use of their content in any way, you are on notice that any
distribution of this messages, in any form is strictly prohibited and
please notify sender immediately — by replying to this message or by
sending an e-mail to info@quizup.com -- and destroy all copies of this
message and any attachments.
Thank you.

[mpirri]

Thanks for your feedback, all!
Since Xcode 7.2.1 fixes the scan timeout originally referenced by the issue original poster, I'm going to close this issue. Please do not hesitate to open other another issue if you feel that there are other, unaddressed behaviors captured in the above thread that were not fixed by Xcode 7.2.1.

[lucatorella]

With Xcode 7.2.1 I still get:

xcodebuild[17860:76616]  iPhoneSimulator: Could not launch simulator: -600

Testing failed:
Test target xxx encountered an error (The operation couldn’t be completed. (OSStatus error -600.)

My lane is just:

desc "Runs all the tests"
  lane :test do
    scan(
      workspace: "xxx.xcworkspace",
      scheme: "xxx",
      clean: true
    )
  end

how did you fix this issue?

[lam2558]

I usually have xcode boot a specific simulator first, then when i run scan, i tell it to run that specific one that was booted up.

killall "Simulator"
xcrun instruments -t 'Blank' -l 1 -w "${name} (${OS})"
scan --clean "xxx" --configuration "Test" --device "${name} (${OS})" --scheme "xxx"

[fastlanebot]

This issue was migrated to fastlane/fastlane#3368. Please post all further comments there.

fastlane is now a mono repo, you can read more about the change in our blog post. All tools are now available in the fastlane main repo 🚀