New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot run tests with scan (error 65) #3368
Comments
I have seen error 65s as well. Strangely though, usually they show up after the final touch step but sometimes they show up after all of the tests have run. It's bizarre.
That run also produced crash |
Any clue on this? |
I see error 65 when tests fail, but not when they pass... Would it be possible to have an option that allows error 65 to indicate a failed build, but to let the lane continue? My specific case is that I would like to run tests in Jenkins across multiple simulators and then gather JUnit reports. When the scan task fails, the report gathering never happens and so I don't see test results in Jenkins! |
Some information regarding that 120 second timeout issue - I know it's not necessarily the main culprit here, but I want to share what I've learned (painfully). |
@augustj Thanks for the explanation, that makes so much sense. Given that you can build for testing separately from actually running the tests, it may be possible to do what you want. I don't think the test action requires a clean, as you can pass it separately. Or are you saying it always cleans every time, even when the clean flag isn't passed? |
As I dig through the cobwebs in my head - and look at the fastlane source code, I think the issue wasn't the
|
After some more experimentation, even calling Update: I am unable to successfully setup a two step build-test process using xcodebuild (which is what scan uses). So far I am only able to accomplish this with xctool's build-tests and run-tests actions. This is a shortcoming of xcodebuild, not fastlane or any of the tools built on top of it. If someone can make this work, let me know. |
@augustj please also have a look at facebookarchive/xctool#248 regarding the behavior of xctool |
@augustj I was able to get a two phase build to work just fine, or am I missing something?
Running those commands results in the |
@jshier - I should clarify a little - I was able to get a two step process to build and test, but the build step basically didn't seem to do much and the test step was stuck with compiling (again) - BUT I just realized what I did wrong. In my scheme, I specific different configurations for Run/Test. Once I specified the same configuration for both build and test, it was fine. Works like a charm. Now I can yank xctool out completely. I've always liked the output/reporting options for the fastlane tools much better. |
I ended up with a 3 step process
I gave up on xctest - seeing an update to scan renewed my interest in abandoning xctool. Here is what I have in my fastlane script at the moment (subject to cleanup, of course):
|
I fixed this by updating ruby and ruby gem: |
@augustj Hmm, running @KrauseFx Any ideas? Would a two phase build be possible for scan? |
👍 Would really like to see scan patched to address this.. |
Latest release 0.3.0 still throws a Ruby runtime exception whenever a unit test fails. Also, it seems the math being used to count the total number of tests and the number of failures is a bit off in the initial summary string. The "Test Results" summary table is correct however.
|
Hi everyone. Just wanted to add a couple of notes, as we did quite a lot of research when we implemented the Bitrise Xcode Test step. The issue is that if you call The sample project includes links to a related radar ticket and some more explanation. What's the solution?
Couple of pointers:
EDIT: just checked with Xcode 7.1 and there might have been changes, as running |
I can't use scan either.... I ended up with the following (thanks for all great examples):
|
@viktorbenei Thanks for your really helpful comments. Any chance to get a PR for scan to either:
|
This issue keeps popping up for us. The workarounds here offered a partial fix. However, we ran into another problem. We need to run Xcode UI Tests, which require iOS 9 or above. Our CI server (Bitrise) defaults to an iOS 8 device. So we are trying to use the above workarounds while specifying building and testing on an iOS 9 device, but we still run into the double compilation/timeout problem. It looks something like this:
Any help would be appreciated. |
|
FWIW, my current work around is to test via xctool. |
Hi there, since I'm having the same issue here, that I want to further process the tests result, I ended up forking scan. My suggestion is to provide an option to omit the exception. Additionally, the number of failing tests is provided as return value, so that processing this value is possible. Please review fastlane-old/scan#64 and let me hear what you think. Cheers, |
@KrauseFx is there any work being done to resolve this 65 error in scan? The alternative of using xcclean, xcbuild and xctest works, but then you are no longer using scan and its awesome features. Is there any work being done to resolve this with scan? |
@alliannas My workaround was to use the |
Try booting up the simulator first before you run scan. |
It looks like XCode 7.2.1 may solve this problem
|
@frobichaud that would be amazing. Will check it out and hope for the best! |
I just checked it out, and I confirm that
|
I just tested this with Xcode 7.2.1 and fastlane 1.57.0 and newest scan No more weird timeouts issues. On Fri, Feb 5, 2016 at 12:09 PM, Vittorio Monaco notifications@github.com
Alexander Annas Helgason |
Thanks for your feedback, all! |
With Xcode 7.2.1 I still get:
My lane is just:
how did you fix this issue? |
I usually have xcode boot a specific simulator first, then when i run scan, i tell it to run that specific one that was booted up. killall "Simulator" |
I'm still getting this issue, |
Hi @jcampbell05! Was scan working before? What version of xCode are you using? Thanks! |
I'm using Xcode 7.3 and I 'm getting the same error. I updated ruby and ruby gem, but it didn't work |
@kmilitagaitanm Thanks for reaching out. We are going to track this issue over on #3886. Can you update over there with your full output? Closing this out as a duplicate. Thanks! 👍 |
Hi all, is it okay already? Currently, I am on fastlane 1.8 and using Jenkins. I still got from /usr/local/lib/ruby/gems/2.2.0/gems/scan-0.5.2/lib/scan/runner.rb:31:in `block in test_app' And I have tried to clean and build before I scan. (run by Jenkins). |
Hi @khantthulinn. You can track this issue on #3886. Thanks! |
@TKBurner Thanks. I will. |
Scanfile:
Output:
FWIW:
I saw a red line in the console a few seconds before I got this stack trace (that weirdly enough is not there anymore) about "waiting 120 seconds for the simulator to boot"
FWIW/2:
Running tests with CMD+U from Xcode works fine
Possibly related:
Bitrise issue
P.s.
Thanks for scan, I was really waiting for something like this!
The text was updated successfully, but these errors were encountered: