New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve xctestrun documentation #463

Closed
gtroshin opened this Issue Jan 17, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@gtroshin
Copy link

gtroshin commented Jan 17, 2019

my .zip file and .xctestrun are in different folders.

Seems like flank is trying to find xctest in a wrong directory (related where custom .xctestrun is).

java -version
java version "11.0.2" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.2+7-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.2+7-LTS, mixed mode)
java -jar ./flank.jar -v
v4.2.0
5148c339e697778a3714bfb1aa6d4a9a120c5164
xxx@xxx-27:~/ios/test_runner$ ls -la
total 49168
drwxr-xr-x   4 x  staff   128B Jan 17 09:37 ./
drwxr-xr-x@ 53 x  staff   1.7K Jan 17 09:03 ../
-rw-r--r--   1 x  staff   1.5K Jan 17 09:37 flank.ios.yml
-rw-r--r--   1 x  staff    24M Jan 11 23:27 flank.jar
x@xxx:~/ios/test_runner$ java -jar ./flank.jar firebase test ios run
IosArgs
    gcloud:
      results-bucket: test-lab-xxx
      results-dir: null
      record-video: true
      timeout: 30m
      async: false
      project: xxx
      results-history-name: null
      # iOS gcloud
      test: /Users/x/MyTests.zip
      xctestrun-file: /Users/x/Desktop/EmailSignupMethodTest_XXXUITests_iphoneos12.1-arm64e.xctestrun
      xcode-version: 10.1
      device:
        - model: iphonex
          version: 12.0
          locale: en_US
          orientation: portrait
      flaky-test-attempts: 3

    flank:
      testShards: -1
      repeatTests: 1
      smartFlankGcsPath: 
      test-targets-always-run:

      files-to-download:
        - .*\.png$
      # iOS flank
      test-targets:

RunTests
  Uploading MyTests.zip ..
Found xctest: /Users/x/Desktop/Debug-iphoneos/xxxUITests-Runner.app/PlugIns/xxxUITests.xctest
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while running command (ftl.cli.firebase.test.ios.IosRunCommand@2d3ef181): java.lang.RuntimeException: File /Users/x/Desktop/Debug-iphoneos/xxxUITests-Runner.app/PlugIns/xxxUITests.xctest/xxxUITests does not exist!
	at picocli.CommandLine.execute(CommandLine.java:1168)
	at picocli.CommandLine.access$800(CommandLine.java:141)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
	at picocli.CommandLine.run(CommandLine.java:1974)
	at picocli.CommandLine.run(CommandLine.java:1920)
	at ftl.Main$Companion.main(Main.kt:48)
	at ftl.Main.main(Main.kt)
Caused by: java.lang.RuntimeException: File /Users/x/Desktop/Debug-iphoneos/xxxUITests-Runner.app/PlugIns/xxxUITests.xctest/xxxUITests does not exist!
	at ftl.ios.Parse.validateFile(Parse.kt:22)
	at ftl.ios.Parse.parseObjcTests$test_runner(Parse.kt:36)
	at ftl.ios.Xctestrun.testsForTarget(Xctestrun.kt:36)
	at ftl.ios.Xctestrun.findTestNames(Xctestrun.kt:94)
	at ftl.ios.Xctestrun.findTestNames(Xctestrun.kt:78)
	at ftl.args.IosArgs$testShardChunks$2.invoke(IosArgs.kt:63)
	at ftl.args.IosArgs$testShardChunks$2.invoke(IosArgs.kt:26)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at ftl.args.IosArgs.getTestShardChunks(IosArgs.kt)
	at ftl.run.IosTestRunner$runTests$2.invokeSuspend(IosTestRunner.kt:38)
	at ftl.run.IosTestRunner$runTests$2.invoke(IosTestRunner.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:186)
	at ftl.run.IosTestRunner.runTests(IosTestRunner.kt:23)
	at ftl.run.TestRunner.runTests(TestRunner.kt:53)
	at ftl.run.TestRunner.newRun(TestRunner.kt:350)
	at ftl.cli.firebase.test.ios.IosRunCommand$run$1.invokeSuspend(IosRunCommand.kt:32)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.kt:116)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:76)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:53)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:35)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at ftl.cli.firebase.test.ios.IosRunCommand.run(IosRunCommand.kt:31)
	at picocli.CommandLine.execute(CommandLine.java:1160)
	... 9 more
x@x:~/ios/test_runner$
@gtroshin

This comment has been minimized.

Copy link
Author

gtroshin commented Jan 17, 2019

An answer was in .xctestrun description. Because __TESTROOT__ is directory containing the xctestrun file and flank goes and try to find that file there. But in a readme there is no information about it.

So basically the custom .xctestrun file should be in the unzipped folder. In my case in Xcode derived data folder + /Build/Products.

http://www.manpagez.com/man/5/xcodebuild.xctestrun/

x@x:~/xcuitest/Build/Products$ ls -la
total 16
drwxr-xr-x@  6 x  staff   192B Jan 17 10:06 ./
drwxr-xr-x   4 x  staff   128B Jan 16 14:37 ../
drwxr-xr-x  39 x  staff   1.2K Jan 16 14:39 Debug-iphoneos/
drwxr-xr-x   7 x  staff   224B Jan 16 14:37 Debug-watchos/
-rw-r--r--@  1 x  staff   3.3K Jan 17 10:06 EmailSignupMethodTest_xxxUITests_iphoneos12.1-arm64e.xctestrun
-rw-r--r--   1 x  staff   3.3K Jan 16 14:39 xxxUITests_iphoneos12.1-arm64e.xctestrun
@bootstraponline

This comment has been minimized.

Copy link
Contributor

bootstraponline commented Jan 17, 2019

I agree this is confusing. I'll work on improving the documentation.

The reason we look int he unzipped folder is processing the potentially very large zip file in code is error prone. In the typical use case, the user already has the necessary files on disk. It's faster to access the files directly than via the zip.

@bootstraponline bootstraponline changed the title Error while running command (ftl.cli.firebase.test.ios.IosRunCommand@2d3ef181): java.lang.RuntimeException Improve xctestrun documentation Jan 17, 2019

@gtroshin

This comment has been minimized.

Copy link
Author

gtroshin commented Jan 18, 2019

@bootstraponline

OK, I got it. Thanks! Basically I have to remove the default generated .xctestrun and put the custom one the the .zip and to the /Build/Products as well to make it works properly?

UPD: I found that use test-targets is more easier option in case of iOS tests than having custom .xctestrun files.

@bootstraponline

This comment has been minimized.

Copy link
Contributor

bootstraponline commented Jan 18, 2019

here's the way I use Flank on iOS

#!/bin/bash

set -euxo pipefail

if ! [ -x "$(command -v xcpretty)" ]; then
  gem install xcpretty
fi

DD="dd_tmp"
SCHEME="StudentUITests"
ZIP="ios_student_earlgrey.zip"

rm -rf "$DD"

xcodebuild build-for-testing \
  -workspace ../../Canvas.xcworkspace \
  -scheme "$SCHEME" \
  -derivedDataPath "$DD" \
  -sdk iphoneos \
  | xcpretty

pushd "$DD/Build/Products"
zip -r "$ZIP" *-iphoneos *.xctestrun
popd
mv "$DD/Build/Products/$ZIP" .
  test: ./ios_student_earlgrey.zip
  xctestrun-file: ./dd_tmp/Build/Products/StudentUITests_iphoneos12.0-arm64e.xctestrun

and yes test-targets is the best way to specify what tests to run. Ideally you shouldn't have to modify your .xctestrun file at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment