Improve xctestrun documentation

[gtroshin]

my .zip file and .xctestrun are in different folders.

Seems like flank is trying to find xctest in a wrong directory (related where custom .xctestrun is).

java -version
java version "11.0.2" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.2+7-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.2+7-LTS, mixed mode)

java -jar ./flank.jar -v
v4.2.0
5148c339e697778a3714bfb1aa6d4a9a120c5164

xxx@xxx-27:~/ios/test_runner$ ls -la
total 49168
drwxr-xr-x   4 x  staff   128B Jan 17 09:37 ./
drwxr-xr-x@ 53 x  staff   1.7K Jan 17 09:03 ../
-rw-r--r--   1 x  staff   1.5K Jan 17 09:37 flank.ios.yml
-rw-r--r--   1 x  staff    24M Jan 11 23:27 flank.jar
x@xxx:~/ios/test_runner$ java -jar ./flank.jar firebase test ios run
IosArgs
    gcloud:
      results-bucket: test-lab-xxx
      results-dir: null
      record-video: true
      timeout: 30m
      async: false
      project: xxx
      results-history-name: null
      # iOS gcloud
      test: /Users/x/MyTests.zip
      xctestrun-file: /Users/x/Desktop/EmailSignupMethodTest_XXXUITests_iphoneos12.1-arm64e.xctestrun
      xcode-version: 10.1
      device:
        - model: iphonex
          version: 12.0
          locale: en_US
          orientation: portrait
      flaky-test-attempts: 3

    flank:
      testShards: -1
      repeatTests: 1
      smartFlankGcsPath: 
      test-targets-always-run:

      files-to-download:
        - .*\.png$
      # iOS flank
      test-targets:

RunTests
  Uploading MyTests.zip ..
Found xctest: /Users/x/Desktop/Debug-iphoneos/xxxUITests-Runner.app/PlugIns/xxxUITests.xctest
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while running command (ftl.cli.firebase.test.ios.IosRunCommand@2d3ef181): java.lang.RuntimeException: File /Users/x/Desktop/Debug-iphoneos/xxxUITests-Runner.app/PlugIns/xxxUITests.xctest/xxxUITests does not exist!
	at picocli.CommandLine.execute(CommandLine.java:1168)
	at picocli.CommandLine.access$800(CommandLine.java:141)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
	at picocli.CommandLine.run(CommandLine.java:1974)
	at picocli.CommandLine.run(CommandLine.java:1920)
	at ftl.Main$Companion.main(Main.kt:48)
	at ftl.Main.main(Main.kt)
Caused by: java.lang.RuntimeException: File /Users/x/Desktop/Debug-iphoneos/xxxUITests-Runner.app/PlugIns/xxxUITests.xctest/xxxUITests does not exist!
	at ftl.ios.Parse.validateFile(Parse.kt:22)
	at ftl.ios.Parse.parseObjcTests$test_runner(Parse.kt:36)
	at ftl.ios.Xctestrun.testsForTarget(Xctestrun.kt:36)
	at ftl.ios.Xctestrun.findTestNames(Xctestrun.kt:94)
	at ftl.ios.Xctestrun.findTestNames(Xctestrun.kt:78)
	at ftl.args.IosArgs$testShardChunks$2.invoke(IosArgs.kt:63)
	at ftl.args.IosArgs$testShardChunks$2.invoke(IosArgs.kt:26)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at ftl.args.IosArgs.getTestShardChunks(IosArgs.kt)
	at ftl.run.IosTestRunner$runTests$2.invokeSuspend(IosTestRunner.kt:38)
	at ftl.run.IosTestRunner$runTests$2.invoke(IosTestRunner.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:186)
	at ftl.run.IosTestRunner.runTests(IosTestRunner.kt:23)
	at ftl.run.TestRunner.runTests(TestRunner.kt:53)
	at ftl.run.TestRunner.newRun(TestRunner.kt:350)
	at ftl.cli.firebase.test.ios.IosRunCommand$run$1.invokeSuspend(IosRunCommand.kt:32)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.kt:116)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:76)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:53)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:35)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at ftl.cli.firebase.test.ios.IosRunCommand.run(IosRunCommand.kt:31)
	at picocli.CommandLine.execute(CommandLine.java:1160)
	... 9 more
x@x:~/ios/test_runner$

[gtroshin]

An answer was in .xctestrun description. Because __TESTROOT__ is directory containing the xctestrun file and flank goes and try to find that file there. But in a readme there is no information about it.

So basically the custom .xctestrun file should be in the unzipped folder. In my case in Xcode derived data folder + /Build/Products.

http://www.manpagez.com/man/5/xcodebuild.xctestrun/

x@x:~/xcuitest/Build/Products$ ls -la
total 16
drwxr-xr-x@  6 x  staff   192B Jan 17 10:06 ./
drwxr-xr-x   4 x  staff   128B Jan 16 14:37 ../
drwxr-xr-x  39 x  staff   1.2K Jan 16 14:39 Debug-iphoneos/
drwxr-xr-x   7 x  staff   224B Jan 16 14:37 Debug-watchos/
-rw-r--r--@  1 x  staff   3.3K Jan 17 10:06 EmailSignupMethodTest_xxxUITests_iphoneos12.1-arm64e.xctestrun
-rw-r--r--   1 x  staff   3.3K Jan 16 14:39 xxxUITests_iphoneos12.1-arm64e.xctestrun

[bootstraponline]

I agree this is confusing. I'll work on improving the documentation.

The reason we look int he unzipped folder is processing the potentially very large zip file in code is error prone. In the typical use case, the user already has the necessary files on disk. It's faster to access the files directly than via the zip.

[gtroshin]

@bootstraponline

OK, I got it. Thanks! Basically I have to remove the default generated .xctestrun and put the custom one the the .zip and to the /Build/Products as well to make it works properly?

UPD: I found that use test-targets is more easier option in case of iOS tests than having custom .xctestrun files.

[bootstraponline]

here's the way I use Flank on iOS

#!/bin/bash

set -euxo pipefail

if ! [ -x "$(command -v xcpretty)" ]; then
  gem install xcpretty
fi

DD="dd_tmp"
SCHEME="StudentUITests"
ZIP="ios_student_earlgrey.zip"

rm -rf "$DD"

xcodebuild build-for-testing \
  -workspace ../../Canvas.xcworkspace \
  -scheme "$SCHEME" \
  -derivedDataPath "$DD" \
  -sdk iphoneos \
  | xcpretty

pushd "$DD/Build/Products"
zip -r "$ZIP" *-iphoneos *.xctestrun
popd
mv "$DD/Build/Products/$ZIP" .

  test: ./ios_student_earlgrey.zip
  xctestrun-file: ./dd_tmp/Build/Products/StudentUITests_iphoneos12.0-arm64e.xctestrun

and yes test-targets is the best way to specify what tests to run. Ideally you shouldn't have to modify your .xctestrun file at all.