How to run patrol with GitLab CI/CD?

[haomun0731]

I can run the patrol test locally and successfully pass all tests without any problems, but encounter errors when it comes to GitLab.

My gitlab-ci.yml:

variables:
  FLUTTER_VERSION: "3.7.12"
  ENVIRONMENT: "dev"
  FLAVOR: "dev"
  MATCH_TYPE: "adhoc"
  BUILD_PLAN: # this is useful for setting multiple variables with one variable value, easier when running pipeline manually
    value: ""
    description: "The build plan to execute (dev/test/prod), will override the ENVIRONMENT and FLAVOR variables, leave empty for default"

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE != 'merge_request_event' # by default gitlab create a pipeline when a merge request is created

.setup:
  # Set variables based on BUILD_PLAN
  before_script:
    - |
      if [ "$BUILD_PLAN" = "dev" ]; then
        ENVIRONMENT="dev"
        FLAVOR="dev"
      fi
    - |
      if [ "$BUILD_PLAN" = "test" ]; then
        ENVIRONMENT="test"
        FLAVOR="qa"
      fi
    - |
      if [ "$BUILD_PLAN" = "prod" ]; then
        ENVIRONMENT="prod"
        FLAVOR="prod"
      fi

include: # https://docs.gitlab.com/ee/ci/yaml/includes.html
  - project: my/gitlab/pipeline-templates
    file:
      - gradle/gradle-settings.yml
      - git/git-pull.yml

default:
  tags:
    - linux
    - docker

stages:
  - test

.generate-local-model:
  before_script:
    - flutter clean
    - flutter pub get
    - echo "Generating local model..."
    - flutter pub run build_runner build --delete-conflicting-outputs

generate-openapi:
  image: "ghcr.io/cirruslabs/flutter:${FLUTTER_VERSION}"
  stage: .pre
  extends:
    - .gradle_settings
  script:
    - echo "Generating openapi..."
    - ./gradlew generateApiModel
  artifacts:
    expire_in: 1 day
    paths:
      - packages/api/

download-secure-files:
  stage: .pre
  extends:
    - .gradle_settings
  variables:
    SECURE_FILES_DOWNLOAD_PATH: "./secure_files"
  script:
    - echo "Downloading secure files"
    - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
  artifacts:
    expire_in: 1 day
    paths:
      - secure_files/

integration-android:
  image: "ghcr.io/cirruslabs/flutter:${FLUTTER_VERSION}"
  extends:
    - .gradle_settings
  stage: test
  needs:
    - generate-openapi
  variables:
    KUBERNETES_MEMORY_REQUEST: 14G
    KUBERNETES_MEMORY_LIMIT: 16G
    KUBERNETES_HELPER_MEMORY_REQUEST: 2G
    KUBERNETES_HELPER_MEMORY_LIMIT: 10G
    KUBERNETES_EPHEMERAL_STORAGE_REQUEST: 20G
    KUBERNETES_EPHEMERAL_STORAGE_LIMIT: 50G
    KUBERNETES_HELPER_EPHEMERAL_STORAGE_REQUEST: 20G
    KUBERNETES_HELPER_EPHEMERAL_STORAGE_LIMIT: 50G
    KUBERNETES_CPU_REQUEST: 1
    KUBERNETES_CPU_LIMIT: 20
    KUBERNETES_HELPER_CPU_REQUEST: 1
    KUBERNETES_HELPER_CPU_LIMIT: 20
    apiLevel: "27"
    target: "google_apis"
    arch: "x86"
    deviceName: "pixel_api27_google_apis_emulator"
    deviceType: "pixel"
  before_script:
    - !reference [.gradle_settings, before_script]
    - !reference [.setup, before_script]
    - !reference [.generate-local-model, before_script]
    - export PATH="$PATH:$HOME/.pub-cache/bin"
    - dart pub global activate patrol_cli 2.1.5 --overwrite
  script:
    - time sdkmanager "platform-tools" "platforms;android-${apiLevel}"
    - time sdkmanager --install "system-images;android-${apiLevel};${target};${arch}"
    - time sdkmanager --update
    - echo "y" | sdkmanager --licenses
    - sdkmanager --list
    - echo "no" | avdmanager -v create avd --force --name "${deviceName}" --package "system-images;android-${apiLevel};${target};${arch}" --tag "${target}" --sdcard 128M --device "${deviceType}"
    - time emulator -avd "${deviceName}" -no-audio -no-window -no-snapstorage -no-snapshot -gpu swiftshader_indirect -no-boot-anim -no-accel -wipe-data -skip-adb-auth &
    - time adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done; input keyevent 82'
    - flutter doctor -v
    - patrol --version
    - java --version
    - ./gradlew --version
    - patrol test \
      --dart-define="devApiBasePath=$devApiBasePath" \
      --dart-define="devBasicAuthUsername=$devBasicAuthUsername" \
      --dart-define="devBasicAuthPassword=$devBasicAuthPassword" \
      --flavor dev \
      --target integration_test/e2e_login_test.dart --verbose

The logs from GitLab:

patrol_cli v2.1.5
�[32;1m$ java --version�[0;m
openjdk 11.0.17 2022-10-18
OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu222.04)
OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu222.04, mixed mode, sharing)
�[32;1m$ ./gradlew --version�[0;m
Downloading https://services.gradle.org/distributions/gradle-7.6-all.zip
...............
10%................20%................30%................40%...............50%................60%................70%...
.............80%...............90%................100%

Welcome to Gradle 7.6!

Here are the highlights of this release:
 - Added support for Java 19.
 - Introduced `--rerun` flag for individual task rerun.
 - Improved dependency block for test suites to be strongly typed.
 - Added a pluggable system for Java toolchains provisioning.

For more details see https://docs.gradle.org/7.6/release-notes.html


------------------------------------------------------------
Gradle 7.6
------------------------------------------------------------

Build time:   2022-11-25 13:35:10 UTC
Revision:     daece9dbc5b79370cc8e4fd6fe4b2cd400e150a8

Kotlin:       1.7.10
Groovy:       3.0.13
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          11.0.17 (Ubuntu 11.0.17+8-post-Ubuntu-1ubuntu222.04)
OS:           Linux 5.10.176+ amd64

�[32;1m$ patrol test \ --dart-define="devApiBasePath=$devApiBasePath" \ --dart-define="devBasicAuthUsername=$devBasicAuthUsername" \ --dart-define="devBasicAuthPassword=$devBasicAuthPassword" \ --flavor dev \ --target integration_test/e2e_login_test.dart --verbose�[0;m
In next major release, patrolTest method will be intended for UI tests.
If you want to use Patrol in your widget tests, use patrol_finders package.

For more information, see https://patrol.leancode.co/patrol-finders-release
Disable this warning by setting the PATROL_FINDERS environment variable.
      
Verbose mode enabled. More logs will be printed.
Received 2 test target(s)
Received test target: /builds/my/mobile/integration_test/e2e_login_test.dart
Received test target: /builds/my/mobile/integration_test/e2e_logout_test.dart
Generated entrypoint /builds/my/mobile/integration_test/test_bundle.dart with 2 bundled test(s)
$ flutter --no-version-check devices --machine
No device specified, using the first one (emulator-5554)
Received 1 device(s) to run on
Received device: emulator-5554
Received 7 --dart-define(s) (0 custom, 7 internal)
Received internal --dart-define: PATROL_WAIT=0
Received internal --dart-define: PATROL_APP_PACKAGE_NAME=net.my.dev
Received internal --dart-define: PATROL_APP_BUNDLE_ID=net.my.dev
Received internal --dart-define: PATROL_ANDROID_APP_NAME=mobile
Received internal --dart-define: PATROL_IOS_APP_NAME=mobile
Received internal --dart-define: INTEGRATION_TEST_SHOULD_REPORT_RESULTS_TO_NATIVE=false
Received internal --dart-define: PATROL_TEST_LABEL_ENABLED=true
• Building apk with entrypoint test_bundle.dart...
$ ./gradlew :app:assembleDebug -Ptarget=(#my details)
✗ Failed to build apk with entrypoint test_bundle.dart (Gradle build failed with code 127) (7ms)
Exception: Gradle build failed with code 127
#0      AndroidTestBackend.build.<anonymous closure> (package:patrol_cli/src/android/android_test_backend.dart:67:9)
<asynchronous suspension>
#1      DisposeScope.run (package:dispose_scope/src/dispose_scope.dart:46:7)
<asynchronous suspension>
#2      AndroidTestBackend.build (package:patrol_cli/src/android/android_test_backend.dart:42:5)
<asynchronous suspension>
#3      TestCommand._build (package:patrol_cli/src/commands/test.dart:234:7)
<asynchronous suspension>
#4      TestCommand.run (package:patrol_cli/src/commands/test.dart:167:5)
<asynchronous suspension>
#5      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#6      PatrolCommandRunner.runCommand (package:patrol_cli/src/runner/patrol_command_runner.dart:321:18)
<asynchronous suspension>
#7      PatrolCommandRunner.run (package:patrol_cli/src/runner/patrol_command_runner.dart:265:18)
<asynchronous suspension>
#8      patrolCommandRunner (package:patrol_cli/src/runner/patrol_command_runner.dart:77:20)
<asynchronous suspension>
#9      main (file:///root/.pub-cache/hosted/pub.dev/patrol_cli-2.1.5/bin/main.dart:6:20)
<asynchronous suspension>

See the logs above to learn what happened. Also consider running with --verbose. If the logs still aren't useful, then it's a bug - please report it.
Exception: Gradle build failed with code 127
#0      AndroidTestBackend.build.<anonymous closure> (package:patrol_cli/src/android/android_test_backend.dart:67:9)
<asynchronous suspension>
#1      DisposeScope.run (package:dispose_scope/src/dispose_scope.dart:46:7)
<asynchronous suspension>
#2      AndroidTestBackend.build (package:patrol_cli/src/android/android_test_backend.dart:42:5)
<asynchronous suspension>
#3      TestCommand._build (package:patrol_cli/src/commands/test.dart:234:7)
<asynchronous suspension>
#4      TestCommand.run (package:patrol_cli/src/commands/test.dart:167:5)
<asynchronous suspension>
#5      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#6      PatrolCommandRunner.runCommand (package:patrol_cli/src/runner/patrol_command_runner.dart:321:18)
<asynchronous suspension>
#7      PatrolCommandRunner.run (package:patrol_cli/src/runner/patrol_command_runner.dart:265:18)
<asynchronous suspension>
#8      patrolCommandRunner (package:patrol_cli/src/runner/patrol_command_runner.dart:77:20)
<asynchronous suspension>
#9      main (file:///root/.pub-cache/hosted/pub.dev/patrol_cli-2.1.5/bin/main.dart:6:20)
<asynchronous suspension>

Any ideas on how to solve this issue would be greatly appreciated. 🙏

[haomun0731]

I fixed the issue by building the APK before the patrol build command: https://stackoverflow.com/a/76572343

[adrian-moisa]

@haomun0731 Have you ever had issues with patrol not releasing the pipeline? It happens to me quite often that the pipeline is hanging despite clearly visible on VNC the tests are complete. I'm using waydroid to run the tests since adb is way to resource hungry for VPS with no GPU.

[haomun0731]

@adrian-moisa No, I haven't encountered such hanging issues. However, I did discover that ADB is extremely slow, so I switched to using Firebase Test Lab instead.

[adrian-moisa]

yes adb is teribly slow on a vps without GPU. I initially tried adb and I had a horrible experience. However I had better luck with waydroid. It's a simulator, meaning it runs directly on debian. far better perf. Still 70% of my mac m1 but way better than adb. I'm currently having this issue with the hanging pipeline. Otherwise it would be quite a nice setup. I'm planning to have thousands of tests per day and for me the bill from such a service will become significant.