Testing
This page covers the UI tests for the app, they're based on the official Flutter integration test concepts. The flutter_driver library provides an API to test Flutter applications on real devices and emulators.
A big thank for your contribution and for reading. Feel free to make any suggestions.
All tests are located in two packages outside of the production code and one file within: ox-coi/test_driver, ox-coi/tools and keyMapping.
-
test_driver contains the logs folder, the setup folder, and all integration tests. The setup folder contains additional files which provide methods for test related helpers to run before, during, and after the actual tests. A very important aspect is that only test files (e.g. login_test.dart) and other folders (e.g. setup) are allowed be in the test_driver folder. This requirement originates from the shell script used for easier execution, see below. Other folders like setup can contain any type of file.
-
Tools contains mostly shell scripts to simplify common tasks, but only the shell scripts for single and complete integration tests are important to run the tests. Those scripts allow the fast and consistent execution of all / single tests. They will be also used on CI systems.
-
keyMapping.dart contains keys which the driver needs to find UI elements. The key mapping should be the only place where tests directly interfere with production code.
The way to run the tests is the same for all operating systems, where the project is already configured. To prepare the operating systems to build and start the app, please see here. A general guide on how to run Flutter integration tests with the flutter_driver can be found on the flutter official website.
The Android Debug Bridge (adb) is only required if you want to run test for the Android Platform and the applesimutils are required for the iOS platform. Both will be used to grant permission on the Android emulator, or on the iOS simulator respectively, before flutter_drive starts the tests. Adb is included in Android SDK Platform-Tools (installed during Android Studio Setup), so it can be reused if already installed and up to date. Contrary to adb, applesimutils has to be installed separately for iOS devices.
Even more important before the tests are performed is the access data for the servers to be reached (e.g. debug (mobile-qa)) and the login information. This information is contained in the credential.json, which for security reasons had to be added locally in the project in the folder test_driver/setup.
- Open a terminal and navigate to the ox-coi/tools folder in the project
-
Execute a single test
/test.runSingleIntegrationTest.sh (android, ios) (Emulator / device id) (App id / bundle identifier) (test_file.dart) (id [coi_debug_beta, coi_debug_mobile_qa]) -
Execute the full test suite
/test.runIntegrationTests.sh (android, ios) (Emulator / device id) (App id / bundle identifier) (id [coi_debug_beta, coi_debug_mobile_qa]) - The server to reach have to be given as string
1. parameter: Target [android, ios] specifies the Platform for the test.
2. parameter: Emulator / device id [One of the devices listed when executing 'flutter devices'.
Use one of the entries from the second column.]
3. parameter: App id / bundle identifier
4. parameter: Provider id [coi_debug_beta, coi_debug_mobile_qa]
5. parameter: Test which should get executed [All files in the test_driver folder are usable]
- Run the login test (login_test.dart)
/test.runSingleIntegrationTest.sh android emulator-5554 com.openxchange.oxcoi.dev login_test.dart coi_debug_beta
- Run all test (executes every .dart file in the test_driver folder)
/test.runIntegrationTests.sh android emulator-5554 com.openxchange.oxcoi.dev coi_debug_beta
This will run all integration tests for the coi_debug_beta server one by one. A log file will be available in /test_driver/logs.
Writing and running tests consists of 5 steps, but initially we have to import the flutter_driver dependency.
- Create the instrumented app
- Perform the setup (e.g. grant all permissions, define and connect the driver, etc.)
- Create SerializableFinders to locate specific widgets
- Test the scenario you want
- Disconnect the flutter_driver
The instrumented app has two general tasks: enable the flutter driver extensions and run the app which we want to test. Running the commands below, from the root of the project, will execute a test. The following actions are performed:
- Builds the --target app.dart for the given device
- Installs it on the emulator or simulator
- Starts the app
- Runs the test for test_files.dart
flutter drive -d ${deviceId} --target=test_driver/setup/app.dart --driver=${test_file.dart}
Grant all permissions in before testing, to get access to all functions you need to test: like microphone, calendar, camera, etc, as the flutter_driver has only access to the app and not to the system itself.
There is a fundamental difference between adb and applesimutils. While adb does not require a restart of the emulator when setting permissions on the Android Emulator, the simulator must be restarted after setting the permissions with applesimutils on the iOS Simulator. After restarting the device flutter_driver loses connection to the app and all to be executed tests will fail. This is the reason why the iOS permissions are granted in the shell script, each time before execution of a test. The app is uninstalled at the end of every test from the iOS device.
Dart input and output library.
import 'dart:io';
Flutter driver and test library.
import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';
Set the adb path (normally not needed, as adb should be on the PATH environment variable).
const String adbPath = 'adb';
Grant all required permissions.
const permissionAudio = 'android.permission.RECORD_AUDIO';
const permissionReadStorage = 'android.permission.READ_EXTERNAL_STORAGE';
...
setUpAll(() async {
await grantPermission(adbPath, permissionAudio);
await grantPermission(adbPath, permissionReadStorage);
}
As the iOS permission handling is done via the shell scripts (test.runIntegrationTests.sh and test.runSingleIntegrationTest.sh) no further iOS specific steps are required. The shell scripts will perform the following action:
applesimutils --byId {deviceId} --bundle {appId} --setPermissions "camera=YES, contacts=YES, calendar=YES, photos=YES, speech=YES, microphone=YES, medialibrary=YES, notifications=YES, faceid=YES, homekit=YES, location=always, reminders=unset, motion=YES"
Define the flutter_driver and connect to it.
FlutterDriver _driver;
setUpAll(() async {
_driver = await FlutterDriver.connect();
});
Close the connection to the driver after tests are completed.
tearDownAll(() async {
if (_driver != null) {
_driver.close();
}
});
The complete code is available here: main_test_setup.dart
Flutter_driver uses SerializableFinders to get access to widgets. Using keys to find widgets is one of the best ways to test particular widgets from the UI. The keys have to be defined and set in production code. It's important that keys are pretty important for the Flutter framework during run time, as Flutter uses keys e.g. to determine if widgets can be reused. Keep this in mind and never replace production keys (e.g. automatically created and defined during run time with an ID or index).
key: Key(keyUserSettingsUserSettingsUsernameLabel)
key: Key(testUserNameUserProfile)
SerializableFinder userSettingsUsernameLabelFinder = find.byValueKey(keyUserSettingsUserSettingsUsernameLabel);
SerializableFinder userProfileUserNameTextFinder = find.text(testUserNameUserProfile);
One test file can contain multiple groups of tests, with multiple tests per group.
test('Test create profile integration tests', () async {
// write your test here.
});
group('Group more than one test together', () {
test('Test1', () async {
// write your test here.
});
test('Test2', () async {
// write your test here.
});
test('Test3', () async {
// write your test here.
});
});
For real world examples please see the test_driver folder.