Skip to content

Commit

Permalink
fix: stable iOS e2e tests (#392)
Browse files Browse the repository at this point in the history
## 📜 Description

Make iOS tests on CI stable again.

## 💡 Motivation and Context

Looks like after updating some of runners to XCode 15.3 booting up a
simulator takes long time (actually it boots pretty fast, but after that
CPU usage is close to 100% for quite big period of time). As a result
tests are failing because of timeout.

I've tried many things - switching to other XCode versions, increasing
timeout, changing the order of tests, changing runners, pre-booting
simultator etc. and still tests very flaky in some cases.

However what I discovered is that after ~20mins and after several
failures tests start to work okay. So I decided to add `--retry 2` flag
to overcome the problem of flaky simulator - it makes CI green again.
But if a regression really introduced - it'll fail anyway just with
linger time of execution 🙂

## 📢 Changelog

### E2E

- added `patch-package`;
- updated `detox`;
- updated assets;
- moved tests preparation in `jest-setup.ts` file;
- run tests with retries;
- use scroll instead of swipe in `AwareScrollView` test;

### CI

- increased `timeout` (60 -> 90);

## 🤔 How Has This Been Tested?

Tested on CI.

## 📝 Checklist

- [x] CI successfully passed
- [x] I added new mocks and corresponding unit-tests if library API was
changed
  • Loading branch information
kirillzyusko committed Apr 2, 2024
1 parent 6348481 commit fdd7f84
Show file tree
Hide file tree
Showing 14 changed files with 233 additions and 42 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ios-e2e-test.yml
Expand Up @@ -25,7 +25,7 @@ jobs:
test:
name: ⚙️ Automated test cases
runs-on: macos-14
timeout-minutes: 60
timeout-minutes: 90
env:
WORKING_DIRECTORY: example
concurrency:
Expand Down
6 changes: 0 additions & 6 deletions e2e/kit/001-keyboard-animation.e2e.ts
Expand Up @@ -5,14 +5,8 @@ import {
waitForElementById,
waitForExpect,
} from "./helpers";
import setDemoMode from "./utils/setDemoMode";

describe("Example", () => {
beforeAll(async () => {
await setDemoMode();
await device.launchApp();
});

it("should navigate to `Animated transition` screen", async () => {
await waitAndTap("animated_transition");
await waitForElementById("keyboard_animation_text_input");
Expand Down
Expand Up @@ -6,14 +6,8 @@ import {
waitForElementById,
waitForExpect,
} from "./helpers";
import setDemoMode from "./utils/setDemoMode";

describe("Example", () => {
beforeAll(async () => {
await setDemoMode();
await device.launchApp();
});

it("should navigate to `Enabled/disabled` screen", async () => {
await scrollDownUntilElementIsVisible(
"main_scroll_view",
Expand Down
Expand Up @@ -7,14 +7,8 @@ import {
waitAndType,
waitForElementById,
} from "./helpers";
import setDemoMode from "./utils/setDemoMode";

describe("`onTextChange` functionality", () => {
beforeAll(async () => {
await setDemoMode();
await device.launchApp();
});

it("should fire `onTextChange` with correct text", async () => {
await scrollDownUntilElementIsVisible(
"main_scroll_view",
Expand Down
Expand Up @@ -7,7 +7,6 @@ import {
waitAndType,
waitForExpect,
} from "./helpers";
import setDemoMode from "./utils/setDemoMode";

const BLINKING_CURSOR = 0.35;

Expand All @@ -17,11 +16,6 @@ const closeKeyboard = async () => {
};

describe("AwareScrollView test cases", () => {
beforeAll(async () => {
await setDemoMode();
await device.launchApp();
});

it("should push input above keyboard on focus", async () => {
await waitAndTap("aware_scroll_view");
await waitAndTap("TextInput#3");
Expand Down Expand Up @@ -55,7 +49,7 @@ describe("AwareScrollView test cases", () => {
});

it("should auto-scroll when user types a text", async () => {
await element(by.id("TextInput#4")).swipe("down", "slow", 0.15);
await element(by.id("aware_scroll_view_container")).scroll(80, "up");
await typeText("TextInput#4", "1");
await waitForExpect(async () => {
await expectBitmapsToBeEqual(
Expand Down
6 changes: 0 additions & 6 deletions e2e/kit/005-keyboard-toolbar.e2e.ts
Expand Up @@ -11,14 +11,8 @@ import {
waitForElementById,
waitForExpect,
} from "./helpers";
import setDemoMode from "./utils/setDemoMode";

describe("`KeyboardToolbar` specification", () => {
beforeAll(async () => {
await setDemoMode();
await device.launchApp();
});

it("should navigate to `Toolbar` screen", async () => {
await scrollDownUntilElementIsVisible("main_scroll_view", "toolbar");
await waitAndTap("toolbar");
Expand Down
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions e2e/kit/jest.config.js
Expand Up @@ -8,5 +8,6 @@ module.exports = {
globalTeardown: "detox/runners/jest/globalTeardown",
reporters: ["detox/runners/jest/reporter"],
testEnvironment: "detox/runners/jest/testEnvironment",
setupFilesAfterEnv: ["<rootDir>/setup-jest.ts"],
verbose: true,
};
10 changes: 10 additions & 0 deletions e2e/kit/setup-jest.ts
@@ -0,0 +1,10 @@
import setDemoMode from "./utils/setDemoMode";

beforeAll(async () => {
await setDemoMode();
await device.launchApp();
});

afterAll(async () => {
await device.terminateApp();
});
10 changes: 6 additions & 4 deletions e2e/package.json
Expand Up @@ -5,11 +5,12 @@
"main": "index.js",
"scripts": {
"build-example:android": "detox build --configuration example.android.emu.release",
"test-example:android": "detox test --configuration example.android.emu.release --loglevel verbose --take-screenshots failing --record-videos failing",
"test-example:android": "detox test --configuration example.android.emu.release --loglevel verbose --take-screenshots failing --record-videos failing --retries 2",
"build-example:ios": "detox build --configuration example.ios.sim.release",
"test-example:ios": "detox test --configuration example.ios.sim.release --loglevel verbose --take-screenshots failing --record-videos failing",
"test-example:ios": "detox test --configuration example.ios.sim.release --loglevel verbose --take-screenshots failing --record-videos failing --retries 2",
"test": "echo \"Error: no test specified\" && exit 1",
"detox-clean": "detox clean-framework-cache && detox build-framework-cache"
"detox-clean": "detox clean-framework-cache && detox build-framework-cache",
"postinstall": "patch-package"
},
"repository": {
"type": "git",
Expand All @@ -30,8 +31,9 @@
"@types/pngjs": "^6.0.1",
"async-retry": "^1.3.3",
"colors": "^1.4.0",
"detox": "^20.19.0",
"detox": "^20.19.3",
"jest": "^29",
"patch-package": "^8.0.0",
"pixelmatch": "^5.3.0",
"pngjs": "^7.0.0",
"ts-jest": "^29.1.1"
Expand Down

0 comments on commit fdd7f84

Please sign in to comment.