Detox 21.0.1-rc.0 is very slow

[bkasten-rbi]

What happened?

I tested out the XCUITest based runner built into detox 21 today and the tests operate incredibly slowly compared to detox 20.

Is this expected to be the new norm with using XCUITest or do we suspect there are some performance issues with the new implementation that will be resolved?

What was the expected behaviour?

No response

Was it tested on latest Detox?

• I have tested this issue on the latest Detox release and it still reproduces.

Did your test throw out a timeout?

• I have followed the instructions under Identifying which synchronization mechanism causes us to wait too much.

Help us reproduce this issue!

No response

In what environment did this happen?

Detox version:
React Native version:
Has Fabric (React Native's new rendering system) enabled: (yes/no)
Node version:
Device model:
iOS version:
macOS version:
Xcode version:
Test-runner (select one): jest / other

Detox logs

Detox logs

paste logs here!

Device logs

Device logs

paste your device.log here!

More data, please!

No response

[d4vidi]

I believe it might be somewhat slower but not incredibly so. What's the scale in terms of a performance hit factor (2x, 10x, ...?)
@asafkorem

[d4vidi]

In any case, thanks a bunch for testing out the v21 RC!

[asafkorem]

@bkasten-rbi We greatly value your feedback, thank you!

It's true that we anticipated a slight dip in performance, but nothing too drastic. I'm currently working on plans to enhance the performance as well.

To help us better understand and address this issue, could you provide some additional details? Specifically, it would be beneficial to know the duration of your test runs before and after the update. Moreover, have you observed the slowdown to be test-specific or is it a general performance decrease across all types of tests?

[bkasten-rbi]

Hey, thanks for the kind and welcoming spirit!

So I just ran a version of our test that just runs our sign up flow (3 text inputs and a checkbox). On version 20, the test took 15s to fill out the inputs and hit the submit button. On version 21 the test took 85s. It seems to mostly be much much slower in how it handles the inputText command. I'll include two videos so you can see the difference (on a side note, I didn't capture the video to time it, but it felt like the startup time to get the app actually running in simulator felt slower in 21 as well)

v20	v21

[asafkorem]

@bkasten-rbi, I appreciate your detailed feedback!

I have released another RC revision with accelerated typing. I would be grateful if you could test this version (21.0.0-rc.2) 🙏🏼

[bkasten-rbi]

Hey @asafkorem - I think something went awry with this release. -rc.1 is slow, but works. In -rc.2 the test fails on the first invocation with this output:

Test Failed: XCUITest executor failed to handle request: No elements were found for the given matcher

[asafkorem]

@bkasten-rbi, what kind of invocation did it fail on?

[asafkorem]

I'm guessing based on the version differences, this may have occurred due to typing too quickly.. causing React Native to mishandle it. Have you noticed that the typing occurs, but some letters are missing?

[asafkorem]

@bkasten-rbi can you please check 21.0.0-rc.3? Typing should be slower but still quite fast on this version.

[bkasten-rbi]

I just tested and the same thing is happening. I don't believe the issue I'm running into is about the typing speed. It's failing on the very first assertion. It seems something changed in the synchronization and waiting of executing the first command. Here's the command that is failing for me:

Test Failed: XCUITest executor failed to handle request: No elements were found for the given matcher

    > 49 |   await element(by.id('app-header-signup-button')).tap();

[bkasten-rbi]

This is how my app operates in slight redaction:

beforeEach(async () => {
  await device.launchApp({
    delete: true,
    permissions: {
      userTracking: 'NO',
      location: 'inuse',
    },
    launchArgs: {
      EXKernelDisableNuxDefaultsKey: true,
      EXHomeIsNuxFinishedDefaultsKey: true,
      DETOX: true,
      EXDevMenuDisableAutoLaunch: true,
    },
  });

  await device.setURLBlacklist([
    'https://firebaselogging-pa.googleapis.com/v1/firelog/legacy/batchlog',
    'https://sdk.iad-05.braze.com/api/v3/data',
    'https://sdk.iad-05.braze.com/api/v3/content_cards/sync',
    'https://play.googleapis.com/log/batch',
    'https://www.googleapis.com/experimentsandconfigs/v1/getExperimentsAndConfigs',
  ]);

  await device.openURL({
    url: `exp+${
      global.__detox__scheme
    }://expo-development-client/?url=${getDevBundleUrl()}`,
  });

  await sleep(1000);
});
test('add-payment-method.e2e.yml', async () => {
  const outputs = {};

  await element(by.id('app-header-signup-button')).tap();
  
  // ...
});

[asafkorem]

Thanks for the response and the testing @bkasten-rbi!
That's very interesting.. since we only changed the typing speed. Does it work when you downgrade back to 21.0.1-rc.0?

[bkasten-rbi]

Okay, I just spent some time testing and I think i've got a better identification of the problems.

First is good news, all the versions do work and 21.0.0-rc.3 definitely has a healthy typing speed compared to 21.0.1-rc.0 and 21.0.0-rc.2 does seem too fast 🤣

So the problems I reported above with the error of

Test Failed: XCUITest executor failed to handle request: No elements were found for the given matcher

    > 49 |   await element(by.id('app-header-signup-button')).tap();

is still actually happening on the first run after installing the new version.

So effectively what happens is if I change from version 21.0.1-rc.0 to 21.0.0-rc.2 the first time I run my tests, it'll fail with that command. If I let it fail and run it again, then it operates the test properly. Is there some sort of caching mechanism that is happening that could be failing on the first run? It seems like maybe it's not waiting long enough on either the launchApp command or maybe the openLink on first run.

While that version change is the example i explained with, it happens in any variation where I change from a version X to either of the latest 2 rc's.

[asafkorem]

@bkasten-rbi thanks for this valuable information! This is very weird(!), I will try to reproduce & investigate that on a clean RN app. There isn't any caching operation by Detox (shouldn't be, at least), might be something of Xcode that we are not aware of.. 🤔
Anyway, I'm glad to hear that the typing speed is healthy with rc.3! 🙏🏼

[bkasten-rbi]

Sounds like we can close this issue as the "slow" nature of the issue has been resolved. I'll raise a new one later if I find the initial startup to keep crashing as more RC's come out

[asafkorem]

Sure, sounds good. Also, thanks for the feedback! It helped us, much appreciated!