30s delay on .js asset request using testcafe proxy with localhost and origin hosted behind Cloudflare proxy (cache disabled)

[awinecki]

What is your Test Scenario?

We have an existing testcafe E2E test suite. Recently, we've routed our app's API through a Cloudflare proxy (without cache). This resulted in a peculiar behaviour; when running our E2E tests, one particular JS asset request is slowed down by precisely 30 seconds before finally loading (it always loads, but slowly). As we have many individual tests doing this operation, this slows down our test suite run time by 40-50% overall, which is not acceptable.

Ideally, we'd like to keep our E2E run time the same, but benefit from testing the staging environment with the CF proxy enabled (stage/prod env parity) to catch any potential proxy-related regressions.

What is the Current behavior?

Running yarn run testcafe chrome test.ts --hostname localhost loads the /sign_in page, but in the Chrome's network tab, the zpkj-app.24804fb7a7cae8e.js asset hangs when loading (about 30s every time), then loads.

Repo with test.ts: https://github.com/packhelp/e2e-lab-cf-debug.

What is the Expected behavior?

Run yarn run testcafe chrome test.ts --hostname localhost without the 30-second delay on the JS asset load.

The desired behaviour is exactly what happens when running yarn run testcafe chrome test.ts.

We'd like to keep using the --hostname localhost flag but also avoid the 30-second delay.

Without the --hostname localhost, we cannot use retryTestPages flag which causes some of our tests to fail. Additionally, we're running our testcafe suite using Google Cloud Functions, which run fine with --hostname localhost, but return intermittent errors without this setting: .

Repo with test.ts: https://github.com/packhelp/e2e-lab-cf-debug.

What is your web application and your TestCafe test code?

Repo: https://github.com/packhelp/e2e-lab-cf-debug.
Test code: https://github.com/packhelp/e2e-lab-cf-debug/blob/main/test.ts
Running the test:

./cmd-ok.sh # control
./cmd-fail.sh # reproduce the fail

Your website URL (or attach your complete example):

Site: https://app.stagingqa.dev.packhelp.com/sign_in
Username: testcafe
Password: sent via email to support@devexpress.com

Your complete test code (or attach your test files):

Code: https://github.com/packhelp/e2e-lab-cf-debug/blob/main/test.ts

Base test code:

import { Selector } from 'testcafe';

fixture `Packhelp login`
  .httpAuth({ username: 'testcafe', password: '**************REDACTED**************' })

test('Login sample test', async t => {
  await t
    .navigateTo('https://app.stagingqa.dev.packhelp.com/sign_in');
  await t
    .debug()
    .expect(Selector(`[e2e-target-name="confirm"]`).innerText)
    .eql('Sign in')
});

Expanded test code with RequestLogger and Hooks for HTTP headers debugging:

import { Selector, RequestHook, RequestLogger } from 'testcafe';

const logger = RequestLogger(
  /.*zpkj-app.*/,
  { logRequestHeaders: true }
);

class UpdateZpkjAppHeaders extends RequestHook {
  constructor () {
    super();
  }

  async onRequest (event) {
    // e.requestOptions.headers['Authorization'] = 'generate token here';
    if (event.requestOptions.url.match('zpkj-app')) {
      console.log(event.requestOptions.headers);
      console.log(event.requestOptions);
      event.requestOptions.headers = Object.assign({}, event.requestOptions.headers, {
        "sec-ch-ua": null,
        "sec-ch-ua-mobile": null,
        "sec-Fetch-Site": null,
        "sec-Fetch-Mode": null,
        "sec-Fetch-Dest": null
      })
      event.requestOptions.headers['MY-CYSTOM-HEADER'] = 'secret';
      console.log(event.requestOptions.headers);
    }
  }

  async onResponse (e) {
  }
}

fixture `Packhelp login`
  // .page `https://app.stagingqa.dev.packhelp.com/sign_in`
  // .page(`https://stagingqa.dev.packhelp.com`)
  .httpAuth({ username: 'testcafe', password: '**************REDACTED**************' })
  // .requestHooks(new UpdateZpkjAppHeaders());

test('Login sample test', async t => {
  await t
    .addRequestHooks([new UpdateZpkjAppHeaders(), logger])
    .navigateTo('https://app.stagingqa.dev.packhelp.com/sign_in');
  await t
    // .debug()
    .expect(Selector(`[e2e-target-name="confirm"]`).innerText)
    .eql('Sign in')

  console.log(logger.requests);
  console.log(logger.requests[0].request.headers);
});

Your complete configuration file (if any):

Config as per repo https://github.com/packhelp/e2e-lab-cf-debug.

Your complete test report:

Not applicable.

Screenshots:

Bad behaviour:

Good behaviour:

Steps to Reproduce:

1. Clone the repo https://github.com/packhelp/e2e-lab-cf-debug
2. Run yarn install
3. Run ./cmd-fail.sh
4. Open up Network tab in Chrome and check out zpkj-app.24804fb7a7cae8e.js asset load time and HTTP headers

Your Environment details:

• testcafe version: 1.14.2
• node.js version: v16.2.0
• command-line arguments: --hostname localhost
• browser name and version: Chrome Version 91.0.4472.114
• platform and version: macOS Big Sur 11.3, macbook M1 (although a colleague tested this on Windows and the same thing happens; not likely to be an OS thing)
• additional notes:

The Cloudflare proxy is configured to act as an intermediary between https://app.stagingqa.dev.packhelp.com and the real origin server. The proxy does not cache anything. It's just a base proxy to utilise Cloudflare antiDDoS defenses. Disabling the proxy removes the 30-second slowdown (we can do it upon request for debugging). Now, I'm aware that this seems like a Cloudflare issue. They have their own rules to block malicious traffic. But what's weird is that they block just the 1st request. If you run the above scripts and then curl the same request (attached in the GitHub repo), it works OK. It seems unlikely to me that Cloudflare would "slow down" just a 1st request, the pass the subsequent ones, but it's possible.

We've noticed that adding --hostname localhost adds quite a lot of security related HTTP headers to each request. We've tried to override these with a RequestHook but failed (see the test.ts code for this).

At this point, we've spent a lot of time debugging this and it seems:

• We can't enable the proxy on E2E-tested environments because of the delay
• We can't change the --hostname localhost flag because the tests break in GCP env

CC: @pmirecki

[AlexKamaev]

Thank you for reporting this.
We got the email with the password at support@devexpress.com.
However, our policy prevents us from accessing internal resources without prior written approval from a site owner. If you want us to further research the problem directly on your side, please ask the website owner to send us (support@devexpress.com) a written confirmation. It must permit DevExpress personnel to remotely access the website and its internal resources for research/testing/and debugging purposes.

[no-response]

This issue was automatically closed because there was no response to our request for more information from the original author. Currently, we don't have enough information to take action. Please reach out to us if you find the necessary information and are able to share it. We are also eager to know if you resolved the issue on your own and can share your findings with everyone.

[awinecki]

Thank you @AlexKamaev. Sorry for the delay, I've been off the grid. The issue is still as relevant as it was to us. I've just sent an email with the written permission to access our servers. Please let me know if you need anything more. We'd appreciate any help or hints on how to debug this further.

[viktoria2506]

@awinecki

Hello,
Thank you for the information. I reproduced the issue. We need some time to research it. Once we get any results, we will update this thread.

[Evilweed]

@viktoria2506 does this mean you don't need our server with repro anymore?

[viktoria2506]

We need a sample to reproduce the issue and access to your server so that we can fix it.

[miherlosev]

Hi @Evilweed

I cannot log in to the tested site using the emailed credentials. Please fix the site or send us valid credentials one more time.

[Evilweed]

@miherlosev we have switched from TestCafe to Playwright because TestCafe became too unstable for us to use. I cannot provide access to the dev server anymore.

[miherlosev]

I see. Thank you for informing us of your decision. It seems that the issue is not actual now, so I will close it.