[Lighthouse Flake]: No node found for selector: mat-option[ng-reflect-value="ADMIN"]

[U8NWXD]

Log:

Running Lighthouse 3 time(s) on http://127.0.0.1:8181/donate
Run #1...done.
Run #2...done.
Run #3...done.
Error: No node found for selector: mat-option[ng-reflect-value="ADMIN"]
    at assert (/home/runner/work/oppia/oppia/node_modules/puppeteer/lib/cjs/puppeteer/common/assert.js:26:15)
    at DOMWorld.click (/home/runner/work/oppia/oppia/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:287:32)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async setRole (/home/runner/work/oppia/oppia/puppeteer-login-script.js:103:5)
    at async module.exports (/home/runner/work/oppia/oppia/puppeteer-login-script.js:47:5)
    at async PuppeteerManager.invokePuppeteerScriptForUrl (/home/runner/work/oppia/oppia/node_modules/@lhci/cli/src/collect/puppeteer-manager.js:112:5)
    at async Object.runCommand (/home/runner/work/oppia/oppia/node_modules/@lhci/cli/src/collect/collect.js:244:7)
    at async run (/home/runner/work/oppia/oppia/node_modules/@lhci/cli/src/cli.js:103:7)
Running Lighthouse 3 time(s) on http://127.0.0.1:8181/emaildashboard

Examples:

• 2023-02-08: https://github.com/oppia/oppia/actions/runs/4121545830/jobs/7117336965#step:8:6781

• 2023-02-22: https://github.com/oppia/oppia/actions/runs/4246546386/jobs/7383494361

• 2023-02-28: https://github.com/oppia/oppia/actions/runs/4289833043/jobs/7473254839#step:8:6815

• 2023-03-10: https://github.com/oppia/oppia/actions/runs/4387678295/jobs/7683259972#step:8:7056

• 2023-03-11: https://github.com/oppia/oppia/actions/runs/4390071617/jobs/7688233476#step:8:7054

  Note that this log does not show the characteristic mat-option[ng-reflect-value="ADMIN"] error, but the uploaded lighthouse report does show the donorbox 403 error. In earlier examples only the donor page was affected, but now checks on all pages are failing, likely because of Fix #17608: Add donorbox widget.js script tag #17661.

This flake hasn't been seen before, so it was probably introduced in the last few days. Here are the PRs merged over the last 5 days:

• 2023-02-08: Fix #14464 Remove null as a possible option for interactionId #17132
• 2023-02-07: Fix #16830: Remove unclear sentence from save progress popup #16838
• 2023-02-07: Changes to the release process and test improvements #17154
  • Only changes backend tests and release scripts
• 2023-02-06: Fix #16787: Uninstall and reinstall dev dependencies around package build #16901
  • Only changes how we install dependencies during the build process
• 2023-02-06: Fix hashes correctly #17223
  • Only changes GitHub Actions workflow
• 2023-02-05: Trying to fix the 'Confirm Delete State Button' Flake #16777
  • Only changes E2E tests
• 2023-02-04: Fix #12910 use app.constants.ts instead of constants.ts #17084
• 2023-02-03: Fix #17021: Now Chromevox can read close button for x icon after uploading a picture #17163

Despite the error in the output being No node found for selector: mat-option[ng-reflect-value="ADMIN"], this appears to be related to donorbox. The Lighthouse reports say:

• 2023-02-28:
  

• 2023-03-11:
  

This looks to be an issue with donorbox rate limiting us

• 29 APR 2023: https://github.com/oppia/oppia/actions/runs/4834852941/jobs/8617066119
• 16 May 2023 https://github.com/oppia/oppia/actions/runs/4986559169/jobs/8927969550

[U8NWXD]

@kevintab95 it looks like this flake might be related to the issue you hotfixed with #17661. The hotfix didn't fix it, but now more pages are failing due to a 403 error from donorbox

[kevintab95]

Hi @U8NWXD, just to clarify, I think we'll need to have a "dev mode" flag for donorbox and not fetch from their domain when running e2e tests. The hotfix wasn't meant to fix a flake, the donation flow wasn't working as expected so we tried to fix that.

[U8NWXD]

Thanks! I understood that your hotfix wasn't about the flake--I just wanted to loop you in since it looked related.

Any reason you prefer skipping donorbox during the tests rather than ignoring this particular error? I'm worried that if we cut donorbox out of the tests entirely, we'll miss bugs with it.

Also /cc @seanlip since we're considering reducing test coverage

[seanlip]

I approve skipping donorbox during the tests -- I think there's a bit of an issue with exercising an external prod dependency multiple times a day due to our tests and on balance we probably shouldn't do it.

(We've actually already missed bugs even with the tests anyway.)

Will probably need to earmark this flow for manual testing on each release -- I will have a chat with @kevintab95 about that.

[U8NWXD]

OK, sounds good. Any suggestions on how to skip donorbox during the tests? Note that many tests run in prod mode, so that flag won't be enough

[U8NWXD]

I also agree with manual testing on each release. The tests have missed donorbox bugs, but if we don't test it at all, we should count on missing even more

[seanlip]

Any suggestions on how to skip donorbox during the tests? Note that many tests run in prod mode, so that flag won't be enough

Hm, I'm not familiar enough with this part -- @kevintab95 might need to comment here.

[kevintab95]

I think there are a couple of things to do here to fully solve this:

1. Fix [TODO]: Fetch widget.js from donorbox only in the donate page #17664 so that we limit fetching widget.js to the donate page only.
2. Create a new flag similar to CAN_SEND_ANALYTICS_EVENTS (e.g. DONORBOX_IS_SHOWN). Should be OK to repurpose analytics-constants.json for this if necessary. Use this flag to show / hide the script tag in the HTML.
3. Update the release scripts such that the constant is updated correctly depending on the environment.

[gp201]

Dropping this flake to priority level since we haven't seen this flake since the beginning of April.

[462702985]

occured Jun 14: https://github.com/oppia/oppia/actions/runs/5272986435/jobs/9536380740
Jun 16: https://github.com/oppia/oppia/actions/runs/5292394900/jobs/9580330457?pr=18393

[gp201]

This issue has been moved back to high priority since @462702985 has identified that the flake has been occurring at a high rate. Thanks for catching this @462702985

[seanlip]

Hi -- just a note. This looks like it is due to console errors in the /donate page which are caused by an error in a third-party component (stripe). See https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1687048851060-5704.report.html and look at the accordion under "Browser errors were logged to the console".

We should update our lighthouse config to try and exclude those errors. I saw GoogleChrome/lighthouse#9480 but I can't quite figure out how to modify .lighthouserc-1.js or .lighthouserc-base.js to include the ignoredPatterns option (my first attempt didn't work). I suggest trying to do this or just reducing the threshold for now so that PRs can go in.

[462702985]

@seanlip I didn't find where we can add ignoredPatterns too. And I tried to add Content Security Policy but I cannot find http header of stripe.com.

[seanlip]

Note, this is causing consistent failures on Docker -- it needs to be investigated.