With Firefox page.goto() times out on some websites like Twitter due to COOP headers

[EpsilonMoves]

The following code prints "success" to console

puppeteer = require("puppeteer");

(async () => {
const browser = await puppeteer.launch({
product: "firefox",
headless: false
});
const page = await browser.newPage();
await page.goto("https://google.com/");
console.log('success');
await browser.close();
})();
Changed the site from 'google' to 'twitter'. The following code results in TimeoutError: Navigation timeout of 30000 ms exceeded

const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch({
product: "firefox",
headless: false
});
const page = await browser.newPage();
await page.goto("https://twitter.com/");
console.log('success');
await browser.close();
})();

Steps to reproduce

Tell us about your environment:

• Puppeteer version: 8.0.0
• Platform / OS version: Windows 10
• Node.js version: 14.16.0

What steps will reproduce the problem?

Code pasted above

What is the expected result?

"success" printed to console

What happens instead?

TimeoutError: Navigation timeout of 30000 ms exceeded

[ex1st]

Hi, it should help:

await page.goto("https://twitter.com/", { waitUntil: ['load', 'domcontentloaded'] });

[whimboo]

@EpsilonMoves could you please enable dumpio when launching Firefox and also set the Firefox preference remote.log.level to Trace? It will give way more detailed output that you can then attach to this issue. Thanks.

[czinck]

I can reproduce this too, on both 8.0.0 and 7.1.0. I attached both the remote.log.level: Trace logs and the puppeteer DEBUG=* logs. It pretty clearly shows that page.goto('https://google.com') gets a load and DOMContentLoaded event, and twitter doesn't get any events. I'm also running into this with local files.

puppeteer.log
firefox.log

The Firefox logs have some lines truncated, not sure how to fix that, but I hope it doesn't matter.

[juliandescottes]

The CDP implementation in Firefox (which puppeteer uses) is not compatible with navigations that force the new page to run in a different process.

twitter.com defines COOP headers which enforce this (see https://hacks.mozilla.org/2020/07/safely-reviving-shared-memory/ & https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cross-Origin-Opener-Policy)

Supporting cross process navigations is not currently on our roadmap for CDP. We are focusing on implementing WebDriver BiDi which should be used by puppeteer in the future and which will support cross process navigations.

In the meantime I wonder if we could have any way to give better user feedback here (eg "navigating to twitter.com triggered a process change which is not supported at the moment"). We should have the necessary events to detect that, it might be worth doing.

[whimboo]

Please note that we track that problem via https://bugzilla.mozilla.org/show_bug.cgi?id=1706353.

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[whimboo]

We are tracking this problem on our own. See #6940 (comment).

It's very unlikely that we will get this fixed in Firefox for our CDP implementation, but this kind of navigation should already work with WebDriver BiDi. As such closing this issue.