Failed to connect Puppeteer to existing Firefox instance run by web-ext

[denar90]

Hi,
At @treosh we are working on analyzing extensions (ext) impact on real user performance.
We try to install extensions for Firefox (FF) and handle browser with Puppeteer (PP) to do measurements and other stuff, but struggling with some sort of issues.

Tell us about your environment:

• Puppeteer version: "puppeteer-firefox": "^0.5.0"

• Platform / OS version: macOS Mojave 10.14.3

• Node.js version: v11.1.0

What steps will reproduce the problem?

There are several cases we tried and all of them based on using web-ext library for installing ext for FF.

---

Case 1: Connect FF instance run by webExt to PP

Code simple:

    webExt.util.logger.consoleStream.makeVerbose();

    extensionRunners = await webExt.cmd.run({
      sourceDir: extPath,
      // comment if connect to default FF
      firefox: PP_FF.executablePath(),
    }, {
      // These are non CLI related options for each function.
      // You need to specify this one so that your NodeJS application
      // can continue running after web-ext is finished.
      shouldExitProgram: false,
    })

    const extensionRunner = extensionRunners.extensionRunners[0]

    const browserWSEndpoint = `ws://127.0.0.1:${extensionRunner.remoteFirefox.client.client.client.remotePort}`
    browser = await PP_FF.connect({
      browserWSEndpoint,
    })

Error:

Error: Parse Error
  at Socket.socketOnData (_http_client.js:449:20)
  at Socket.emit (events.js:182:13)
  at addChunk (_stream_readable.js:283:12)
  at readableAddChunk (_stream_readable.js:264:11)
  at Socket.Readable.push (_stream_readable.js:219:10)
  at TCP.onStreamRead (internal/stream_base_commons.js:122:17) bytesParsed: 0, code: 'HPE_INVALID_CONSTANT' } 

Fairly saing it crashes in PP helper line 112, but after commenting it - error HPE_INVALID_CONSTANT can be seen.

---

Case 2: Connect webExt to FF instance run by PP

install web ext from built fork https://github.com/denar90/web-ext-firefox-port instead of web-ext original

Code simple:

    browser = await PP_FF.launch({
      headless: false,
    })

    const wsEndpoint = new URL(browser.wsEndpoint())

    webExt.util.logger.consoleStream.makeVerbose();

    extensionRunners = await webExt.cmd.run({
      sourceDir: extPath,
      firefoxPort: wsEndpoint.port,
    }, {
      // These are non CLI related options for each function.
      // You need to specify this one so that your NodeJS application
      // can continue running after web-ext is finished.
      shouldExitProgram: false,
    })

Error:
Browser is connected but never install extension

Log from console:

[firefox/remote.js][debug] Connecting to the remote Firefox debugger
[firefox/remote.js][debug] Connecting to Firefox on port 8000
[firefox/remote.js][debug] Connected to the remote Firefox debugger on port 8000
...nothing happens...

---

Case 3 (dummy one): Connect FF instance run by webExt to PP FF instance via webExt connect

    browser = await PP_FF.launch({
      headless: false,
    })

    webExt.util.logger.consoleStream.makeVerbose();

    extensionRunners = await webExt.cmd.run({
      sourceDir: extPath,
      firefox: PP_FF.executablePath(),
    }, {
      // These are non CLI related options for each function.
      // You need to specify this one so that your NodeJS application
      // can continue running after web-ext is finished.
      shouldExitProgram: false,
    })
    
    const extensionRunner = extensionRunners.extensionRunners[0]
    const connect = (port, host) => {
      return new Promise(resolve => {
        extensionRunner.remoteFirefox.client.connect(port, host, resolve);
      });
    }

    const wsEndpoint = new URL(browser.wsEndpoint())

    await connect(wsEndpoint.port, wsEndpoint.hostname)

Error:

Extension doesn't have influence on perf of page run in PP_FF browser because it's installed in FF run by web-ext.

---

Will be super cool if folks from @GoogleChrome and @mozilla can find some time and help us with solving this problem, hence we will be able to audit impact of Firefox extensions on UX.

Thanks)

[aslushnikov]

@denar90 ok I see what's going on. There's some confusion related to remote protocols and what tool uses what when connects.

• IIRC the web-ext uses firefox's RDP protocol that has a hook to install an extension
• puppeteer-firefox uses a "juggler" - a new CDP-like protocol that currently exists only in our fork

So what you want is:

1. use web-ext to launch the firefox bundled with puppeteer-firefox
2. when launching the firefox, pass a -juggler 1234 flag to open juggler websocket on port 1234
3. once the browser is launched, connect pptr-firefox to it:

puppeteerFirefox.connect({
  browserWSEndpoint: 'ws://localhost:1234',
})

One issue I got stuck with is that web-ext run doesn't let me specify cli flags for the firefox. It shouldn't be hard to modify it though.

Let me know how it goes.

[denar90]

@aslushnikov thank you very much! It works like a charm 💃
I opened PR to add browser flags support for web-ext :)