page.keyboard.*: pressing Ctrl-F doesn't act as pressing it manually

[kbulygin]

Steps to reproduce

Tell us about your environment:

• Puppeteer version: 1.2.0
• Platform / OS version: Debian GNU/Linux 8.8 (jessie)
• URLs (if applicable): https://example.com
• Node.js version: 9.7.1
• Chrome version: 65.0.3325.146 (Official Build) (64-bit)

What steps will reproduce the problem?

I run this code:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: false,
    executablePath: '/usr/bin/google-chrome',
  });
  const page = await browser.newPage();
  await page.goto('https://example.com');

  await page.keyboard.down('Control');
  await page.keyboard.press('KeyF');
  await page.keyboard.up('Control');
  console.log('Pressed Ctrl-F');

  await page.keyboard.type('example');

  await page.keyboard.press('F3');
  console.log('Pressed F3');
})();

What is the expected result?

I expect that the "Find in page" widget of Chrome is opened and the word "Example" on the page is highlighted.

What happens instead?

The page just opens as if there were no page.keyboard.* calls.

---

The rationale is the following. I am developing an extension that changes the DOM. Sometimes it conflicts with the Ctrl-F ("Chrome menu > Find...") mechanism:

• The user searches something on the page and jumps to a match.
• New elements become visible.
• The extension changes the new visible elements (namely, text nodes with the match are detached and equivalent nodes are attached instead).
• The match is no more highlighted.

I'd like to use puppeteer to make regression tests for this (among other things). It seems that if Ctrl-F worked, I'd get along with only puppeteer (other "meta-DOM" things work, like double-clicking and making screenshots with visible text selections and Ctrl-F highlights), but because of the issue I'll probably have to use Xvfb and xdotool.

Is such an issue on the agenda for a future release of puppeteer? Is there some other way to use the Ctrl-F mechanism (like a JavaScript API)?

The only thing I managed to find on the issue is #2009, but that is only seemingly related.

[kbulygin]

It turns out that window.find() is implemented in Chrome. So the following code emulates the Ctrl-F mechanism (except the color of highlights and perhaps some other things):

/* global window */
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: false,
    executablePath: '/usr/bin/google-chrome',
  });
  const page = await browser.newPage();
  await page.goto('https://example.com');

  await page.evaluate(() => window.find('example')); // to the first match
  await page.evaluate(() => window.find('example')); // to the second match

  await page.screenshot({ path: 'example.png' });
  console.log('Made a screenshot'); // the second match is highlighted
})();

Anyway, it would be interesting to hear on the page.keyboard.* issue. On the one hand, I see that the basic idea is dispathing a key event to the page. On the other hand, these bits on https://github.com/GoogleChrome/puppeteer/ suggest that Ctrl-F should work:

For finer control, you can use keyboard.down, keyboard.up, and keyboard.sendCharacter to manually fire events as if they were generated from a real keyboard. (https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-keyboard)

Puppeteer doesn't emulate native shortcuts because native shortcuts depend on the active window, which is out of control for puppeteer. [...] On Win and Linux, Ctrl-A is handled by Chrome, and puppeteer does emulate this. (#1313, referenced from https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-keyboard)

[aslushnikov]

@JoelEinbinder can you please take a look? Sounds similar to Ctrl-A case.

[aslushnikov]

I got confused. The Ctrl-F you try to emulate is a chromium shortcut; pptr doesn't dispatch input events to chromium - only to the webpage. So this kind of functionality is not possible with pptr.

[dvdvdmt]

@aslushnikov, I'm trying to simulate Alt + Y shortcut to trigger my Chrome extension. It is listening this combination in a background script:

// manifest.json
{
  ...
  "commands": {
    "next": {
      "suggested_key": {
        "default": "Alt+Y"
      },
      "description": "Open recent tabs list. Select next"
    }
  }
}

// background.js
browser.commands.onCommand.addListener(async (command) => {
  ...
});

I'm trying to trigger command next by the following script:

const path = require('path');
const puppeteer = require('puppeteer');

(async () => {
  const pathToExtension = path.join(__dirname, '../../dist');
  const browser = await puppeteer.launch({
    headless: false,
    slowMo: 500,
    args: [
      `--disable-extensions-except=${pathToExtension}`,
      `--load-extension=${pathToExtension}`,
    ],
  });
  const [page] = await browser.pages();
  await page.goto('https://www.wikipedia.org');
  await page.keyboard.down('Alt');
  await page.keyboard.press('KeyY');
})();

When I execute this script Chromium appears, wikipedia loads, but then nothing happens. Although when I press Alt + Y manually my extension works as it should.

Is there a solution to my problem? Or is it the same limitation of Puppeteer as you described to OP?

[BlackHole1]

@dvdvdmt hi, Has your problem been solved? I have the same problem as you.

[dvdvdmt]

@BlackHole1 hello, yes I've managed it by writing additional code that listens for keyboard shortcuts on page level (content script) and sends it back to background script. You can see my code in this issue dvdvdmt/popup-tab-switcher#1

[BlackHole1]

My plugin requires activeTab permissions. One of the ways to open this permission is to use shortcut keys. So I want to use the puppeterer input shortcut to open the activeTab permission.

[BlackHole1]

However, this method does not work. But thank you very much for your answer.

[bmoers]

@BlackHole1 any luck so far? I have the same problem with activeTab permissions.

[bmoers]

@BlackHole1 any luck so far? I have the same problem with activeTab permissions.

--whitelisted-extension-id= does the work for me

[BlackHole1]

@bmoers Wow, brother, you are so great. Thank you very much.

According to your answer, I looked at the Chrome source and found this to be feasible!

https://cs.chromium.org/chromium/src/chrome/browser/extensions/api/tab_capture/tab_capture_api.cc?sq=package:chromium&dr=CSs&targetos=chromium&g=0&l=253

[298068845]

@BlackHole1 After add the whitelisted-extension-id ，I can‘t use the puppeterer input shortcut to open the activeTab permission.Is there anything to watch out for?

[BlackHole1]

@298068845 You don't need to use a shortcut, just call this function directly.

[298068845]

@BlackHole1 https://github.com/s16h/puppeteer-tab-capture-repro/blob/master/test.js
I try it with this project，I get the log “Extension has not been invoked for the current page (see activeTab permission). Chrome pages cannot be captured.”，but I can see the extension's actionbutton and click it to use.Maybe I can't call this function directly with Chorme version 80?

[BlackHole1]

@298068845 Maybe you can try my open source project: https://github.com/alo7/rebirth

open Chrome Code: https://github.com/alo7/rebirth/blob/master/src/lib/startChrome.js#L7-L27

start capture Code: https://github.com/alo7/rebirth/blob/master/src/chrome-extension/lib/recording.ts#L24-L76

[Matos7]

Try do not use await.

async function keyPress(page, char) {
await page.keyboard.press(char);
}

keyPress(page, 'Control');
keyPress(page, 'f');

[sergkop]

Chrome has deprecated --whitelisted-extension-id a couple years ago, --allowlisted-extension-id={id} should be used now