Stress test: scrape search results user story

[pavelfeldman]

I attempted to create a following stress test:

• Navigate to google.com
• Enter "Blin"
• Capture screenshot with results
• Go over the results
• Capture screenshot for every result page

This is pretty much impossible using existing API :) Dumping observations here...

• page.waitFor resolves even if element is not on the screen (display:none, etc). Many elements are in DOM too early, no way to click-via-screen them.
• page.waitFor does not time timeout
• keyboard API surface is large and cryptic. Did you figure out how to press 'Enter' using just page.*.
• keyboard API does not allow for natural click delays, does not populate timestamps. google.com does not like that
• Expose custom raf or timer-based page.waitFor(predicate)
• Expose page.sleep(time)
• Need to be able to page.waitForNavigation in case navigation is initiated via click [i'll fix]
• Reload brings me back to the originally navigated page [i'll fix]
• page.waitFor(selector) in my code is always followed by either page.click or page.focus with that element. Handles would make it look like (await page.waitFor(selector)).click().
• navigation API is not exposed, I need to be able to click browser's back/forward.
• Taking screenshot halts on mac headless good half of the time [i'll look at it]

^^ @aslushnikov @JoelEinbinder @dgozman @paulirish

[JoelEinbinder]

Why do you want to press enter? Usually there is an equally clickable element.

[JoelEinbinder]

let Browser = require('../lib/Browser.js');
let browser = new Browser({ args: ['--no-sandbox'] });
browser.newPage().then(async page => {
  await page.navigate('https://www.google.com');
  let query = 'input[title="Search"]';
  page.focus(query);
  page.type('blin');
  page.keyboard.press('Enter');
  await page.waitFor('h3 a');
  let urls = await page.$$('h3 a', a => a.href);
  let i = 0;
  for (let url of urls) {
    i++;
    await page.navigate(url);
    await page.screenshot({
      path: i + '.png'
    });
  }
  browser.close();
});

page.waitFor didn't work for me. Added text nodes crash it, and a new node could make the selector resolve anywhere in the document.

[pavelfeldman]

• Running your script on a Mac, it does not work. I get stuck with blink| in the search field.
• Also, this script does not do what I want - I want to be clicking search results, not navigating manually. That covers the back/forward use cases
• And to add stress to it, I want to be clicking the search button. It is in DOM all the time, but invisible half of the time which adds fun!

[aslushnikov]

The page.waitFor was broken and is now fixed in 28265fc

A few observations:

• For some reason, google.com handles "Enter" in input field differently every other time. Sometimes it issues a navigation to the https://www.google.com/search?q=blin endpoint, and sometimes it behaves as SPA and navigates to https://www.google.com/#q=Blin.
• Opening multiple tabs and navigating them altogether doesn't work - half of the tabs timeout navigation
• Trying to take screenshot simultaneously from multiple tabs doesn't work good - screenshotting hangs

The following works for me, but I don't really click the links and don't really click the "search" button. But we'll fix it!

const {Browser} = new require('.');
const browser = new Browser({headless: false});

const VIEWPORT = {width: 1000, height: 600};

browser.newPage().then(async page => {
  await page.setViewport(VIEWPORT);
  await page.navigate('https://google.com');
  await page.focus('#lst-ib');
  await page.type('Blin');
  await page.keyboard.press('Enter');

  try {
    await page.waitFor('div.g');
  } catch(e) {
    // Page might issue a navigation, so the page.waitFor will throw.
    await page.waitFor('div.g');
  }

  await page.screenshot({path: 'ee.png'});
  let links = await page.$$('div.g h3 > a', link => link.href);
  // have to do this sequentially =/
  for (let i = 0; i < links.length; ++i)
    await screenshotURL(links[i], `${i}.png`);
  browser.close();
});

async function screenshotURL(url, name) {
  let page = await browser.newPage();
  await page.setViewport(VIEWPORT);
  try {
    await page.navigate(url, { maxTime: 5000 });
  } catch (e) {
    // we did our best.
  }
  await page.screenshot({path: name});
  page.close();
  console.log('Done: ' + name);
}

[aslushnikov]

After the offline discussion of the stress-test, here are the bullets:

• make mouse to scroll element into view before clicking (otherwise it doesn't work!)
• introduce page.press as a shortcut for page.keyboard.press
• don't do any network throttling other than offline mode. We're bad at emulating it
• implement page.waitForVisible(selector) which comes handy
• implement page.waitForNavigation
• implement page.back / page.forward
• improve support for DEBUG module for public api methods
• make screenshot tasks per browser, not per page. Screenshotting doesn't work across multiple tabs
• screenshots on mac seem to be unstable
• rename the keyboard's hold into down, and release into up
• page.waitForSelector should survive navigation (currently it doesn't)
• page.waitForSelector should timeout after some time (e.g. 30s by default)

[aslushnikov]

The resulting script looks like this:

const {Browser} = require('puppeteer');
const browser = new Browser({headless: false});

browser.newPage().then(async page => {
  page.on('load', () => console.log('LOADED: ' + page.url()));
  await page.navigate('https://google.com');
  await page.waitFor('input[name=q]');
  await page.focus('input[name=q]');
  await page.type('blin');
  await page.press('Enter');
  for (let i = 0; i < 10; ++i) {
    let searchResult = `div.g:nth-child(${i + 1}) h3 a`;
    await page.waitFor(searchResult, {visible: true});
    page.click(searchResult);
    await page.waitForNavigation();
    await page.screenshot({path: `screenshot-${i + 1}.png`});
    await page.goBack();
  }
  browser.close();
});

The two checkboxes from feedback which are not addressed are filed separately:

• "screenshots on mac seem to be unstable" - which is UnitTests on Mac are timeouting #100
• "don't do any network throttling other than offline mode. We're bad at emulating it" - which is Emulate offline #63

[ralyodio]

You might be in different buckets too. I'm sure google is doing A/B testing. There's a way to force a bucket, but I'm not sure how.