Puppeteer are now restricted on certain websites?

[bdrtsky]

Hello. From past a half year we used Puppeteer to automate screenshots of hateful and discriminating messages and posts from certain people on several Social Media sites, like Twitter. But now this is what we have when we try to make a screenshot

I am not talking about any actions that require signin or any kind of Bot behavior - just a screenshot of the link!

The question is - is this some kind of sophisticated Puppeteer detection from Twitter side, or this is baked-in Puppeteer rule from now on? (I see some weird things in DevTools and it looks like it's Puppeteer limitation).

[vsemozhetbyt]

Can this be the same issue with some https connections? #3880

[bdrtsky]

@vsemozhetbyt looks like it! I can reproduce your issue also.

[aslushnikov]

@bdrtsk can you please come up with any simple script that reproduces the problem? It'll help tremendously

[bdrtsky]

@aslushnikov sure! Code is basic

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://twitter.com/kvlly/status/1091730024175161344');
  await page.screenshot({path: 'twitter-post.png'});
  await browser.close();
})();

My result of screenshot is:

As you can see, there's somethings going on here...

[vsemozhetbyt]

You can just run this, open console and see many failed connections for images and CSS:

'use strict';

const puppeteer = require('puppeteer');

(async function main() {
  try {
    const browser = await puppeteer.launch({ headless: false });
    const [page] = await browser.pages();

    await page.goto('https://twitter.com/kvlly/status/1091730024175161344');
  } catch (err) {
    console.error(err);
  }
})();

[vsemozhetbyt]

I am not sure if this is connected, but I've reported some similar issue just before the involved version and it was fixed just in the involved version: https://bugs.chromium.org/p/chromium/issues/detail?id=923685
Could it be that the fix somehow was not included yet?

[bdrtsky]

@vsemozhetbyt I assume this connections are failed because of detection? I can't explain it otherwise, because from the same maschine I have no problems in regular Chrome browser...This looks pretty frustrating.

[JoelEinbinder]

Found the culprit: https://chromium.googlesource.com/chromium/src/+/cee722a155e623809cc241d27bdb35fe278135f4

Looks like we picked a bad Chromium revision.

[bdrtsky]

Nice found! Hope this will be resolved.

[AlloryDante]

Nice, Thank you Joel!