Impossible to scrape this website

[simjoeweb]

I get blocked on the first or second page. If I complete the initial verification then it works fine after that. I tried using proxies but I still get blocked after 1 page. Any ideas what it could be?

puppeteerExtra.use(pluginStealth());
const browser = await puppeteerExtra.launch({
    headless: false,
    defaultViewport: {
      width: 1200,
      height: 800,
    }
});
const page = await browser.newPage();
await page.waitFor(5000);
await page.goto("https://www.fnac.com/Telephone-d-occasion/Telephonie-et-Objets-Connectes-d-occasion/Fnac-Darty-Occasion/nsh411995/w-4?PageIndex=1", { waitUntil: "networkidle2" });
await page.waitFor(60000);
await page.goto("https://www.fnac.com/Telephone-d-occasion/Telephonie-et-Objets-Connectes-d-occasion/Fnac-Darty-Occasion/nsh411995/w-4?PageIndex=2", { waitUntil: "networkidle2" });
await page.waitFor(60000);
await page.goto("https://www.fnac.com/Telephone-d-occasion/Telephonie-et-Objets-Connectes-d-occasion/Fnac-Darty-Occasion/nsh411995/w-4?PageIndex=3", { waitUntil: "networkidle2" });
await page.waitFor(60000);
await page.goto("https://www.fnac.com/Telephone-d-occasion/Telephonie-et-Objets-Connectes-d-occasion/Fnac-Darty-Occasion/nsh411995/w-4?PageIndex=4", { waitUntil: "networkidle2" });

[JohnDotOwl]

@simjoeweb , try this https://bot.sannysoft.com/
await page.screenshot({ path: 'screenshot.jpg', type: 'jpeg' , fullPage: true});

See if it's setup properly, there are stuff such as viewport that cna affect the result. Otherwise, check if you're on VPS , most site block famous VPS like DO , or AWS.

To check this, just use your home IP address or mobile phone , it's a good way to see if those "VPS" are blocked.

[brunogaspar]

On top of what @Rainbowhat mentioned, i would recommend to also send a more valid user agent, you can use the user-agents library for that.

Also try to send some headers like accept, accept encoding, accept language, etc..

For example:

await page.setExtraHTTPHeaders({
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,q=0.9',
    'Cache-Control': 'none',
})

You can inspect what the browser is sending and choose

[yfe404]

https://datadome.co/fr/bot-management-protection/une-detection-efficace-cote-client-est-essentielle/ They are talking about you

[brunogaspar]

Considering that your bot is being detected, i would apply some proxies and clear cookies on each run (if you don't close the browser, between the runs).

There's work being done to hopefully help on avoid being detected, but, it's impossible to guarantee that.

Closing for now.