better bot detection protection

[am0o0]

Hey man, I am using webchanges, and it is fantastic!
What is your opinion about using https://crawlee.dev/docs/quick-start? It'll add a configuration layer for bot detection escape! It's really helpful, I think

[mborsetti]

I am not familiar with it but super supportive of anything that makes notifications/monitoring more useful, and like the bot detection escape angle of it.

However, it seems to lack a Python interface, so am a bit confused as to how it would be integrated. Do you have any thoughts?

[am0o0]

I don't know how to use it in python, either, maybe with the help of js2py?? But I think it can cause some problems afterward.
There are exciting articles like this one that tell us your crawler should pass all tests in this page
it was a fascinating article for me! I write a playwright code for it too! ( I passed all tests except one because I couldn't find any API for that in playwright)

const { chromium } = require('playwright');

(async () => {
    const browser = await chromium.launch()
    const context = await browser.newContext({
        userAgent: 'Mozilla/15.0 (X11; Linux x86_64)' +
        'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36'
      });
    const page = await context.newPage()
    await page.addInitScript(() => {
        Object.defineProperty(navigator, 'webdriver', {
          get: () => false,
        });
    });
      
    await page.addInitScript(() => {
        // We can mock this in as much depth as we need for the test.
        window.navigator.chrome = {
          runtime: {},
          // etc.
        };
      });
      
    await page.addInitScript(() => {
        const originalQuery = window.navigator.permissions.query;
        return window.navigator.permissions.query = (parameters) => (
            parameters.name === 'notifications' ?
            Promise.resolve({ state: Notification.permission }) :
            originalQuery(parameters)
        );
    });
    await page.addInitScript(() => {
        // Overwrite the `plugins` property to use a custom getter.
        Object.defineProperty(navigator, 'plugins', {
            // This just needs to have `length > 0` for the current test,
            // but we could mock the plugins too if necessary.
            get: () => [1, 2, 3, 4, 5],
        });
    });
    await page.goto('https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html')
    await page.screenshot({ path: 'my_screenshot.png' })
    await browser.close()
})()

[mborsetti]

Thanks for the link to https://intoli.com/blog/not-possible-to-block-chrome-headless/. Great stuff.

Release 3.11 is out, and all the suggestions have been implemented. Jobs using use_browser: true pass the test at https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html

Thanks again for (indirectly) suggesting this!

[am0o0]

You're welcome, mate.