feat(puppeteer): enable new headless mode

[B4nan]

https://developer.chrome.com/articles/new-headless/

To opt out of it and keep using the old headless, add headless: 'old to your puppeteer crawler options.

[Strajk]

Technically, the old Headless was a separate, alternate browser implementation that happened to be shipped as part of the same Chrome binary. It doesn’t share any of the Chrome browser code in //chrome.

wowo, this new proper headless could solve a lot of small issue 🔥

[B4nan]

We will probably keep this for later, from what I've read the new headless mode can be much slower. I haven't seen that in my initial testing, but we need to be more careful here. Will probably make this configurable from outside, so you can opt in to the new headless mode and give it a try yourself - now it will probably fail on some ow validation that requires a boolean, but maybe you could hack it through the launcher args.

[Dineshhardasani]

any update on this?

[B4nan]

Nope, same as my last comment. But I think you can opt-in for the new headless mode already, this PR only changes the default.

Something like this should work:

const crawler = new PuppeteerCrawler({
    launchContext: {
        launchOptions: {
            headless: 'new',
        },
    },
    // ...
});

[Dineshhardasani]

I have tried headless new but it is taking very high memory, have you tried benchmarking it with headless true? @B4nan

[abhisheksurve45]

Can we try to test this out @B4nan. Facing similar memory issues with headless new.

[B4nan]

I feel like you guys misunderstood this completely. This PR enables the new headless mode. That is what you can already do yourself, as I showed in my last comment. And as I already said, the new headless mode has a significant memory impact, so we don't want to adopt this just now.

Saying you have memory issues with the new headless mode just confirms what I said already - this is not something we can fix anyhow.

[github-actions]

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

[B4nan]

Potential cause of the broken tests: puppeteer/puppeteer#10017

Things are working fine locally, so might be a linux issue (but that's the main target for running crawlee in the cloud).