Feature Request: Undetected Browser / Bot Bypassing Capability

[dx-bear]

Are there any plans to add undetected browser support or bot bypassing capabilities similar to how tools like Playwright Stealth or Puppeteer Stealth work ?

[tallesdantas]

call me on discord _tallesjp _ (remove space after p )

[krichprollsch]

Lightpanda is very easy to spot regarding its differences with mainstream browsers, and it will take time to fill the gap.
We don't plan to do specific work to bypass bot protection.

Relates with #397

[handpickencounter]

One aspect of this should not be that difficult to achieve, you just need to mimic the TLS and HTTP behavior of Chrome.

See for example: https://github.com/enetx/surf

I don't see how lightpanda will reach even a fraction of its potential when it's blocked by almost everything.

Currently, the above project (surf) that doesn't even have javascript is more useful than lightpanda. Because it only gets blocked (JS challenge) when crawling with dirty IPs.

[dx-bear]

I stopped planning to work with Lightpanda after they said, “We don’t plan to…”

[krichprollsch]

That's true we could implement these kind of chrome's mimic.
But executing JS offers a big surface for bot protection to detect the differences w/ chrome.
So it will require a lot of work for a really small gain in my opinion.

Our focus is to provide more web apis compatibilty to be usable in more use cases.
If you want to bypass bot protection, it will be always better to try w/ a real human facing browser.

It's maybe naive, but I strongly suggest you to always try a first respectful approach by announcing yourself a respect robots.txt rules. if you well intended, owners could let you pass in the long run.

But maybe I'm wrong, and I will be happy to discuss your use-cases, here or privately by email or any other way.

[elidickinson]

curl-impersonate is a patched curl lib that does a good job of mimicking the TLS and HTTP signature of different browsers. So that part is sort of a solved problem, though I'm not sure how hard it would be to switch the zig bindings from the official curl to the fork

[dx-bear]

curl-impersonate is a patched curl lib that does a good job of mimicking the TLS and HTTP signature of different browsers. So that part is sort of a solved problem, though I'm not sure how hard it would be to switch the zig bindings from the official curl to the fork

I don’t understand why people consider curl-impersonate
a good library — in my experience, it doesn’t perform well for serious web scraping or automation. I still get detected by anti-bot systems almost every time I use it.

[elidickinson]

I don’t understand why people consider curl-impersonate a good library — in my experience, it doesn’t perform well for serious web scraping or automation. I still get detected by anti-bot systems almost every time I use it.

It handles the network level requests very well in my tests. But of course many/most bot detection happens at the browser level with JavaScript (and fonts and canvas and webgl etc). Not really possible to get past that without running a real browser engine on some level (it’s tricky to even make a real browser undetectable when running it headless). Doesn’t seem like that is compatible with the goals of lightpanda, but there are tons of projects for undetectable playwright etc.

Making that initial first request undetectable should be close to a drop-in replacement of the standard curl library it uses though.