Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed 4chan #107

Merged
merged 2 commits into from Mar 8, 2018

Conversation

@ghost
Copy link

commented Mar 7, 2018

Removing referrer bypasses captcha

Juan Solo added some commits Mar 7, 2018

Juan Solo
No more captcha
<meta name="referrer" content="no-referrer" />
Juan Solo
No more Captcha
Fixes 4chan, probably other cloudflare providers or whatever
@noqqe

This comment has been minimized.

Copy link
Member

commented Mar 7, 2018

Can you explain how this fix works?

As you can see, im not an active user of the 4chan crawler, but appriciate work on any module.

@ghost

This comment has been minimized.

Copy link
Author

commented Mar 7, 2018

Basically when you add a referrer to the browser's request URL to 4chan, it sends your host to 4chan and it can detect you crawling and sends a captcha. So the code removes the referrer of your host being sent to 4chan and then you don't get a captcha.

However there is a host header still sent to cloudflare which is necessary. But only sends i.4cdn.org

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img#attr-referrerpolicy

As an example: If I ran this code at crawler.net
Crawler.net would be sent in the GET header a bunch throwing the captcha.
Same for 192.168.0.100:5000

4chan doesnt like that many requests from the same referrer (and sending a referrer implies crawling as well) However, I'm not sure how their page loading works on their end and probably turn off referrers in their own code or have some sort of workaround. (Or from 4chan directly their own referrer would be 4chan.org)

@noqqe noqqe merged commit cf164b5 into k4cg:master Mar 8, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.