URLHeadBear: fake the user agent #2879
Conversation
|
Comment on 5d00e26. Shortlog of HEAD commit does not match given regex: ([^:]|[^:]+: [A-Z0-9].*) Origin: GitCommitBear, Section: |
Akhelesh
force-pushed
the
URLHeadBear
branch
from
5d00e26
to
87fa05e
Compare
|
Comment on 87fa05e. Shortlog of HEAD commit does not match given regex: ([^:]|[^:]+: [A-Z0-9].*) Origin: GitCommitBear, Section: |
1 similar comment
|
Comment on 87fa05e. Shortlog of HEAD commit does not match given regex: ([^:]|[^:]+: [A-Z0-9].*) Origin: GitCommitBear, Section: |
Akhelesh
force-pushed
the
URLHeadBear
branch
from
6f10567
to
258838f
Compare
bear-requirements.txt
Outdated
| @@ -11,6 +11,7 @@ cpplint~=1.3 | |||
| dennis~=0.9 | |||
| docutils-ast-writer~=0.1.2 | |||
| eradicate~=0.1.6 | |||
| fake-useragent~=0.1.11 | |||
There was a problem hiding this comment.
headers present in fake-useragent are pretty old. Do we really want to use it?
bears/general/URLHeadBear.py
Outdated
| @@ -82,8 +83,9 @@ def check_prerequisites(cls): | |||
| @staticmethod | |||
| def get_head_response(url, timeout): | |||
| try: | |||
| headers = {'User-Agent': UserAgent().random} | |||
There was a problem hiding this comment.
And why are we using a random UserAgent? Using a specific browser header like firefox/chrome would be much better.
On that note, why do we even need to generate a user-agent. Why not just hardcode the latest chrome header?
There was a problem hiding this comment.
@frextrite I tried using a random user agent since some websites block the user if it makes many requests but I Googled more to find out that it would be useless unless IP addresses are rotated too. So in this case your solution suits better.
Akhelesh
force-pushed
the
URLHeadBear
branch
from
a50aec0
to
258838f
Compare
A random fake user agent is used to request webpages that do not allow bot visits. Fixes coala#1203
Akhelesh
force-pushed
the
URLHeadBear
branch
from
258838f
to
76cb8cc
Compare
|
You could run |
Yeah I have created an issue for the same to keep it separate from this one. |
A random fake user agent is used to request webpages
that do not allow bot visits.
Fixes #1203
For short term contributors: we understand that getting your commits well
defined like we require is a hard task and takes some learning. If you
look to help without wanting to contribute long term there's no need
for you to learn this. Just drop us a message and we'll take care of brushing
up your stuff for merge!
Checklist
them.
individually. It is not sufficient to have "fixup commits" on your PR,
our bot will still report the issues for the previous commit.) You will
likely receive a lot of bot comments and build failures if coala does not
pass on every single commit!
After you submit your pull request, DO NOT click the 'Update Branch' button.
When asked for a rebase, consult coala.io/rebase
instead.
Please consider helping us by reviewing other peoples pull requests as well:
corobo mark wip <URL>to get it outof the review queue.
The more you review, the more your score will grow at coala.io and we will
review your PRs faster!