Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add reference user agent for ebay crawler #8

Merged
merged 1 commit into from
May 27, 2020

Conversation

codders
Copy link

@codders codders commented May 26, 2020

EBay appears to filter some user agents that correspond to scripts. Since the URLs we access don't seem to be excluded by robots.txt, one assumes this is over-zealous DoS attack protection. Use a browser User Agent to avoid the filter.

@codders codders requested a review from nihalsid May 26, 2020 17:55
@codecov-commenter
Copy link

codecov-commenter commented May 26, 2020

Codecov Report

Merging #8 into master will increase coverage by 0.09%.
The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master       #8      +/-   ##
==========================================
+ Coverage   59.05%   59.15%   +0.09%     
==========================================
  Files           7        7              
  Lines         425      426       +1     
  Branches       50       50              
==========================================
+ Hits          251      252       +1     
  Misses        159      159              
  Partials       15       15              
Impacted Files Coverage Δ
flathunter/crawl_ebaykleinanzeigen.py 61.42% <66.66%> (+0.55%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5438b81...58ee4f8. Read the comment docs.

@mordax7
Copy link

mordax7 commented May 26, 2020

LGTM

@mordax7 mordax7 self-requested a review May 26, 2020 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants