Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix broken links and use https links wherever possible #2982

merged 4 commits into from Oct 28, 2017


Copy link

I haven't touched scrapy shell commands and other places which have the potential to break something else if not careful. Besides the changes made, I would like to point out two more issues:

  1. - WrappedRequest class mentions and links to urllib2.Request (the link is broken). Since in Python 3 it is urllib.request.Request, I am not sure about the best way to fix this.

  2. Shall we switch the default user agent to https as well?

Copy link

codecov bot commented Oct 28, 2017

Codecov Report

Merging #2982 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2982      +/-   ##
- Coverage   84.59%   84.58%   -0.02%     
  Files         164      164              
  Lines        9249     9249              
  Branches     1376     1376              
- Hits         7824     7823       -1     
  Misses       1167     1167              
- Partials      258      259       +1
Impacted Files Coverage Δ
scrapy/ 80.95% <ø> (ø) ⬆️
scrapy/utils/ 88.5% <ø> (ø) ⬆️
scrapy/utils/ 95.38% <ø> (ø) ⬆️
scrapy/core/downloader/ 70.96% <ø> (ø) ⬆️
scrapy/utils/ 90.9% <ø> (ø) ⬆️
scrapy/extensions/ 75% <ø> (ø) ⬆️
scrapy/ 91.35% <ø> (ø) ⬆️
scrapy/pipelines/ 67.04% <ø> (ø) ⬆️
scrapy/ 57.14% <ø> (ø) ⬆️
scrapy/downloadermiddlewares/ 60% <ø> (ø) ⬆️
... and 6 more

@@ -163,7 +163,7 @@ def _get_max_age(self, cc):

def _compute_freshness_lifetime(self, response, request, now):
# Reference nsHttpResponseHead::ComputeFreshnessLifetime
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


@kmike kmike merged commit 108f8c4 into scrapy:master Oct 28, 2017
Copy link

kmike commented Oct 28, 2017

Thanks @codeaditya for the careful pull request!
Yes, I think that'd be good to change User-Agent as well.

Copy link
Contributor Author

Awesome. Thanks @kmike.. It's been a very pleasurable experience.. Would send a PR for User-Agent too..

@kmike kmike added this to the v1.5 milestone Dec 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

2 participants