Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix broken links and use https links wherever possible #2982

Merged
merged 4 commits into from Oct 28, 2017

Conversation

@codeaditya
Copy link
Contributor

@codeaditya codeaditya commented Oct 28, 2017

I haven't touched scrapy shell commands and other places which have the potential to break something else if not careful. Besides the changes made, I would like to point out two more issues:

  1. https://github.com/scrapy/scrapy/blob/master/scrapy/http/cookies.py#L115 - WrappedRequest class mentions and links to urllib2.Request (the link is broken). Since in Python 3 it is urllib.request.Request, I am not sure about the best way to fix this.

  2. Shall we switch the default user agent to https as well?

@codecov
Copy link

@codecov codecov bot commented Oct 28, 2017

Codecov Report

Merging #2982 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2982      +/-   ##
==========================================
- Coverage   84.59%   84.58%   -0.02%     
==========================================
  Files         164      164              
  Lines        9249     9249              
  Branches     1376     1376              
==========================================
- Hits         7824     7823       -1     
  Misses       1167     1167              
- Partials      258      259       +1
Impacted Files Coverage Δ
scrapy/signalmanager.py 80.95% <ø> (ø) ⬆️
scrapy/utils/log.py 88.5% <ø> (ø) ⬆️
scrapy/utils/deprecate.py 95.38% <ø> (ø) ⬆️
scrapy/core/downloader/contextfactory.py 70.96% <ø> (ø) ⬆️
scrapy/utils/http.py 90.9% <ø> (ø) ⬆️
scrapy/extensions/telnet.py 75% <ø> (ø) ⬆️
scrapy/crawler.py 91.35% <ø> (ø) ⬆️
scrapy/pipelines/files.py 67.04% <ø> (ø) ⬆️
scrapy/_monkeypatches.py 57.14% <ø> (ø) ⬆️
scrapy/downloadermiddlewares/chunked.py 60% <ø> (ø) ⬆️
... and 6 more
@@ -163,7 +163,7 @@ def _get_max_age(self, cc):

def _compute_freshness_lifetime(self, response, request, now):
# Reference nsHttpResponseHead::ComputeFreshnessLifetime
# http://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpResponseHead.cpp#410
# https://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpResponseHead.cpp#706

This comment has been minimized.

@kmike

kmike Oct 28, 2017
Member

👍

@kmike kmike merged commit 108f8c4 into scrapy:master Oct 28, 2017
2 checks passed
2 checks passed
codecov/patch Coverage not affected when comparing 79df51a...dae7b1c
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@kmike
Copy link
Member

@kmike kmike commented Oct 28, 2017

Thanks @codeaditya for the careful pull request!
Yes, I think that'd be good to change User-Agent as well.

@codeaditya
Copy link
Contributor Author

@codeaditya codeaditya commented Oct 28, 2017

Awesome. Thanks @kmike.. It's been a very pleasurable experience.. Would send a PR for User-Agent too..

@kmike kmike added this to the v1.5 milestone Dec 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants