Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make lazy loading Download Handlers optional #3394

Merged
merged 2 commits into from
Dec 26, 2018

Conversation

hcoura
Copy link
Contributor

@hcoura hcoura commented Aug 17, 2018

Download Handlers load only when first request for a scheme is needed, this was put in place because s3 download handler used to be very slow to create.

This makes it optional for Download Handlers to be lazy loaded which is relevant when writing custom handlers that have required settings and wants to raise NotConfiguredException during instantiation.

@codecov
Copy link

codecov bot commented Aug 17, 2018

Codecov Report

Merging #3394 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3394      +/-   ##
==========================================
+ Coverage   84.46%   84.49%   +0.02%     
==========================================
  Files         167      167              
  Lines        9362     9372      +10     
  Branches     1390     1391       +1     
==========================================
+ Hits         7908     7919      +11     
  Misses       1199     1199              
+ Partials      255      254       -1
Impacted Files Coverage Δ
scrapy/core/downloader/handlers/http10.py 100% <100%> (ø) ⬆️
scrapy/core/downloader/handlers/file.py 100% <100%> (ø) ⬆️
scrapy/core/downloader/handlers/datauri.py 93.75% <100%> (+0.41%) ⬆️
scrapy/core/downloader/handlers/__init__.py 83.63% <100%> (+3.63%) ⬆️
scrapy/core/downloader/handlers/ftp.py 98.24% <100%> (+0.03%) ⬆️
scrapy/core/downloader/handlers/http11.py 92.53% <100%> (+0.02%) ⬆️

@kmike kmike added this to the v1.6 milestone Aug 24, 2018
@ghost ghost mentioned this pull request Oct 13, 2018
Copy link
Member

@kmike kmike left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks @hcoura! //cc @dangra

@dangra dangra merged commit 93cf383 into scrapy:master Dec 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants