Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install the reactor after reading spider settings #5352

Merged
merged 17 commits into from
Jan 6, 2022

Conversation

wRAR
Copy link
Member

@wRAR wRAR commented Dec 23, 2021

Closes: #4485

@codecov
Copy link

codecov bot commented Dec 23, 2021

Codecov Report

Merging #5352 (3577c00) into master (c5ab580) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #5352   +/-   ##
=======================================
  Coverage   88.56%   88.57%           
=======================================
  Files         163      163           
  Lines       10624    10631    +7     
  Branches     1809     1812    +3     
=======================================
+ Hits         9409     9416    +7     
  Misses        939      939           
  Partials      276      276           
Impacted Files Coverage Δ
scrapy/commands/shell.py 92.85% <ø> (ø)
scrapy/crawler.py 88.64% <100.00%> (+0.38%) ⬆️
scrapy/utils/log.py 89.47% <100.00%> (+0.11%) ⬆️

default.install()
log_reactor_info()
if self.settings.get("TWISTED_REACTOR"):
verify_installed_reactor(self.settings["TWISTED_REACTOR"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

works fine, tested this in my project. Minor thing, you could probably simplify this to something like this to avoid repeatedly calling self.settings.get('TWISTED_REACTOR')

reactor_class = self.settings.get("TWISTED_REACTOR")
if init_reactor:
    if reactor_class:
        install_reactor(reactor_class, self.settings["ASYNCIO_EVENT_LOOP"])
... etc

I also wonder if default value for init_reactor should be False? Are there any legitimate cases when you don't want to init reactor if you have it in settings?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'll implement this simplification. As for the flag, it was the default behavior before this change, and the use case to not init the reactor is CrawlerRunner.

docs/topics/practices.rst Outdated Show resolved Hide resolved
docs/topics/settings.rst Outdated Show resolved Hide resolved
Copy link
Member

@Gallaecio Gallaecio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but I leave the conflict resolution to you :)

@AlvinSartorTrityum
Copy link

When is the next version containing this fix supposed to be released? Thanks!

@wRAR wRAR deleted the crawler-process-reactor-later branch January 20, 2022 12:12
@wRAR
Copy link
Member Author

wRAR commented Jan 20, 2022

@AlvinSartorTrityum we don't currently have a date for the next release.

@JSPromisel
Copy link

Any updates on when this will be released? Would be very useful to have.

@elacuesta
Copy link
Member

This was released as part of 2.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TWISTED_REACTOR setting not honored from Spider.custom_settings
6 participants