Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Reactor threadpool max size setting #1123

Merged
merged 6 commits into from Apr 1, 2015

Conversation

@sibiryakov
Copy link
Contributor

@sibiryakov sibiryakov commented Mar 31, 2015

continuing of #1092

@sibiryakov
Copy link
Contributor Author

@sibiryakov sibiryakov commented Mar 31, 2015

@@ -12,6 +12,7 @@
from scrapy.utils.ossignal import install_shutdown_handlers, signal_names
from scrapy.utils.misc import load_object
from scrapy import log, signals
from pdb import set_trace

This comment has been minimized.

@kmike

kmike Mar 31, 2015
Member

this import is not needed

.. setting:: REACTOR_THREADPOOL_MAXSIZE

REACTOR_THREADPOOL_MAXSIZE
------------------

This comment has been minimized.

@kmike

kmike Mar 31, 2015
Member

The underline is too short - it'll break the markup. ReST requires underline length to be >= text length.


The maximum limit for Twisted Reactor thread pool size. This is common multi-purpose thread pool used by various
Scrapy components. Threaded DNS Resolver, BlockingFeedStorage, S3FilesStore just to name a few. Increase this value if
you're experiencing problems with insufficient blocking IO.

This comment has been minimized.

@kmike

kmike Mar 31, 2015
Member

Could you please format the text to fit 80 (or 70) columns?

This comment has been minimized.

@kmike

kmike Mar 31, 2015
Member

What do you think about adding a note to the "broad crawl" docs?

This comment has been minimized.

@@ -152,6 +153,8 @@ def start(self, stop_after_crawl=True):
if self.settings.getbool('DNSCACHE_ENABLED'):
reactor.installResolver(CachingThreadedResolver(reactor))

tp = reactor.getThreadPool()
tp.adjustPoolsize(maxthreads=self.settings.getint('REACTOR_THREADPOOL_MAXSIZE'))

This comment has been minimized.

@kmike

kmike Mar 31, 2015
Member

adjustPoolSize raises an exception if maxthreads is greater than minthreads. What is the default min value? I can only find "5" in twisted sourc code; does it mean REACTOR_THREADPOOL_MAXSIZE <= 5 are not allowed? It could make sense to decrease min value if user sets REACTOR_THREADPOOL_MAXSIZE to a smaller value.

This comment has been minimized.

@sibiryakov

sibiryakov Apr 1, 2015
Author Contributor

The default is "0".
https://github.com/twisted/twisted/blob/trunk/twisted/internet/base.py#L955
It hits assert if minthreads is greater than maxthreads.
I think, we're fine here.

@kmike
Copy link
Member

@kmike kmike commented Apr 1, 2015

LGTM.

@kmike kmike changed the title Reactor threadpool max size setting [MRG+1] Reactor threadpool max size setting Apr 1, 2015
dangra added a commit that referenced this pull request Apr 1, 2015
[MRG+1] Reactor threadpool max size setting
@dangra dangra merged commit 27591b5 into scrapy:master Apr 1, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.