Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs #2190

Merged
merged 1 commit into from Aug 31, 2016
Merged

[MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs #2190

merged 1 commit into from Aug 31, 2016

Conversation

@stummjr
Copy link
Member

@stummjr stummjr commented Aug 19, 2016

The documentation currently states:

By default, Scrapy doesn’t wait a fixed amount of time between requests, but uses a random interval between 0.5 and 1.5 * DOWNLOAD_DELAY

It might mislead readers to think that, for a DOWNLOAD_DELAY=2, the interval would be from 0.5 to 3.0, when it's actually from 1.0 to 3.0.

See: https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/__init__.py#L38

@stummjr
Copy link
Member Author

@stummjr stummjr commented Aug 19, 2016

In addition to that, it looks like the BOT_NAME description is incorrect:

The name of the bot implemented by this Scrapy project (also known as the project name). This will be used to construct the User-Agent by default, and also for logging.

From what I tested and skimmed through the source code, it looks like it's not used to build the User-Agent. Am I right?

If so, I'll fix that too and update the PR.

@codecov-io
Copy link

@codecov-io codecov-io commented Aug 19, 2016

Current coverage is 83.40% (diff: 100%)

Merging #2190 into master will increase coverage by 0.01%

Powered by Codecov. Last update cacd038...d61650d

@kmike kmike changed the title Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs [MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs Aug 25, 2016
@kmike
Copy link
Member

@kmike kmike commented Aug 25, 2016

In addition to that, it looks like the BOT_NAME description is incorrect

Hm, a good catch. What happens is that when a project is generated by startproject command, settings.py file is created, and it has the following lines:

BOT_NAME = '$project_name'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = '$project_name (+http://www.yourdomain.com)'
@redapple
Copy link
Contributor

@redapple redapple commented Aug 31, 2016

@stummjr , let's make it another PR for bot name.

@redapple redapple merged commit eedb6ce into scrapy:master Aug 31, 2016
2 checks passed
2 checks passed
codecov/patch Coverage not affected when comparing cacd038...d61650d
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants