[MRG+1] Adding more settings to project template

[eliasdorneles]

Hi, folks!

Here is a proposal for addressing issue #665.
I changed some wording, and added a few more settings that I felt were important or commonly missed.

So, what do you think? Does this look good?

Thank you,
Elias

[nramirezuy]

Which one is the default used? 😄

[nramirezuy]

CONCURRENT_REQUESTS is also missing.

[eliasdorneles]

thanks, you're right, lemme update this adding those too.

[nramirezuy]

What about Autothrottle?

[eliasdorneles]

it's at the bottom.

[nramirezuy]

saw it, but doesn't worth to mention?
Configure a delay for requests (affected by Autothrottle)

for the same website depends on the concurrents requests you choose, default is website.

[eliasdorneles]

Well, I see it like, DOWNLOAD_DELAY affects Autothrottle extension, not the other way around. Also, Autothrottle is disabled by default.
Therefore, I think we don't need to call attention for it here.

Regarding the observation for the same website -- agreed, that's why I put the two concurrent requests per domain and per ip settings soon after.
I was thinking that'd be enough, but I'm open to suggestions. Anything in mind?

[nramirezuy]

no extensions or pipelines (even tho I'm not agree with item pipelines 👅 )? :(

[eliasdorneles]

Right, I'll put those too. :)

[nramirezuy]

Configure a delay for requests for the same slot and they have to investigate on the doc what a slot is 👅

[kmike]

I think "website" is fine - it could mean either "ip" or "domain" depending on settings. Slots are not documented, so investigation won't help :)

[nramirezuy]

Oh, I guess website will have to do. I is also that way on the docs. It's a shame that slots aren't documented 😢

[eliasdorneles]

I've changed the detail settings for autothrottle and HTTP cache to the default values.

It seems more sensible in these cases that have an on/off setting, because a user enabling all of them together would expect the defaults. Thanks @redapple for the heads up. ;)

[kmike]

I think we should be consistent with comments vs commented out code - either write it like this everywhere:

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS=32

or like this:

# Crawl responsibly by identifying yourself (and your website) on the user-agent
# USER_AGENT = '$project_name (+http://www.yourdomain.com)'

[eliasdorneles]

it took me a few seconds to realize you were meaning the indentation -- you're right, I'll fix it :)

[eliasdorneles]

done

[kmike]

I don't have an opinion on what settings should we put to the default template. For the existing set of settings the PR looks good.