[MRG+1] Startproject templates override

[palego]

Hello Scrapy-makers :-)

This would be my first, modest, contribution to Scrapy, that I have been pleased to use on and off for a couple of years now.

This is an attempt to address the missing override of TEMPLATES_DIR at project creation described in issue #671.

I saw PR #1035, but it seems to be on a dead end, and the reasons for that seem to be:

• it implemented multiple override levels in a way that is not compatible with the new override mechanism,
• considering that there is no settings.py (let alone spiders) at startproject time, the only level of override is the command-line, so it was possibly a bit over-engineered.

With this in mind, this PR keeps to the simple route proposed in the issue and copies/pastes from genspider.py.

Looking forwards to your feedback.

[codecov-io]

Current coverage is 82.89%

Merging #1575 into master will increase coverage by +0.01% as of 2dc97ae

Powered by Codecov. Updated on successful CI builds.

[kmike]

Looks good, +1 to merge. Could you please squash the commits?

[palego]

This way?

[kmike]

👍

[palego]

Finding now that the content in the first level of the templates_dir is ignored, except for scrapy.cfg.
I would find useful to add some reusable contents in there such as:

• files that belong to the project root (i.e. .gitignore),
• data directories that belong better out of the way of the code (like a csv feed output directory).

The change is pretty easily done by doing the copytree at the project root level, and then rename 'module' to project_name.

This would impact both the files that were touched by this PR (startproject.py & test_commands.py).

My question is about the workflow: should I update this PR?
Or should I make a new branch, based on this branch, as there would be a dependency on the test code?
Or is there a smarter way around this?

[palego]

Found https://github.com/codecov/support/wiki/Commit-Coverage-Status , but did not understand the slightest part of it.
Could someone give me a hint (be it explanation, action or documentation) about the codecov failure, I would be grateful.
EDIT: more specifically, the fact that docs are touched does not explain why over a dozen files, most untouched by this PR, are listed in the report.

[palego]

@kmike should I squash it all again? or is it too messed up at this point?

[kmike]

Hey @palego,

Sorry for the delay. Yeah, codecov thinks you've touched many files because scrapy master changed since the PR was opened. Rebasing on master should fix it. I think there is no need to squash, but it doesn't hurt either :)

The PR looks fine to me, +1 to merge it.

My main worry is that we probably shouldn't go too far improving scrapy template generating features; if it is hard to support IMHO it is better to switch to http://cookiecutter.readthedocs.org/en/latest/.

[palego]

Hi @kmike,

Nevermind the delay, I understand this is low priority.

Thanks for the explanation (sorry, I should have realized that by myself instead of wasting your time...). I will rebase/squash this evening.

Regarding template generation, I won't go any further. The purpose here is just to have the current mechanism working as documented, with no harm to a future cookiecutter implementation.

[palego]

Done.
And improving: now I pulled a strike ;-)
The test failures do not seem related to my changes though.