fix utils.template.render_templatefile() bug +test

[Digenis]

I fixed a potential bug (untriggered so far because of the .py extension of the template files)
but while writing the test, it ended taking up most of the changes done in this PR.
Please comment whether I should just push the bugfix.

The potential bug is that .rstrip() in utils.template.render_templatefile()
is used like it is expected to strip a string, not a character class
(think of using re.sub(r'[\.tmpl]*$, '', path) instead of re.sub(r'\.tmpl$', '', path).
On filepaths that always end with '.py.tmpl', the problem goes unnoticed.

Also, shouldn't there be some expectation or even an assertion
that the template files have the '.tmpl' extension?

[kmike]

@Digenis - a good catch about rstrip, +1 to fix it.

I'm not a fan of mock tests though - they are unreadable, and they may require a rewrite if the internal implementation changes. But some tests may be better than no tests :)

Maybe we should just switch to https://github.com/audreyr/cookiecutter to reduce maintenance burden. It is well-supported, provide more features and already works in Python 3.

[dangra]

I prefer to use simple string functions instead of regex whenever I can, my suggestion is to use path.rpartition('.')[0] if this is about striping the template extension only.
In any case it looks good to me.

[Digenis]

cookiecutter brings several dependencies https://github.com/audreyr/cookiecutter/blob/master/setup.py#L26-L31

[Digenis]

@dangra I would use .rpartition if there was some assurance about the filename .tmpl extension.
Since there is none, I switched to something equivalent of the previous regex
(and to what I think was intended on the original expression)

What I would do in order of preference is:
either always pass the path to the output file and let the func append the .tmpl extension where needed,
or assert inside the function body that the extension is there.

Should I go for the first one?

[dangra]

missing w mode.

[dangra]

>>> path = 'templ.tmpl'
>>> path[:len('.tmpl')]
'templ'
>>> path = 'footempl.tmpl'
>>> path[:len('.tmpl')]
'foote'
>>> path[:-len('.tmpl')]
'footempl'

[dangra]

@dangra I would use .rpartition if there was some assurance about the filename .tmpl extension.
Since there is none, I switched to something equivalent of the previous regex
(and to what I think was intended on the original expression)

What I would do in order of preference is:
either always pass the path to the output file and let the func append the .tmpl extension where needed,
or assert inside the function body that the extension is there.

Should I go for the first one?

+1 to keep same behavior. Ignore the rpartition suggestion.

[Digenis]

What about striping the extension in the argument to the render_templatefile() call?
(as I said above)

[dangra]

by stripping in the caller you are changing the function public interface and that will lead to more discussions for a bugfix that is very simple to fix within the function.

[Digenis]

ok
I had greped just two calls to it in the codebase,
this is why I suggested so (because of the easiness).

[Digenis]

Is there something left to do (in the scope of this PR)?

[dangra]

what is the point of mocking open() when the test can use a real file?

[Digenis]

I thought this is how people mostly test side-effects.
Is there a different convention in scrapy?

[dangra]

Is there a different convention in scrapy?

for things that creates actual files we let them create temporal files.

[kmike]

+1 to create a real file; in my experience tests without mocks are easier to read and less likely to break (as they don't rely on implementation details). You can also pause the test and check if correct files exists manually. The downside is execution speed (creating/removing real files is slower), but it shouldn't be a problem in this test case.

[Digenis]

OK, fair enough, I'll use the test_dir.
To rebase or not to rebase?

[dangra]

+1 to rebase

[codecov-io]

Current coverage is 82.23%

Merging #1212 into master will not affect coverage as of d151499

@@            master   #1212   diff @@
======================================
  Files          165     165       
  Stmts         8172    8173     +1
  Branches      1133    1133       
  Methods          0       0       
======================================
+ Hit           6720    6721     +1
  Partial        262     262       
  Missed        1190    1190       

Review entire Coverage Diff as of d151499

Powered by Codecov. Updated on successful CI builds.

[Digenis]

Used tempfile and rebased.
Waiting for feedback.

[dangra]

looks good, squash?

[Digenis]

squahsed

[dangra]

thanks!