allowing to run .pyw files

[akshaysharmajs]

Referenced to issue #4643.
Also looking for other parts of code base for the mentioned issue.

Fixes #4643

[codecov]

Codecov Report

Merging #4646 into master will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4646      +/-   ##
==========================================
- Coverage   87.21%   87.19%   -0.03%     
==========================================
  Files         160      160              
  Lines        9813     9813              
  Branches     1447     1447              
==========================================
- Hits         8558     8556       -2     
- Misses        994      995       +1     
- Partials      261      262       +1     

Impacted Files	Coverage Δ
scrapy/commands/runspider.py	89.13% <100.00%> (ø)
scrapy/core/downloader/__init__.py	90.97% <0.00%> (-1.51%)	⬇️

[Gallaecio]

Could you include a test?

[akshaysharmajs]

Could you include a test?

Okay, I will add some test.

[akshaysharmajs]

scrapy\scrapy\commands\edit.py:`
36
37 sfile = sys.modules[spidercls.module].file
38: sfile = sfile.replace('.pyc', '.py')
39 self.exitcode = os.system('%s "%s"' % (editor, sfile))
40
scrapy\scrapy\commands\genspider.py:
99 spiders_module = None
100 spiders_dir = "."
101: spider_file = "%s.py" % join(spiders_dir, module)
102 shutil.copyfile(template_file, spider_file)
103 render_templatefile(spider_file, **tvars)

scrapy\scrapy\commands\startproject.py:
23: IGNORE = ignore_patterns('*.pyc', '.svn')

I searched code base and these are the concerned results. I don't think they require .pyw?Please check

[akshaysharmajs]

I m getting this error when runnig tox command and not able to test above changes. Tried installing 'itemadapter' again but error is still there.

[wRAR]

@akshaysharmajs try removing your .tox dir, this helps when the project requirements have changed.

[akshaysharmajs]

@akshaysharmajs try removing your .tox dir, this helps when the project requirements have change

Thanks! @wRAR

[akshaysharmajs]

Should I make test for .pyw files windows specific that will only run in windows. I tried mocking windows but '.pyw' doesn't support in linux. giving error (No module named 'myspider.pyw')

[Gallaecio]

I think we should refactor these tests.

I’m thinking we could have a base class that defines the tests and reads the filename (myspider.py or myspider.pyw) from a class variable, and then have 2 subclasses that simply define that variable, each with one of the values. See HttpProxyTestCase and its subclasses for an example of what I mean.

I would also have the tests that will also work out of Windows run in other OSes. Mostly because at the moment our CI does not run Windows tests. For the remaining tests, we can use skipif to skip them outside Windows.

[akshaysharmajs]

Thanks, I can try this implementation.

[akshaysharmajs]

Please check, I just used RunSpiderCommandTest as base class for .py and subclass WindowsRunSpiderCommandTest for '.pyw'.

[Gallaecio]

Looks good to me, thanks!

PS: The test failure is unrelated.

[akshaysharmajs]

Some more tests have been added that cause test failure.
Only these two are WindowsRunSpiderCommandTest.test_output and WindowsRunSpiderCommandTest.test_overwrite_output failing so I should skip them for unix platforms.

[kmike]

Nice, thanks @akshaysharmajs!

[akshaysharmajs]

😃 😃