[MRG+1] Drop py34 support - Update CI envs

[elacuesta]

Based on #3825 (comment) by @dangra

I personally would like to see this because type hints were introduced in py35, and I'm currently working on #3881 (although it seems like variable annotations were introduced in 3.6 😿 - PEP 526)

[codecov]

Codecov Report

Merging #3892 into master will decrease coverage by 3.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3892      +/-   ##
==========================================
- Coverage   85.56%   82.55%   -3.02%     
==========================================
  Files         164      164              
  Lines        9565     9565              
  Branches     1435     1435              
==========================================
- Hits         8184     7896     -288     
- Misses       1133     1409     +276     
- Partials      248      260      +12

Impacted Files	Coverage Δ
scrapy/linkextractors/sgml.py	0% <0%> (-96.81%)	⬇️
scrapy/linkextractors/regex.py	0% <0%> (-95.66%)	⬇️
scrapy/linkextractors/htmlparser.py	0% <0%> (-92.07%)	⬇️
scrapy/core/downloader/handlers/s3.py	62.9% <0%> (-32.26%)	⬇️
scrapy/extensions/statsmailer.py	0% <0%> (-30.44%)	⬇️
scrapy/utils/boto.py	46.66% <0%> (-26.67%)	⬇️
scrapy/_monkeypatches.py	42.85% <0%> (-14.29%)	⬇️
scrapy/link.py	86.36% <0%> (-13.64%)	⬇️
scrapy/core/downloader/tls.py	77.5% <0%> (-12.5%)	⬇️
scrapy/utils/gz.py	91.89% <0%> (-8.11%)	⬇️
... and 17 more

[codecov]

Codecov Report

Merging #3892 into master will decrease coverage by 0.58%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3892      +/-   ##
==========================================
- Coverage   85.54%   84.96%   -0.59%     
==========================================
  Files         166      166              
  Lines        9681     9681              
  Branches     1445     1445              
==========================================
- Hits         8282     8225      -57     
- Misses       1146     1194      +48     
- Partials      253      262       +9

Impacted Files	Coverage Δ
scrapy/core/downloader/handlers/s3.py	62.9% <0%> (-32.26%)	⬇️
scrapy/utils/boto.py	46.66% <0%> (-26.67%)	⬇️
scrapy/core/downloader/tls.py	75.92% <0%> (-12.97%)	⬇️
scrapy/utils/ssl.py	47.22% <0%> (-5.56%)	⬇️
scrapy/extensions/feedexport.py	78.44% <0%> (-5.05%)	⬇️
scrapy/core/downloader/handlers/http11.py	89.92% <0%> (-2.62%)	⬇️
scrapy/core/scraper.py	86.48% <0%> (-2.03%)	⬇️
scrapy/pipelines/files.py	65.38% <0%> (-1.16%)	⬇️

[elacuesta]

The py2 tests are failing due to a problem in a recent docutils upgrade: https://sourceforge.net/p/docutils/bugs/365/.
I don't think it's worth pinning the docutils dependency unless they don't fix it upstream quickly, which I think it should probably happen given the amount of builds that are probably failing because of that.
I deleted my previous comments about it to avoid cluttering the PR.

[kmike]

hey @elacuesta! I think it makes sense to do this as a part of dropping support for old OS, setting a new baseline Ubuntu/Debian. We need to decide what's the baseline, and then bump minimum package requirements and minimum Python version based on that.

[kmike]

My proposal:

• Ubuntu baseline: 16.04 xenial. It is LTS release; Python 3.5.1, Twisted 16.0 (https://launchpad.net/ubuntu/xenial/+package/python3-twisted), lxml 3.5.
• Debian baseline: stretch. It has Python 3.5.3, Twisted 16.6, lxml 3.7.1.

Ubuntu 14.04 trusty in only supported for paid customers (if I read announcements correctly), and it uses Python 3.4, twisted 13.2.0 (no Python 3 support), lxml 3.3.3.

Debian jessie has LTS support until mid-2020. But it is also getting old (Python 3.4.2, Twisted 14.0.2 without Python 3 support, lxml 3.4).

Is it too aggressive? //cc @dangra

[elacuesta]

Xenial sounds good to me 👍
According to the Travis docs Trusty is the default distribution, and it's the one we're currently using for all builds except for the Python 3.7 one.
Excuse my ignorance, but I'm failing to see how the Debian tests are executed. There is no Debian option in the docs I mentioned, and in fact the builds with TOXENV=jessie have Build dist: trusty under the Build system information section at the beginning of the logs.

[wRAR]

@elacuesta jessie is emulated by installing the specific versions of modules into a trusty system.

[wRAR]

There is a CI on https://salsa.debian.org, I planned to learn it and maybe set up something automatical one day.

[elacuesta]

@elacuesta jessie is emulated by installing the specific versions of modules into a trusty system.

Oh nice, that's what I suspected but I wasn't sure. Thanks!

[kmike]

@elacuesta: we agreed to use Ubuntu 16.04 and Debian stretch as a new baseline for the 1.8 release. Feel free to update tox environments, requirements, Python versions, etc. to match it :)

[elacuesta]

Hello everyone. I updated this PR to add an environment with pinned versions that are known to work well. I'm only running it on py36, I think that's probably the most reasonable compromise to avoid adding many more envs to the Travis matrix, specially now that we have two more for the Robots parsing. I updated the requirements-py3.txt file with a few comments explaining the reasons for the picked versions.
To be honest I didn't pay much attention to the py2 tests, I'm not sure it's worth at this point.
Let me know what you think, and as usual, thanks for your comments 🚀

[kmike]

Thanks @elacuesta!

I'm only running it on py36

Why not 3.5? Oldest Python, oldest supported versions of dependencies?

To be honest I didn't pay much attention to the py2 tests, I'm not sure it's worth at this point.

I think it worths it, as we may end up supporting 1.8 a bit longer than other versions. There is also a different Twisted minimum version requirement in setup.py; if we're not bumping it to match Python 3, it would be good to have a py27-pinned-versions env as well.

[elacuesta]

Hey @kmike, I made some updates based on your feedback. Thanks!

[Gallaecio]

Looks good. I’ve left a few minor comments regarding pinned versions.

[elacuesta]

Updated, added minimum and pinned versions of cryptography and zope.interface. botocore and Pillow are only installed for the tests, as they are only required by certain parts (S3 feed storage and images pipeline IIRC) and are not required for Scrapy itself to run.

[elacuesta]

Random Travis error about not being able to install tox in one of the py27 envs. All other tests are green, closing and reopening to trigger another build.

[kmike]

Thanks @elacuesta! Farewell Python 3.4!