Drop Python 3.6 support

[Laerte]

Closes #5512

[codecov]

Codecov Report

Merging #5514 (b6c4a38) into master (bafe874) will decrease coverage by 0.07%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #5514      +/-   ##
==========================================
- Coverage   88.72%   88.64%   -0.08%     
==========================================
  Files         163      162       -1     
  Lines       10671    10667       -4     
  Branches     1819     1819              
==========================================
- Hits         9468     9456      -12     
- Misses        927      937      +10     
+ Partials      276      274       -2     

Impacted Files	Coverage Δ
scrapy/__init__.py	84.21% <0.00%> (ø)
scrapy/utils/ssl.py	53.65% <0.00%> (-17.08%)	⬇️
scrapy/core/scraper.py	85.54% <0.00%> (-2.32%)	⬇️
scrapy/crawler.py	87.56% <0.00%> (-1.09%)	⬇️
scrapy/pipelines/images.py	94.78% <0.00%> (+4.34%)	⬆️

[elacuesta]

Hi, thanks for the update. We still need the typing, pypy and pinned envs though, we need to update their versions instead of removing them completely.

[Laerte]

@elacuesta Sure! I will change to use pypy version 3.7, what you think?

[elacuesta]

Indeed that works, although I don't think we have a strong requirement for pypy to be any specific major version. That's not the case for the pinned one, ideally it should be the earliest version in the 3.7 series (that's supported by CI).

[wRAR]

Looks like pinned lxml is too old for 3.7?

[Laerte]

@wRAR Yeah, i will take a look later.

[Laerte]

@wRAR @elacuesta Had to bump some dependencies in order to make everything work with pinned 3.7. Don't know why codecov & checks (3.10, docs) are failing but is not related with the changes that i made.

[harivamsi9]

was anyone able to figure out why codecov/path check is failling?

[elacuesta]

The docs env is failing in master too (517cbc8), I wouldn't worry about it for this PR. Regarding the coverage check, I suppose it could be possible that parts of the codebase are not reached now with the new requirement versions, but I'm not completely sure that's the case.

We need to update install_requires too, if we know that some versions are not compatible anymore with the earliest Python version we support. For instance, I suppose that if you needed to update Twisted in the pinned env it means that Twisted==17.9.0 is no longer valid (under py37 anyway), so it should be updated in setup.py.

[Laerte]

Some lessons learned with this PR until now:

Earliest supported release for pypy3.7 is v7.3.5, reason:
https://foss.heptapod.net/pypy/pypy/-/issues/3400

Bumped Twisted version, reason:

• Fix syntax error under Python 3.7 for async param twisted/twisted#966

Bumped cryptography and pyOpenSSL, because the test below was failing:
tests/test_webclient.py::WebClientCustomCiphersSSLTestCase::testPayloadDisabledCipher

[Gallaecio]

was anyone able to figure out why codecov/path check is failling?

Looking at https://app.codecov.io/gh/scrapy/scrapy/compare/5514/diff, that is an expected non-covered path (we decided long ago not to run a CI job just to verify that Scrapy fails nicely on an unsupported Python version).

[Gallaecio]

Maybe we should also remove scrapy/utils/py36.py here, it has been more than a year since it was first deprecated (Scrapy 2.5).

[Gallaecio]

Great job! I bet you did not anticipate it to be this involved 🙂

[Gallaecio]

(scrapy/utils/py36.py is still there, but I don’t think it needs to be handled as part of this pull request, is more of a deprecation cleanup task)

[Laerte]

Great job! I bet you did not anticipate it to be this involved 🙂

You bet is right. 😅

[Laerte]

@elacuesta Can you take a look?

[elacuesta]

Thanks!