Support Python 3.9

[itsvinayak]

Resolves #4757

[Gallaecio]

I’m thinking it would be best to:

• Switch the 3 Python 3.8 jobs at the end to Python 3.9, as those jobs are meant to make sure that everything works with the latest Python.
• Add a 3.8 job right after the 3.7 one (one identical to the current PYPI_RELEASE_JOB one).
• Remove PYPI_RELEASE_JOB from the 3.9 job, there must only be one PYPI_RELEASE_JOB job and an unreleased Python version is probably not the best choice to build our PyPI packages.

[Gallaecio]

I don’t think I explained myself very well, sorry. I meant the 3 Python jobs at the end of the whole job list:

    - env: TOXENV=py PYPI_RELEASE_JOB=true
      python: 3.8
      dist: bionic
    - env: TOXENV=extra-deps
      python: 3.8	      
      dist: bionic	      
    - env: TOXENV=asyncio	    
      python: 3.8	      
      dist: bionic	      

Please re-read my previous comment taking that into account, hopefully it makes more sense.

As for the static tests (security, flake, etc.), it probably makes sense to upgrade them as you did, even though I did not had that among my goals originally. However, please keep their current order, even if the docs one stays at 3.7. And switch pylint also to 3.9; I know it failed, we’ll have to figure out why and fix that.

[itsvinayak]

@Gallaecio ,

matrix:
  include:
    - env: TOXENV=py 
      python: 3.9-dev
      dist: bionic
    - env: TOXENV=extra-deps
      python: 3.9-dev
      dist: bionic
    - env: TOXENV=asyncio
      python: 3.9-dev
      dist: bionic
    - env: TOXENV=security
      python: 3.9-dev
    - env: TOXENV=flake8
      python: 3.9-dev
    - env: TOXENV=typing
      python: 3.9-dev
      
    - env: TOXENV=docs
      python: 3.7  # Keep in sync with .readthedocs.yml

    - env: TOXENV=pinned
      python: 3.6.1
    - env: TOXENV=asyncio-pinned
      python: 3.6.1
    - env: TOXENV=pypy3-pinned PYPY_VERSION=3.6-v7.2.0

    - env: TOXENV=py
      python: 3.6
    - env: TOXENV=pypy3 PYPY_VERSION=3.6-v7.3.1

    - env: TOXENV=py
      python: 3.7
    - env: TOXENV=py PYPI_RELEASE_JOB=true
      python: 3.8
      dist: bionic

does this make sense?

[Gallaecio]

Other than a minor 💄 aesthetic thing, the Travis CI looks good to me. Let’s now look at the issues.

We can either try to solve them here, if they are trivial, or open separate issues for them, and hold this pull request until those are fixed.

[Gallaecio]

The 3 failing tests seem caused by a bug in passlib, which will be fixed in their next version (1.7.3). We need to wait for them to release that version.

[Gallaecio]

They’ve released a new passlib version, it should be possible to move this forward.

[Gallaecio]

Now the issue is a bug in mitmproxy. It looks like we need to wait for a release including mitmproxy/mitmproxy@8e5e43d.

[wRAR]

We decided that it's fine to put the unreleased mitmproxy commit into the reqs, as it's only required for tests (it can be also marked as only required for Python 3.9)

[wRAR]

src/twisted/internet/iocpreactor/iocpsupport/iocpsupport.c(7614): error C2039: 'tp_print': is not a member of '_typeobject'

In the Windows tests.

[wRAR]

And a similar error in the pylint job (?!)

[Gallaecio]

Reppy does not support Python 3.9 at the moment: seomoz/reppy#122

[wRAR]

But the Windows error seems to be about Twisted (and a Windows-specific file)

[Gallaecio]

Indeed, https://twistedmatrix.com/trac/ticket/9990

[wRAR]

Looks like mitmproxy 5.3.0 includes the 3.9 fix. Reppy is still not fixed though.

[kmike]

I think that's fine to not install reppy in py39-extra-deps environment. It is not a required dependency; if it doesn't work, and the problem is upstream, that's fine to skip the test configuration on our side.

That said, we shouldn't be fixing it in this PR, as we're moving to Github Actions due to recent Travis CI changes (see #4924)

[codecov]

Codecov Report

Merging #4759 (b5de69a) into master (904a501) will increase coverage by 0.20%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4759      +/-   ##
==========================================
+ Coverage   87.81%   88.02%   +0.20%     
==========================================
  Files         158      158              
  Lines        9719     9719              
  Branches     1433     1433              
==========================================
+ Hits         8535     8555      +20     
+ Misses        929      910      -19     
+ Partials      255      254       -1     

Impacted Files	Coverage Δ
scrapy/core/downloader/__init__.py	92.48% <0.00%> (+1.50%)	⬆️
scrapy/robotstxt.py	97.53% <0.00%> (+22.22%)	⬆️