Use getfullargspec under the scenes for py3 to stop DeprecationWarning

[andreip]

fixes #2862

I also ran tox for the tests, but there are slight differences for the packages used inside the library, so in tests/requirements.txt:

• netlib can be easily updated to 0.17 to support both py2 and py3
• mitmproxy on the other hand can't w/o some modifications; its version 0.18+ that supports both py3 and py2 doesn't contain libmproxy that's used in tests/test_proxy_connect.py. So yea, related to enable test_proxy_connect in Python 3 #2545 actually, so nevermind.

Other than that issue, everything worked fine, no errors for py2 nor py3.

Also I tested the branch in our project and the DeprecationWarning has disappeared now :).

[codecov-io]

Codecov Report

Merging #2864 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #2864      +/-   ##
=========================================
+ Coverage   84.69%   84.7%   +<.01%     
=========================================
  Files         164     164              
  Lines        9189    9192       +3     
  Branches     1369    1370       +1     
=========================================
+ Hits         7783    7786       +3     
  Misses       1154    1154              
  Partials      252     252

Impacted Files	Coverage Δ
scrapy/utils/python.py	84.35% <100%> (+0.26%)	⬆️

[kmike]

What do you think about creating an utility function and using it, without patching inspect module?

[andreip]

any name suggestion? getargspec_py23?

[redapple]

If I recall, @kmike prefers if not six.PY2

[dangra]

I would go the other way around:

if six.PY2:
    return inspect.getargspec(func)

return inspect.ArgSpec(*inspect.getfullargspec(func)[:4])

[dangra]

And I would make this function private, so we reserve the right to drop it if not needed by other utils in this module.

[andreip]

ok, makes sense to do that as long as usage of inspect.getargspec stays within the utils.python module, as is now. If at some point inspect.getargspec gets used outside, it probably makes sense to do from scrapy.utils.python import getargspec instead of from inspect import getargspec to be able to use that for both py2 and py3 without triggering deprecation warning.

[dangra]

thanks!