Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Py3] Item loaders: allow single argument functions as processors #3899

Merged
merged 1 commit into from Nov 25, 2019

Conversation

elacuesta
Copy link
Member

@elacuesta elacuesta commented Jul 24, 2019

Python 3 only: __qualname__ was added in Python 3.3 (PEP 3155). I thought about a py2-compatible version using six.get_method_function (which accesses __func__ in py3 and im_func in py2) and doing some regex checks on the obtained function's name, but that wasn't pretty. Since we're dropping py2 support soon and this is technically backward incompatible I figured it would be ok to leave this for Scrapy 2.0

Initially reported in #2735

@codecov
Copy link

@codecov codecov bot commented Jul 24, 2019

Codecov Report

Merging #3899 into master will decrease coverage by 3.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3899      +/-   ##
==========================================
- Coverage   85.47%   82.42%   -3.05%     
==========================================
  Files         165      165              
  Lines        9624     9629       +5     
  Branches     1446     1447       +1     
==========================================
- Hits         8226     7937     -289     
- Misses       1145     1423     +278     
- Partials      253      269      +16
Impacted Files Coverage Δ
scrapy/loader/__init__.py 95.51% <100%> (+0.14%) ⬆️
scrapy/linkextractors/sgml.py 0% <0%> (-96.81%) ⬇️
scrapy/linkextractors/regex.py 0% <0%> (-95.66%) ⬇️
scrapy/linkextractors/htmlparser.py 0% <0%> (-92.07%) ⬇️
scrapy/core/downloader/handlers/s3.py 62.9% <0%> (-32.26%) ⬇️
scrapy/extensions/statsmailer.py 0% <0%> (-30.44%) ⬇️
scrapy/utils/boto.py 46.66% <0%> (-26.67%) ⬇️
scrapy/_monkeypatches.py 42.85% <0%> (-14.29%) ⬇️
scrapy/link.py 86.36% <0%> (-13.64%) ⬇️
scrapy/core/downloader/tls.py 75.92% <0%> (-12.97%) ⬇️
... and 20 more

@codecov
Copy link

@codecov codecov bot commented Jul 24, 2019

Codecov Report

Merging #3899 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3899      +/-   ##
==========================================
- Coverage   83.73%   83.73%   -0.01%     
==========================================
  Files         165      165              
  Lines        9708     9713       +5     
  Branches     1445     1446       +1     
==========================================
+ Hits         8129     8133       +4     
  Misses       1332     1332              
- Partials      247      248       +1
Impacted Files Coverage Δ
scrapy/loader/__init__.py 95.56% <100%> (+0.14%) ⬆️
scrapy/utils/trackref.py 83.78% <0%> (-2.71%) ⬇️

@elacuesta elacuesta added this to the v2.0 milestone Aug 29, 2019
@elacuesta elacuesta marked this pull request as ready for review Sep 2, 2019
scrapy/loader/__init__.py Outdated Show resolved Hide resolved
wRAR
wRAR approved these changes Nov 12, 2019
Copy link
Contributor

@wRAR wRAR left a comment

Looks like magic but it's an important improvement IMHO.
@elacuesta I guess you need to rebase or merge to get it to run the tests with the modern env list?

@elacuesta elacuesta force-pushed the py3_single_argument_processors branch from b3981d3 to 40b5cfc Compare Nov 22, 2019
@wRAR wRAR merged commit 8a1c996 into scrapy:master Nov 25, 2019
2 checks passed
@elacuesta elacuesta deleted the py3_single_argument_processors branch Nov 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants