Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Improve document about functions as processors #3224

Merged
merged 1 commit into from Apr 26, 2018

Conversation

@lucywang000
Copy link
Member

@lucywang000 lucywang000 commented Apr 19, 2018

I came across this earlier today, and after quite some (result-less) debugging, google led me to #2735 . Hopefully this improvement can help others avoid this pitfall.

@codecov
Copy link

@codecov codecov bot commented Apr 19, 2018

Codecov Report

Merging #3224 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3224   +/-   ##
=======================================
  Coverage   82.12%   82.12%           
=======================================
  Files         228      228           
  Lines        9593     9593           
  Branches     1385     1385           
=======================================
  Hits         7878     7878           
  Misses       1456     1456           
  Partials      259      259
@kmike kmike changed the title Improve document about functions as processors [MRG+1] Improve document about functions as processors Apr 25, 2018
@kmike
Copy link
Member

@kmike kmike commented Apr 25, 2018

Looks good, thanks @lucywang000!

I think in a long term we should support plain functions somehow. There was also a discussion about moving item loaders to a separate Python package.

@lopuhin lopuhin merged commit c4f096d into scrapy:master Apr 26, 2018
2 checks passed
2 checks passed
codecov/patch Coverage not affected when comparing da1256a...57b0e6b
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@lopuhin
Copy link
Member

@lopuhin lopuhin commented Apr 26, 2018

Thanks @lucywang000 !

@lucywang000 lucywang000 deleted the lucywang000:better-processors-doc branch Apr 26, 2018
@lucywang000
Copy link
Member Author

@lucywang000 lucywang000 commented Apr 26, 2018

Thanks for the review @kmike @lopuhin !

@IAlwaysBeCoding
Copy link
Contributor

@IAlwaysBeCoding IAlwaysBeCoding commented May 10, 2018

@kmike I think we should seriously have a third-party library that would have several processors that are highly needed. I've came up with several that I use on my own Scrapy projects.

Some processors that I have coded myself are:

  • Processor that will output a unique value for true and/or false if an element is found using an xpath/css.

  • Processor that will choose an n value from the extracted list, it's basically like a TakeFirst() but you can decide what n's index you want.

  • A TakeLast() processor as well.

  • Processor that will replace values

  • Processor that will split the output into values from a given delimeter.

and many more...

You can't really fully utlize the ItemLoader if you don't have a library of Processors, the default Processors that come with Scrapy left me no choice than to cook my own.

@kmike kmike added this to the v1.6 milestone Jul 4, 2018
@kmike kmike modified the milestones: v1.6, v1.5.1 Jul 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants