Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Improve document about functions as processors #3224

merged 1 commit into from Apr 26, 2018


Copy link

I came across this earlier today, and after quite some (result-less) debugging, google led me to #2735 . Hopefully this improvement can help others avoid this pitfall.

Copy link

codecov bot commented Apr 19, 2018

Codecov Report

Merging #3224 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3224   +/-   ##
  Coverage   82.12%   82.12%           
  Files         228      228           
  Lines        9593     9593           
  Branches     1385     1385           
  Hits         7878     7878           
  Misses       1456     1456           
  Partials      259      259

@kmike kmike changed the title Improve document about functions as processors [MRG+1] Improve document about functions as processors Apr 25, 2018
Copy link

kmike commented Apr 25, 2018

Looks good, thanks @lucywang000!

I think in a long term we should support plain functions somehow. There was also a discussion about moving item loaders to a separate Python package.

@lopuhin lopuhin merged commit c4f096d into scrapy:master Apr 26, 2018
Copy link

lopuhin commented Apr 26, 2018

Thanks @lucywang000 !

@lucywang000 lucywang000 deleted the better-processors-doc branch April 26, 2018 09:40
Copy link
Member Author

Thanks for the review @kmike @lopuhin !

Copy link

IAlwaysBeCoding commented May 10, 2018

@kmike I think we should seriously have a third-party library that would have several processors that are highly needed. I've came up with several that I use on my own Scrapy projects.

Some processors that I have coded myself are:

  • Processor that will output a unique value for true and/or false if an element is found using an xpath/css.

  • Processor that will choose an n value from the extracted list, it's basically like a TakeFirst() but you can decide what n's index you want.

  • A TakeLast() processor as well.

  • Processor that will replace values

  • Processor that will split the output into values from a given delimeter.

and many more...

You can't really fully utlize the ItemLoader if you don't have a library of Processors, the default Processors that come with Scrapy left me no choice than to cook my own.

@kmike kmike added this to the v1.6 milestone Jul 4, 2018
@kmike kmike modified the milestones: v1.6, v1.5.1 Jul 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

4 participants