Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a FAQ entry about splitting items in item pipelines #3672

Merged
merged 1 commit into from Jul 8, 2019

Conversation

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Mar 8, 2019

Fixes #2240

@Gallaecio Gallaecio closed this Mar 8, 2019
@Gallaecio Gallaecio reopened this Mar 8, 2019
@codecov
Copy link

@codecov codecov bot commented Mar 8, 2019

Codecov Report

Merging #3672 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3672   +/-   ##
=======================================
  Coverage   84.52%   84.52%           
=======================================
  Files         167      167           
  Lines        9410     9410           
  Branches     1397     1397           
=======================================
  Hits         7954     7954           
  Misses       1199     1199           
  Partials      257      257

@codecov
Copy link

@codecov codecov bot commented Mar 8, 2019

Codecov Report

Merging #3672 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3672   +/-   ##
=======================================
  Coverage   85.59%   85.59%           
=======================================
  Files         164      164           
  Lines        9545     9545           
  Branches     1430     1430           
=======================================
  Hits         8170     8170           
  Misses       1128     1128           
  Partials      247      247

docs/faq.rst Outdated

def process_spider_output(self, response, result, spider):
for item in result:
if isinstance(item, scrapy.Item):
Copy link
Member

@kmike kmike Jun 25, 2019

I think this is slightly more precise:

Suggested change
if isinstance(item, scrapy.Item):
if isinstance(item, (scrapy.item.BaseItem, dict)):

(it'd be good to add necessary imports as well)

docs/faq.rst Outdated
for item in result:
if isinstance(item, scrapy.Item):
for _ in range(item['multiply_by']):
yield item.copy()
Copy link
Member

@kmike kmike Jun 25, 2019

What do you think about using deepcopy.copy(item) here, to avoid surprises?
We can't use item.deepcopy() because item can be a dict.

@Gallaecio
Copy link
Member Author

@Gallaecio Gallaecio commented Jul 8, 2019

Done.

I’ve also included the import statements, just in case, and made the question slightly shorter (“how can I” → “how to”).

@kmike
Copy link
Member

@kmike kmike commented Jul 8, 2019

Looks good, thanks @Gallaecio!

@kmike kmike merged commit dda12b6 into scrapy:master Jul 8, 2019
3 checks passed
@kmike kmike added this to the v1.7 milestone Jul 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

2 participants