Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feeds: Item Filters #5178

Merged
merged 11 commits into from
Jul 13, 2021
Merged

Feeds: Item Filters #5178

merged 11 commits into from
Jul 13, 2021

Conversation

drs-11
Copy link
Contributor

@drs-11 drs-11 commented Jun 11, 2021

Scrapy currently doesn't have any convenient APIs to customize conditions for item exports. An ItemChecker class can be used by the user to define constraints for acceptable items for particular feeds.

Resolves #5161, resolves #4576

@codecov
Copy link

codecov bot commented Jun 11, 2021

Codecov Report

Merging #5178 (f429936) into master (016c7e9) will decrease coverage by 4.02%.
The diff coverage is 97.05%.

❗ Current head f429936 differs from pull request most recent head 9dca6d0. Consider uploading reports for the commit 9dca6d0 to get more accurate results

@@            Coverage Diff             @@
##           master    #5178      +/-   ##
==========================================
- Coverage   88.19%   84.16%   -4.03%     
==========================================
  Files         162      162              
  Lines       10497    10523      +26     
  Branches     1517     1522       +5     
==========================================
- Hits         9258     8857     -401     
- Misses        965     1406     +441     
+ Partials      274      260      -14     
Impacted Files Coverage Δ
scrapy/core/engine.py 84.21% <87.50%> (+0.24%) ⬆️
scrapy/extensions/feedexport.py 91.63% <100.00%> (-3.69%) ⬇️
scrapy/utils/signal.py 97.91% <100.00%> (+0.04%) ⬆️
scrapy/core/http2/stream.py 27.01% <0.00%> (-64.37%) ⬇️
scrapy/pipelines/images.py 28.07% <0.00%> (-62.29%) ⬇️
scrapy/core/http2/agent.py 36.14% <0.00%> (-60.25%) ⬇️
scrapy/core/downloader/handlers/http2.py 43.42% <0.00%> (-56.58%) ⬇️
scrapy/core/http2/protocol.py 34.17% <0.00%> (-49.25%) ⬇️
scrapy/utils/ssl.py 53.65% <0.00%> (-17.08%) ⬇️
scrapy/utils/asyncgen.py 83.33% <0.00%> (-16.67%) ⬇️
... and 9 more

@Gallaecio Gallaecio marked this pull request as ready for review June 18, 2021 13:55
Copy link
Member

@Gallaecio Gallaecio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, great job!

@Gallaecio Gallaecio self-requested a review June 18, 2021 13:56
@Gallaecio
Copy link
Member

Sorry, I went ahead of myself. We still need documentation changes (see #4576).

@drs-11 drs-11 requested a review from Gallaecio June 19, 2021 15:52
Comment on lines +934 to +937
items = [
self.MyItem({'foo': 'bar1', 'egg': 'spam1'}),
self.MyItem2({'hello': 'world2', 'foo': 'bar2'}),
{'hello': 'world3', 'egg': 'spam3'},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case you plan to use these inputs for more tests, can we declare these as constants at the top?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep this in mind next time I add tests.

@drs-11 drs-11 changed the title [WIP] Feeds: Item Filters Feeds: Item Filters Jun 21, 2021
@drs-11 drs-11 requested a review from Gallaecio June 22, 2021 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feeds Enhancement: Item Filters
4 participants