Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for the Parsel JMESPath feature #5915

Merged
merged 8 commits into from
May 3, 2023

Conversation

jxlil
Copy link
Contributor

@jxlil jxlil commented Apr 28, 2023

Added support for jmespath

Closes #5894

@codecov
Copy link

codecov bot commented Apr 28, 2023

Codecov Report

Merging #5915 (f5d0fac) into master (8c8fb67) will increase coverage by 0.03%.
The diff coverage is 100.00%.

❗ Current head f5d0fac differs from pull request most recent head a038faf. Consider uploading reports for the commit a038faf to get more accurate results

@@            Coverage Diff             @@
##           master    #5915      +/-   ##
==========================================
+ Coverage   88.77%   88.81%   +0.03%     
==========================================
  Files         162      162              
  Lines       11165    11181      +16     
  Branches     1814     1818       +4     
==========================================
+ Hits         9912     9930      +18     
+ Misses        966      965       -1     
+ Partials      287      286       -1     
Impacted Files Coverage Δ
scrapy/extensions/feedexport.py 95.63% <100.00%> (+0.11%) ⬆️
scrapy/http/response/__init__.py 100.00% <100.00%> (+2.56%) ⬆️
scrapy/http/response/text.py 100.00% <100.00%> (ø)
scrapy/responsetypes.py 94.28% <100.00%> (ø)
scrapy/signals.py 100.00% <100.00%> (ø)

@jxlil jxlil marked this pull request as ready for review April 29, 2023 18:58
@wRAR
Copy link
Member

wRAR commented Apr 30, 2023

The test on pinned envs failed because those use the old parsel version. We want to make sure that Scrapy and user code continue working with old parsel. Can you please check and fix those issues, probably just by marking the tests xfail?

@wRAR
Copy link
Member

wRAR commented Apr 30, 2023

I've checked the code and it doesn't make sure selector.jmespath() is available before calling it, so marking tests as xfail is wrong/not enough, you should add a check to the code and make new tests that test it. You can check scrapinghub/web-poet#167 for both the runtime check and tests implementations.

@jxlil
Copy link
Contributor Author

jxlil commented May 3, 2023

@wRAR I have updated the code, do you think it is necessary to make any other changes?

Copy link
Member

@Gallaecio Gallaecio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, nice work!

It would be great if we could also address #5903, as these 2 are the only Parsel-related changes blocking Scrapy 2.9, but we can handle that separately.

@wRAR
Copy link
Member

wRAR commented May 3, 2023

Thanks!

@wRAR wRAR merged commit ea15ff1 into scrapy:master May 3, 2023
@jxlil jxlil deleted the feature/support-parsel-jmespath branch May 3, 2023 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for the Parsel JMESPath feature
3 participants