You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I find myself doing a lot of scrapy parse when testing out my parsing methods on the spiders that I develop.
However, there is this one thing that I really hate that scrapy doesn't have.
The ability to insert meta fields coming from the response.meta when using it from the scrapy parse command.
For example, say I'm scraping an ecommerce site.
A method such as parse_product_detail_page can have the category_id meta key coming from the previous Request which is not available unless scrapy crawl was used.
I'm doing a lot of commenting out my parse functions like this
All because I cannot pass category_id into meta when calling from scrapy parse.
The text was updated successfully, but these errors were encountered:
IAlwaysBeCoding
changed the title
Feature: add the ability to insert meta fields when calling scrapy parse
Feature Suggest: add the ability to insert meta fields when calling scrapy parseAug 14, 2017
IAlwaysBeCoding
changed the title
Feature Suggest: add the ability to insert meta fields when calling scrapy parse
Feature Suggest: add the ability to insert meta fields when calling from scrapy parseAug 14, 2017
Hey @IAlwaysBeCoding , I also had this experience in the past.
It would be a great addition for scrapy parse;
One thing to design right from the start though: it should also allow more complex data structures, like dicts (as items are often passed in request.meta) or lists. There were some discussions on passing JSON on the command line that may be relevant here too.
... and in fact, there's an old issue already about this for parse command.
I find myself doing a lot of
scrapy parse
when testing out my parsing methods on the spiders that I develop.However, there is this one thing that I really hate that
scrapy
doesn't have.The ability to insert meta fields coming from the response.meta when using it from the
scrapy parse
command.For example, say I'm scraping an ecommerce site.
A method such as
parse_product_detail_page
can have thecategory_id
meta key coming from the previous Request which is not available unlessscrapy crawl
was used.I'm doing a lot of commenting out my parse functions like this
All because I cannot pass
category_id
into meta when calling fromscrapy parse
.The text was updated successfully, but these errors were encountered: