Skip to content

Feature Suggest: add the ability to insert meta fields when calling from scrapy parse #2883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
IAlwaysBeCoding opened this issue Aug 14, 2017 · 1 comment · Fixed by #2957

Comments

@IAlwaysBeCoding
Copy link
Contributor

I find myself doing a lot of scrapy parse when testing out my parsing methods on the spiders that I develop.

However, there is this one thing that I really hate that scrapy doesn't have.

The ability to insert meta fields coming from the response.meta when using it from the scrapy parse command.

For example, say I'm scraping an ecommerce site.

A method such as parse_product_detail_page can have the category_id meta key coming from the previous Request which is not available unless scrapy crawl was used.

I'm doing a lot of commenting out my parse functions like this

#category_id = response.meta.get('category_id', None)
category_id = 3183

All because I cannot pass category_id into meta when calling from scrapy parse.

@IAlwaysBeCoding IAlwaysBeCoding changed the title Feature: add the ability to insert meta fields when calling scrapy parse Feature Suggest: add the ability to insert meta fields when calling scrapy parse Aug 14, 2017
@IAlwaysBeCoding IAlwaysBeCoding changed the title Feature Suggest: add the ability to insert meta fields when calling scrapy parse Feature Suggest: add the ability to insert meta fields when calling from scrapy parse Aug 14, 2017
@redapple
Copy link
Contributor

redapple commented Aug 23, 2017

Hey @IAlwaysBeCoding , I also had this experience in the past.
It would be a great addition for scrapy parse;
One thing to design right from the start though: it should also allow more complex data structures, like dicts (as items are often passed in request.meta) or lists. There were some discussions on passing JSON on the command line that may be relevant here too.
... and in fact, there's an old issue already about this for parse command.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants