-
Notifications
You must be signed in to change notification settings - Fork 11k
Extend Request.meta documentation #5565
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #5565 +/- ##
==========================================
+ Coverage 88.59% 88.75% +0.15%
==========================================
Files 159 162 +3
Lines 11582 10774 -808
Branches 1885 1843 -42
==========================================
- Hits 10261 9562 -699
+ Misses 994 939 -55
+ Partials 327 273 -54 |
docs/topics/request-response.rst
Outdated
:attr:`cb_kwargs` instead. | ||
|
||
To keep some data across 3 or more spider callbacks, however, request | ||
metadata may be the right choice. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, I think cb_kwargs also works ok in this case - you can pass something like partial_data
dict or list as a kwarg to all callbacks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The use case I am trying to describe here is where you pass some specific metadata key to almost every callback, and new future callbacks are likely to get it as well, but often most callbacks only pass it over to the next callback. So, for example, start_url
if you want to keep track of the root URL that originated every callback.
In those cases, adding the same parameter to every callback, specially to callbacks whose only function is to pass the parameter over to the next callback, may be too verbose, and using meta instead (and eventually STICKY_META_KEYS
, #4141, which is what inspired this paragraph) may be better.
I do need to come up with a better wording. Let me give it at least 1 more try.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 I left a comment, but feel free to merge without addressing it.
While reviewing #4141, I realized the existing
Request.meta
documentation could use some more information.