You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Allows people to specify headers in configuration options for crawler
* Added Github identifier to User-Agent.
Co-authored-by: Nick Georgiou <nickgeorgi@gmail.com>
Description
We should allow for custom request headers in all web requests, allowing authenticated access (cookies) and more complex requirements.
Proposed solution
Extend options to support array of request headers in a key/value format
Additional context
None
The text was updated successfully, but these errors were encountered: