You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A better approach is use a HeadersProvider injected in config at processor startup.
If none provided, add a default one
If provided, use it
Some util providers:
DefaultAgentHeaderProvider
RandomAgentHeaderProvider - With a list of headers
FixedHeaderProvider - A fixed header (configured in the provider config)
Good to have:
A MultipleHeaderProvider that gets a list of headerproviders and joins all the results from each provider into a headers list for the request (may be for next Milestone)
Add useragent as HEADER
Configurable (see http://docs.python-requests.org/en/latest/api/ for request and optional parameters in the get method)
Add a constants list for the available agents
Default user agent (mozilla or whatever)
Random pick?
The text was updated successfully, but these errors were encountered: