Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add user agent in request for HtmlProcessor #20

Closed
davidfrigola opened this issue Jan 11, 2014 · 2 comments
Closed

Add user agent in request for HtmlProcessor #20

davidfrigola opened this issue Jan 11, 2014 · 2 comments
Assignees
Milestone

Comments

@davidfrigola
Copy link
Owner

Add useragent as HEADER
Configurable (see http://docs.python-requests.org/en/latest/api/ for request and optional parameters in the get method)

Add a constants list for the available agents
Default user agent (mozilla or whatever)
Random pick?

@ghost ghost assigned davidfrigola Jan 11, 2014
@davidfrigola
Copy link
Owner Author

A better approach is use a HeadersProvider injected in config at processor startup.
If none provided, add a default one
If provided, use it
Some util providers:

  • DefaultAgentHeaderProvider
  • RandomAgentHeaderProvider - With a list of headers
  • FixedHeaderProvider - A fixed header (configured in the provider config)

Good to have:
A MultipleHeaderProvider that gets a list of headerproviders and joins all the results from each provider into a headers list for the request (may be for next Milestone)

@davidfrigola
Copy link
Owner Author

Added a __getHeaders methon in the HtmlProcessor, just for useragent.
Related with #25 for next milestone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant