Wrapper around ibrowse, simple api to use random proxies and user agents and easily create clients for HTTP apis
Erlang
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
priv
src
.gitignore
Emakefile
LICENSE
Makefile
README.md
rebar
rebar.config

README.md

About

A simple erlang application to manage http requests. Will optionally choose a proxy and a user agent at random from the given lists. It's also very useful to create clients for http apis, such as ElasticSearch.

How to use it

  • Start the main supervisor egetter_sup from your own supervisor tree.

  • Configure the application, here's a sample:

{egetter, [
  {user_agents, "/etc/egetter/priv/agents.txt"},
  {proxies, "/etc/egetter/proxies.txt"}
]}
  • Both are optional. Egetter will not try to use a proxy or choose a user-agent if the corresponding option is not set.

  • You will need a list of proxies and a list of user agents. There are examples in the priv directory

  • Call egetter:req/1 passing in a list of options. You will get a result(), which is one of:

    • {ibrowse_error, term()}: on "hard" errors returned by ibrowse.
    • {error, Status::pos_integer(), Headers::[{string(), string()}], Body::binary()}: When the request fails (i.e: status code not 2xx/3xx).
    • {ok, Status::pos_integer(), Headers::[{string(), string()}], Body::binary()}: When the result is 2xx or 3xx.
  • Available options:

    • {url, string()} Either specify this option OR all of:
      • {host, string()}
      • {port, pos_integer()},
      • {scheme, string()} e.g: "http"
      • {path_components, [string()]}
    • {timeout, pos_integer()}
    • {headers, [{string()|atom(), string()}]}
    • {body, binary()}
    • {method, get | post | put | delete | head | options}
    • {follow_redirect, true | false} (defaults to false)
    • {ibrowse_options, [{atom(), term()}]}
    • {use_proxy, true | false} (defaults to false)
    • {query_string, query_string()} e.g: [{"key1", "value1"}], keys and values are automatically urlencoded.

Examples

egetter:req([{url, "http://www.google.com/"}, {follow_redirect, true}, {query_string, [{"key", " value"}]}]).
egetter:req([{host, "google.com"}, {port, 80}, {scheme, "http"}, {path_components, ["some", "path"]}]}]).