Permalink
Browse files

Lots of new features

  • Loading branch information...
1 parent 563afd6 commit c784ab8ded409bcf6f4a310597e7dabc04fbcbcd chandrusf committed Mar 27, 2008
View
@@ -1,4 +1,4 @@
-$Id$
+$Id: README,v 1.15 2008/03/27 01:35:49 chandrusf Exp $
ibrowse is a HTTP client. The following are a list of features.
- RFC2616 compliant (AFAIK)
@@ -7,7 +7,7 @@ ibrowse is a HTTP client. The following are a list of features.
- Understands HTTP/0.9, HTTP/1.0 and HTTP/1.1
- Understands chunked encoding
- Can generate requests using Chunked Transfer-Encoding
- - Named pools of connections to each webserver
+ - Pools of connections to each webserver
- Pipelining support
- Download to file
- Asynchronous requests. Responses are streamed to a process
@@ -22,6 +22,26 @@ Comments to : Chandrashekhar.Mullaparthi@t-mobile.co.uk
CONTRIBUTIONS & CHANGE HISTORY
==============================
+27-03-2008 - * Major rewrite of the load balancing feature. Additional module,
+ ibrowse_lb.erl, introduced to achieve this.
+ * Can now get a handle to a connection process which is not part of
+ the load balancing pool. Useful when an application is making
+ requests to a webserver which are time consuming (such as
+ uploading a large file). Such requests can be put on a separate
+ connection, and all other smaller/quicker requests can use the
+ load balancing pool. See ibrowse:spawn_worker_process/2 and
+ ibrowse:spawn_link_worker_process/2
+ * Ram Krishnan sent a patch to enable a client to send a lot of
+ data in a request by providing a fun which is invoked by the
+ connection handling process. This fun can fetch the data from
+ any where. This is useful when trying to upload a large file
+ to a webserver.
+ * Use the TCP_NODELAY option on every socket by default
+ * Rudimentary support for load testing of ibrowse. Undocumented,
+ but see ibrowse_test:load_test/3. Use the source, Luke!
+ * New function ibrowse:show_dest_status/2 to view state of
+ connections/pipelines to a web server
+
20-02-2008 - Ram Krishnan sent another patch for another hidden bug in the
save_response_to_file feature.
@@ -10,15 +10,15 @@
<h1>Module ibrowse</h1>
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>The ibrowse application implements an HTTP 1.1 client.
-<p>Copyright © 2005-2007 Chandrashekhar Mullaparthi</p>
+<p>Copyright © 2005-2008 Chandrashekhar Mullaparthi</p>
-<p><b>Version:</b> 1.2.7</p>
+<p><b>Version:</b> 1.4</p>
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
<p><b>Authors:</b> Chandrashekhar Mullaparthi (<a href="mailto:chandrashekhar dot mullaparthi at gmail dot com"><tt>chandrashekhar dot mullaparthi at gmail dot com</tt></a>).</p>
<h2><a name="description">Description</a></h2><p>The ibrowse application implements an HTTP 1.1 client. This
module implements the API of the HTTP client. There is one named
-process called 'ibrowse' which acts as a load balancer. There is
+process called 'ibrowse' which assists in load balancing and maintaining configuration. There is one load balancing process per unique webserver. There is
one process to handle one TCP connection to a webserver
(implemented in the module ibrowse_http_client). Multiple connections to a
webserver are setup based on the settings for each webserver. The
@@ -47,10 +47,6 @@
{save_response_to_file, true}], 1000).
<br><br>
- ibrowse:set_dest("www.hotmail.com", 80, [{max_sessions, 10},
- {max_pipeline_size, 1}]).
- <br><br>
-
ibrowse:send_req("http://www.erlang.org", [], head).
<br><br>
@@ -69,21 +65,41 @@
driver isn't actually used.</p>
<h2><a name="index">Function Index</a></h2>
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
-<tr><td valign="top"><a href="#finished_async_request-0">finished_async_request/0</a></td><td>Internal export.</td></tr>
+<tr><td valign="top"><a href="#get_config_value-1">get_config_value/1</a></td><td>Internal export.</td></tr>
+<tr><td valign="top"><a href="#get_config_value-2">get_config_value/2</a></td><td>Internal export.</td></tr>
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
-<tr><td valign="top"><a href="#reply-2">reply/2</a></td><td>Internal export.</td></tr>
+<tr><td valign="top"><a href="#rescan_config-0">rescan_config/0</a></td><td>Clear current configuration for ibrowse and load from the file
+ ibrowse.conf in the IBROWSE_EBIN/../priv directory.</td></tr>
+<tr><td valign="top"><a href="#rescan_config-1">rescan_config/1</a></td><td></td></tr>
<tr><td valign="top"><a href="#send_req-3">send_req/3</a></td><td>This is the basic function to send a HTTP request.</td></tr>
<tr><td valign="top"><a href="#send_req-4">send_req/4</a></td><td>Same as send_req/3.</td></tr>
<tr><td valign="top"><a href="#send_req-5">send_req/5</a></td><td>Same as send_req/4.</td></tr>
<tr><td valign="top"><a href="#send_req-6">send_req/6</a></td><td>Same as send_req/5.</td></tr>
-<tr><td valign="top"><a href="#set_dest-3">set_dest/3</a></td><td>Sets options for a destination.</td></tr>
-<tr><td valign="top"><a href="#shutting_down-0">shutting_down/0</a></td><td>Internal export.</td></tr>
-<tr><td valign="top"><a href="#start-0">start/0</a></td><td></td></tr>
-<tr><td valign="top"><a href="#start_link-0">start_link/0</a></td><td></td></tr>
-<tr><td valign="top"><a href="#stop-0">stop/0</a></td><td></td></tr>
+<tr><td valign="top"><a href="#send_req_direct-4">send_req_direct/4</a></td><td>Same as send_req/3 except that the first argument is the PID
+ returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-5">send_req_direct/5</a></td><td>Same as send_req/4 except that the first argument is the PID
+ returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-6">send_req_direct/6</a></td><td>Same as send_req/5 except that the first argument is the PID
+ returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-7">send_req_direct/7</a></td><td>Same as send_req/6 except that the first argument is the PID
+ returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#set_dest-3">set_dest/3</a></td><td>Deprecated.</td></tr>
+<tr><td valign="top"><a href="#set_max_pipeline_size-3">set_max_pipeline_size/3</a></td><td>Set the maximum pipeline size for each connection to a specific Host:Port.</td></tr>
+<tr><td valign="top"><a href="#set_max_sessions-3">set_max_sessions/3</a></td><td>Set the maximum number of connections allowed to a specific Host:Port.</td></tr>
+<tr><td valign="top"><a href="#show_dest_status-2">show_dest_status/2</a></td><td>Shows some internal information about load balancing to a
+ specified Host:Port.</td></tr>
+<tr><td valign="top"><a href="#spawn_link_worker_process-2">spawn_link_worker_process/2</a></td><td>Same as spawn_worker_process/2 except the the calling process
+ is linked to the worker process which is spawned.</td></tr>
+<tr><td valign="top"><a href="#spawn_worker_process-2">spawn_worker_process/2</a></td><td>Creates a HTTP client process to the specified Host:Port which
+ is not part of the load balancing pool.</td></tr>
+<tr><td valign="top"><a href="#start-0">start/0</a></td><td>Starts the ibrowse process without linking.</td></tr>
+<tr><td valign="top"><a href="#start_link-0">start_link/0</a></td><td>Starts the ibrowse process linked to the calling process.</td></tr>
+<tr><td valign="top"><a href="#stop-0">stop/0</a></td><td>Stop the ibrowse process.</td></tr>
+<tr><td valign="top"><a href="#stop_worker_process-1">stop_worker_process/1</a></td><td>Terminate a worker process spawned using
+ spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
<tr><td valign="top"><a href="#trace_off-0">trace_off/0</a></td><td>Turn tracing off for the ibrowse process.</td></tr>
<tr><td valign="top"><a href="#trace_off-2">trace_off/2</a></td><td>Turn tracing OFF for all connections to the specified HTTP
@@ -100,12 +116,15 @@ <h3 class="function"><a name="code_change-3">code_change/3</a></h3>
<p><tt>code_change() -&gt; term()</tt></p>
</div>
-<h3 class="function"><a name="finished_async_request-0">finished_async_request/0</a></h3>
+<h3 class="function"><a name="get_config_value-1">get_config_value/1</a></h3>
+<div class="spec">
+<p><tt>get_config_value() -&gt; term()</tt></p>
+</div><p>Internal export</p>
+
+<h3 class="function"><a name="get_config_value-2">get_config_value/2</a></h3>
<div class="spec">
-<p><tt>finished_async_request() -&gt; term()</tt></p>
-</div><p>Internal export. Called by a HTTP connection process to
- indicate to the load balancing process (ibrowse) that an
- asynchronous request has finished processing.</p>
+<p><tt>get_config_value() -&gt; term()</tt></p>
+</div><p>Internal export</p>
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
<div class="spec">
@@ -127,12 +146,18 @@ <h3 class="function"><a name="init-1">init/1</a></h3>
<p><tt>init() -&gt; term()</tt></p>
</div>
-<h3 class="function"><a name="reply-2">reply/2</a></h3>
+<h3 class="function"><a name="rescan_config-0">rescan_config/0</a></h3>
<div class="spec">
-<p><tt>reply() -&gt; term()</tt></p>
-</div><p>Internal export. Called by a HTTP connection process to
- indicate to the load balancing process (ibrowse) that a synchronous
- request has finished processing.</p>
+<p><tt>rescan_config() -&gt; term()</tt></p>
+</div><p>Clear current configuration for ibrowse and load from the file
+ ibrowse.conf in the IBROWSE_EBIN/../priv directory. Current
+ configuration is cleared only if the ibrowse.conf file is readable
+ using file:consult/1</p>
+
+<h3 class="function"><a name="rescan_config-1">rescan_config/1</a></h3>
+<div class="spec">
+<p><tt>rescan_config() -&gt; term()</tt></p>
+</div>
<h3 class="function"><a name="send_req-3">send_req/3</a></h3>
<div class="spec">
@@ -205,47 +230,97 @@ <h3 class="function"><a name="send_req-6">send_req/6</a></h3>
</div><p>Same as send_req/5.
All timeout values are in milliseconds.</p>
+<h3 class="function"><a name="send_req_direct-4">send_req_direct/4</a></h3>
+<div class="spec">
+<p><tt>send_req_direct() -&gt; term()</tt></p>
+</div><p>Same as send_req/3 except that the first argument is the PID
+ returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-5">send_req_direct/5</a></h3>
+<div class="spec">
+<p><tt>send_req_direct() -&gt; term()</tt></p>
+</div><p>Same as send_req/4 except that the first argument is the PID
+ returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-6">send_req_direct/6</a></h3>
+<div class="spec">
+<p><tt>send_req_direct() -&gt; term()</tt></p>
+</div><p>Same as send_req/5 except that the first argument is the PID
+ returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-7">send_req_direct/7</a></h3>
+<div class="spec">
+<p><tt>send_req_direct() -&gt; term()</tt></p>
+</div><p>Same as send_req/6 except that the first argument is the PID
+ returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
<h3 class="function"><a name="set_dest-3">set_dest/3</a></h3>
<div class="spec">
-<p><tt>set_dest(Host::string(), Port::integer(), Opts::<a href="#type-opt_list">opt_list()</a>) -&gt; ok</tt>
-<ul class="definitions"><li><tt><a name="type-opt_list">opt_list()</a> = [opt]</tt></li>
-<li><tt><a name="type-opt">opt()</a> = {max_sessions, integer()} | {max_pipeline_size, integer()} | {trace, <a href="#type-boolean">boolean()</a>}</tt></li>
-</ul></p>
-</div><p>Sets options for a destination. If the options have not been
- set in the ibrowse.conf file, it can be set using this function
- before sending the first request to the destination. If not,
- defaults will be used. Entries in ibrowse.conf look like this.
- <code><br>
- {dest, Host, Port, MaxSess, MaxPipe, Options}.<br>
- where <br>
- Host = string(). "www.erlang.org" | "193.180.168.23"<br>
- Port = integer()<br>
- MaxSess = integer()<br>
- MaxPipe = integer()<br>
- Options = optionList() -- see options in send_req/5<br>
- </code></p>
+<p><tt>set_dest() -&gt; term()</tt></p>
+</div><p>Deprecated. Use set_max_sessions/3 and set_max_pipeline_size/3
+ for achieving the same effect.</p>
+
+<h3 class="function"><a name="set_max_pipeline_size-3">set_max_pipeline_size/3</a></h3>
+<div class="spec">
+<p><tt>set_max_pipeline_size(Host::string(), Port::integer(), Max::integer()) -&gt; ok</tt></p>
+</div><p>Set the maximum pipeline size for each connection to a specific Host:Port.</p>
+
+<h3 class="function"><a name="set_max_sessions-3">set_max_sessions/3</a></h3>
+<div class="spec">
+<p><tt>set_max_sessions(Host::string(), Port::integer(), Max::integer()) -&gt; ok</tt></p>
+</div><p>Set the maximum number of connections allowed to a specific Host:Port.</p>
+
+<h3 class="function"><a name="show_dest_status-2">show_dest_status/2</a></h3>
+<div class="spec">
+<p><tt>show_dest_status() -&gt; term()</tt></p>
+</div><p>Shows some internal information about load balancing to a
+ specified Host:Port. Info about workers spawned using
+ spawn_worker_process/2 or spawn_link_worker_process/2 is not
+ included.</p>
-<h3 class="function"><a name="shutting_down-0">shutting_down/0</a></h3>
+<h3 class="function"><a name="spawn_link_worker_process-2">spawn_link_worker_process/2</a></h3>
<div class="spec">
-<p><tt>shutting_down() -&gt; term()</tt></p>
-</div><p>Internal export. Called by a HTTP connection process to
- indicate to ibrowse that it is shutting down and further requests
- should not be sent it's way.</p>
+<p><tt>spawn_link_worker_process() -&gt; term()</tt></p>
+</div><p>Same as spawn_worker_process/2 except the the calling process
+ is linked to the worker process which is spawned.</p>
+
+<h3 class="function"><a name="spawn_worker_process-2">spawn_worker_process/2</a></h3>
+<div class="spec">
+<p><tt>spawn_worker_process(Host::string(), Port::integer()) -&gt; {ok, pid()}</tt></p>
+</div><p>Creates a HTTP client process to the specified Host:Port which
+ is not part of the load balancing pool. This is useful in cases
+ where some requests to a webserver might take a long time whereas
+ some might take a very short time. To avoid getting these quick
+ requests stuck in the pipeline behind time consuming requests, use
+ this function to get a handle to a connection process. <br>
+ <b>Note:</b> Calling this function only creates a worker process. No connection
+ is setup. The connection attempt is made only when the first
+ request is sent via any of the send_req_direct/4,5,6,7 functions.<br>
+ <b>Note:</b> It is the responsibility of the calling process to control
+ pipeline size on such connections.
+ </p>
<h3 class="function"><a name="start-0">start/0</a></h3>
<div class="spec">
<p><tt>start() -&gt; term()</tt></p>
-</div>
+</div><p>Starts the ibrowse process without linking. Useful when testing using the shell</p>
<h3 class="function"><a name="start_link-0">start_link/0</a></h3>
<div class="spec">
-<p><tt>start_link() -&gt; term()</tt></p>
-</div>
+<p><tt>start_link() -&gt; {ok, pid()}</tt></p>
+</div><p>Starts the ibrowse process linked to the calling process. Usually invoked by the supervisor ibrowse_sup</p>
<h3 class="function"><a name="stop-0">stop/0</a></h3>
<div class="spec">
<p><tt>stop() -&gt; term()</tt></p>
-</div>
+</div><p>Stop the ibrowse process. Useful when testing using the shell.</p>
+
+<h3 class="function"><a name="stop_worker_process-1">stop_worker_process/1</a></h3>
+<div class="spec">
+<p><tt>stop_worker_process(Conn_pid::pid()) -&gt; ok</tt></p>
+</div><p>Terminate a worker process spawned using
+ spawn_worker_process/2 or spawn_link_worker_process/2. Requests in
+ progress will get the error response <pre>{error, closing_on_request}</pre></p>
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
<div class="spec">
@@ -279,6 +354,6 @@ <h3 class="function"><a name="trace_on-2">trace_on/2</a></h3>
<hr>
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
-<p><i>Generated by EDoc, Feb 7 2008, 11:49:30.</i></p>
+<p><i>Generated by EDoc, Mar 27 2008, 01:03:49.</i></p>
</body>
</html>
@@ -1,10 +1,11 @@
{application, ibrowse,
[{description, "HTTP client application"},
- {vsn, "1.2.5"},
+ {vsn, "1.4"},
{modules, [ ibrowse,
ibrowse_http_client,
ibrowse_app,
ibrowse_sup,
+ ibrowse_lb,
ibrowse_lib ]},
{registered, []},
{applications, [kernel,stdlib,sasl]},
@@ -3,4 +3,5 @@
'../src/ibrowse_app'.
'../src/ibrowse_sup'.
'../src/ibrowse_lib'.
+'../src/ibrowse_lb'.
'../src/ibrowse_test'.
@@ -3,6 +3,7 @@ ERL_FILES = ibrowse.erl \
ibrowse_app.erl \
ibrowse_sup.erl \
ibrowse_lib.erl \
+ ibrowse_lb.erl \
ibrowse_test.erl
@@ -5,7 +5,8 @@
ibrowse_http_client,
ibrowse_app,
ibrowse_sup,
- ibrowse_lib ]},
+ ibrowse_lib,
+ ibrowse_lb ]},
{registered, []},
{applications, [kernel,stdlib,sasl]},
{env, []},
Oops, something went wrong.

0 comments on commit c784ab8

Please sign in to comment.