Lots of new features

README

@@ -1,4 +1,4 @@
-$Id: README,v 1.14 2008/02/27 23:39:22 chandrusf Exp $
+$Id: README,v 1.15 2008/03/27 01:35:49 chandrusf Exp $
 
 ibrowse is a HTTP client. The following are a list of features.
 	- RFC2616 compliant (AFAIK) 
@@ -7,7 +7,7 @@ ibrowse is a HTTP client. The following are a list of features.
 	- Understands HTTP/0.9, HTTP/1.0 and HTTP/1.1
 	- Understands chunked encoding
         - Can generate requests using Chunked Transfer-Encoding
-	- Named pools of connections to each webserver
+	- Pools of connections to each webserver
 	- Pipelining support
 	- Download to file
 	- Asynchronous requests. Responses are streamed to a process
@@ -22,6 +22,26 @@ Comments to : Chandrashekhar.Mullaparthi@t-mobile.co.uk
 
 CONTRIBUTIONS & CHANGE HISTORY
 ==============================
+27-03-2008 - * Major rewrite of the load balancing feature. Additional module,
+               ibrowse_lb.erl, introduced to achieve this.
+             * Can now get a handle to a connection process which is not part of
+               the load balancing pool. Useful when an application is making
+               requests to a webserver which are time consuming (such as 
+               uploading a large file). Such requests can be put on a separate
+               connection, and all other smaller/quicker requests can use the
+               load balancing pool. See ibrowse:spawn_worker_process/2 and
+               ibrowse:spawn_link_worker_process/2
+             * Ram Krishnan sent a patch to enable a client to send a lot of
+               data in a request by providing a fun which is invoked by the
+               connection handling process. This fun can fetch the data from 
+               any where. This is useful when trying to upload a large file
+               to a webserver.
+             * Use the TCP_NODELAY option on every socket by default
+             * Rudimentary support for load testing of ibrowse. Undocumented, 
+               but see ibrowse_test:load_test/3. Use the source, Luke!
+             * New function ibrowse:show_dest_status/2 to view state of
+               connections/pipelines to a web server
+
 20-02-2008 - Ram Krishnan sent another patch for another hidden bug in the
              save_response_to_file feature.
 

doc/ibrowse.html

@@ -10,15 +10,15 @@
 
 <h1>Module ibrowse</h1>
 <ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>The ibrowse application implements an HTTP 1.1 client.
-<p>Copyright © 2005-2007 Chandrashekhar Mullaparthi</p>
+<p>Copyright © 2005-2008 Chandrashekhar Mullaparthi</p>
 
-<p><b>Version:</b> 1.2.7</p>
+<p><b>Version:</b> 1.4</p>
 <p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
 <p><b>Authors:</b> Chandrashekhar Mullaparthi (<a href="mailto:chandrashekhar dot mullaparthi at gmail dot com"><tt>chandrashekhar dot mullaparthi at gmail dot com</tt></a>).</p>
 
 <h2><a name="description">Description</a></h2><p>The ibrowse application implements an HTTP 1.1 client. This  
 module implements the API of the HTTP client. There is one named  
-process called 'ibrowse' which acts as a load balancer. There is  
+process called 'ibrowse' which assists in load balancing and maintaining configuration. There is one load balancing process per unique webserver. There is  
 one process to handle one TCP connection to a webserver  
 (implemented in the module ibrowse_http_client). Multiple connections to a  
 webserver are setup based on the settings for each webserver. The  
@@ -47,10 +47,6 @@ <h2><a name="description">Description</a></h2><p>The ibrowse application impleme
   		  {save_response_to_file, true}], 1000).
   <br><br>
 
-  ibrowse:set_dest("www.hotmail.com", 80, [{max_sessions, 10},
-  					    {max_pipeline_size, 1}]).
-  <br><br>
-
   ibrowse:send_req("http://www.erlang.org", [], head).
 
   <br><br>
@@ -69,21 +65,41 @@ <h2><a name="description">Description</a></h2><p>The ibrowse application impleme
   driver isn't actually used.</p>
 <h2><a name="index">Function Index</a></h2>
 <table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
-<tr><td valign="top"><a href="#finished_async_request-0">finished_async_request/0</a></td><td>Internal export.</td></tr>
+<tr><td valign="top"><a href="#get_config_value-1">get_config_value/1</a></td><td>Internal export.</td></tr>
+<tr><td valign="top"><a href="#get_config_value-2">get_config_value/2</a></td><td>Internal export.</td></tr>
 <tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
 <tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
 <tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
 <tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
-<tr><td valign="top"><a href="#reply-2">reply/2</a></td><td>Internal export.</td></tr>
+<tr><td valign="top"><a href="#rescan_config-0">rescan_config/0</a></td><td>Clear current configuration for ibrowse and load from the file
+  ibrowse.conf in the IBROWSE_EBIN/../priv directory.</td></tr>
+<tr><td valign="top"><a href="#rescan_config-1">rescan_config/1</a></td><td></td></tr>
 <tr><td valign="top"><a href="#send_req-3">send_req/3</a></td><td>This is the basic function to send a HTTP request.</td></tr>
 <tr><td valign="top"><a href="#send_req-4">send_req/4</a></td><td>Same as send_req/3.</td></tr>
 <tr><td valign="top"><a href="#send_req-5">send_req/5</a></td><td>Same as send_req/4.</td></tr>
 <tr><td valign="top"><a href="#send_req-6">send_req/6</a></td><td>Same as send_req/5.</td></tr>
-<tr><td valign="top"><a href="#set_dest-3">set_dest/3</a></td><td>Sets options for a destination.</td></tr>
-<tr><td valign="top"><a href="#shutting_down-0">shutting_down/0</a></td><td>Internal export.</td></tr>
-<tr><td valign="top"><a href="#start-0">start/0</a></td><td></td></tr>
-<tr><td valign="top"><a href="#start_link-0">start_link/0</a></td><td></td></tr>
-<tr><td valign="top"><a href="#stop-0">stop/0</a></td><td></td></tr>
+<tr><td valign="top"><a href="#send_req_direct-4">send_req_direct/4</a></td><td>Same as send_req/3 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-5">send_req_direct/5</a></td><td>Same as send_req/4 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-6">send_req_direct/6</a></td><td>Same as send_req/5 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#send_req_direct-7">send_req_direct/7</a></td><td>Same as send_req/6 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
+<tr><td valign="top"><a href="#set_dest-3">set_dest/3</a></td><td>Deprecated.</td></tr>
+<tr><td valign="top"><a href="#set_max_pipeline_size-3">set_max_pipeline_size/3</a></td><td>Set the maximum pipeline size for each connection to a specific Host:Port.</td></tr>
+<tr><td valign="top"><a href="#set_max_sessions-3">set_max_sessions/3</a></td><td>Set the maximum number of connections allowed to a specific Host:Port.</td></tr>
+<tr><td valign="top"><a href="#show_dest_status-2">show_dest_status/2</a></td><td>Shows some internal information about load balancing to a
+  specified Host:Port.</td></tr>
+<tr><td valign="top"><a href="#spawn_link_worker_process-2">spawn_link_worker_process/2</a></td><td>Same as spawn_worker_process/2 except the the calling process
+  is linked to the worker process which is spawned.</td></tr>
+<tr><td valign="top"><a href="#spawn_worker_process-2">spawn_worker_process/2</a></td><td>Creates a HTTP client process to the specified Host:Port which
+  is not part of the load balancing pool.</td></tr>
+<tr><td valign="top"><a href="#start-0">start/0</a></td><td>Starts the ibrowse process without linking.</td></tr>
+<tr><td valign="top"><a href="#start_link-0">start_link/0</a></td><td>Starts the ibrowse process linked to the calling process.</td></tr>
+<tr><td valign="top"><a href="#stop-0">stop/0</a></td><td>Stop the ibrowse process.</td></tr>
+<tr><td valign="top"><a href="#stop_worker_process-1">stop_worker_process/1</a></td><td>Terminate a worker process spawned using
+  spawn_worker_process/2 or spawn_link_worker_process/2.</td></tr>
 <tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
 <tr><td valign="top"><a href="#trace_off-0">trace_off/0</a></td><td>Turn tracing off for the ibrowse process.</td></tr>
 <tr><td valign="top"><a href="#trace_off-2">trace_off/2</a></td><td>Turn tracing OFF for all connections to the specified HTTP
@@ -100,12 +116,15 @@ <h3 class="function"><a name="code_change-3">code_change/3</a></h3>
 <p><tt>code_change() -&gt; term()</tt></p>
 </div>
 
-<h3 class="function"><a name="finished_async_request-0">finished_async_request/0</a></h3>
+<h3 class="function"><a name="get_config_value-1">get_config_value/1</a></h3>
+<div class="spec">
+<p><tt>get_config_value() -&gt; term()</tt></p>
+</div><p>Internal export</p>
+
+<h3 class="function"><a name="get_config_value-2">get_config_value/2</a></h3>
 <div class="spec">
-<p><tt>finished_async_request() -&gt; term()</tt></p>
-</div><p>Internal export. Called by a HTTP connection process to
-  indicate to the load balancing process (ibrowse) that an
-  asynchronous request has finished processing.</p>
+<p><tt>get_config_value() -&gt; term()</tt></p>
+</div><p>Internal export</p>
 
 <h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
 <div class="spec">
@@ -127,12 +146,18 @@ <h3 class="function"><a name="init-1">init/1</a></h3>
 <p><tt>init() -&gt; term()</tt></p>
 </div>
 
-<h3 class="function"><a name="reply-2">reply/2</a></h3>
+<h3 class="function"><a name="rescan_config-0">rescan_config/0</a></h3>
 <div class="spec">
-<p><tt>reply() -&gt; term()</tt></p>
-</div><p>Internal export. Called by a HTTP connection process to
-  indicate to the load balancing process (ibrowse) that a synchronous
-  request has finished processing.</p>
+<p><tt>rescan_config() -&gt; term()</tt></p>
+</div><p>Clear current configuration for ibrowse and load from the file
+  ibrowse.conf in the IBROWSE_EBIN/../priv directory. Current
+  configuration is cleared only if the ibrowse.conf file is readable
+  using file:consult/1</p>
+
+<h3 class="function"><a name="rescan_config-1">rescan_config/1</a></h3>
+<div class="spec">
+<p><tt>rescan_config() -&gt; term()</tt></p>
+</div>
 
 <h3 class="function"><a name="send_req-3">send_req/3</a></h3>
 <div class="spec">
@@ -205,47 +230,97 @@ <h3 class="function"><a name="send_req-6">send_req/6</a></h3>
 </div><p>Same as send_req/5.
   All timeout values are in milliseconds.</p>
 
+<h3 class="function"><a name="send_req_direct-4">send_req_direct/4</a></h3>
+<div class="spec">
+<p><tt>send_req_direct() -&gt; term()</tt></p>
+</div><p>Same as send_req/3 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-5">send_req_direct/5</a></h3>
+<div class="spec">
+<p><tt>send_req_direct() -&gt; term()</tt></p>
+</div><p>Same as send_req/4 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-6">send_req_direct/6</a></h3>
+<div class="spec">
+<p><tt>send_req_direct() -&gt; term()</tt></p>
+</div><p>Same as send_req/5 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
+<h3 class="function"><a name="send_req_direct-7">send_req_direct/7</a></h3>
+<div class="spec">
+<p><tt>send_req_direct() -&gt; term()</tt></p>
+</div><p>Same as send_req/6 except that the first argument is the PID
+  returned by spawn_worker_process/2 or spawn_link_worker_process/2</p>
+
 <h3 class="function"><a name="set_dest-3">set_dest/3</a></h3>
 <div class="spec">
-<p><tt>set_dest(Host::string(), Port::integer(), Opts::<a href="#type-opt_list">opt_list()</a>) -&gt; ok</tt>
-<ul class="definitions"><li><tt><a name="type-opt_list">opt_list()</a> = [opt]</tt></li>
-<li><tt><a name="type-opt">opt()</a> = {max_sessions, integer()} | {max_pipeline_size, integer()} | {trace, <a href="#type-boolean">boolean()</a>}</tt></li>
-</ul></p>
-</div><p>Sets options for a destination. If the options have not been
-  set in the ibrowse.conf file, it can be set using this function
-  before sending the first request to the destination. If not,
-  defaults will be used. Entries in ibrowse.conf look like this.
-  <code><br>
-  {dest, Host, Port, MaxSess, MaxPipe, Options}.<br>
-  where <br>
-  Host = string(). "www.erlang.org" | "193.180.168.23"<br>
-  Port = integer()<br>
-  MaxSess = integer()<br>
-  MaxPipe = integer()<br>
-  Options = optionList() -- see options in send_req/5<br>
-  </code></p>
+<p><tt>set_dest() -&gt; term()</tt></p>
+</div><p>Deprecated. Use set_max_sessions/3 and set_max_pipeline_size/3
+  for achieving the same effect.</p>
+
+<h3 class="function"><a name="set_max_pipeline_size-3">set_max_pipeline_size/3</a></h3>
+<div class="spec">
+<p><tt>set_max_pipeline_size(Host::string(), Port::integer(), Max::integer()) -&gt; ok</tt></p>
+</div><p>Set the maximum pipeline size for each connection to a specific Host:Port.</p>
+
+<h3 class="function"><a name="set_max_sessions-3">set_max_sessions/3</a></h3>
+<div class="spec">
+<p><tt>set_max_sessions(Host::string(), Port::integer(), Max::integer()) -&gt; ok</tt></p>
+</div><p>Set the maximum number of connections allowed to a specific Host:Port.</p>
+
+<h3 class="function"><a name="show_dest_status-2">show_dest_status/2</a></h3>
+<div class="spec">
+<p><tt>show_dest_status() -&gt; term()</tt></p>
+</div><p>Shows some internal information about load balancing to a
+  specified Host:Port. Info about workers spawned using
+  spawn_worker_process/2 or spawn_link_worker_process/2 is not
+  included.</p>
 
-<h3 class="function"><a name="shutting_down-0">shutting_down/0</a></h3>
+<h3 class="function"><a name="spawn_link_worker_process-2">spawn_link_worker_process/2</a></h3>
 <div class="spec">
-<p><tt>shutting_down() -&gt; term()</tt></p>
-</div><p>Internal export. Called by a HTTP connection process to
-  indicate to ibrowse that it is shutting down and further requests
-  should not be sent it's way.</p>
+<p><tt>spawn_link_worker_process() -&gt; term()</tt></p>
+</div><p>Same as spawn_worker_process/2 except the the calling process
+  is linked to the worker process which is spawned.</p>
+
+<h3 class="function"><a name="spawn_worker_process-2">spawn_worker_process/2</a></h3>
+<div class="spec">
+<p><tt>spawn_worker_process(Host::string(), Port::integer()) -&gt; {ok, pid()}</tt></p>
+</div><p>Creates a HTTP client process to the specified Host:Port which
+  is not part of the load balancing pool. This is useful in cases
+  where some requests to a webserver might take a long time whereas
+  some might take a very short time. To avoid getting these quick
+  requests stuck in the pipeline behind time consuming requests, use
+  this function to get a handle to a connection process. <br>
+  <b>Note:</b> Calling this function only creates a worker process. No connection
+  is setup. The connection attempt is made only when the first
+  request is sent via any of the send_req_direct/4,5,6,7 functions.<br>
+  <b>Note:</b> It is the responsibility of the calling process to control
+  pipeline size on such connections.
+ </p>
 
 <h3 class="function"><a name="start-0">start/0</a></h3>
 <div class="spec">
 <p><tt>start() -&gt; term()</tt></p>
-</div>
+</div><p>Starts the ibrowse process without linking. Useful when testing using the shell</p>
 
 <h3 class="function"><a name="start_link-0">start_link/0</a></h3>
 <div class="spec">
-<p><tt>start_link() -&gt; term()</tt></p>
-</div>
+<p><tt>start_link() -&gt; {ok, pid()}</tt></p>
+</div><p>Starts the ibrowse process linked to the calling process. Usually invoked by the supervisor ibrowse_sup</p>
 
 <h3 class="function"><a name="stop-0">stop/0</a></h3>
 <div class="spec">
 <p><tt>stop() -&gt; term()</tt></p>
-</div>
+</div><p>Stop the ibrowse process. Useful when testing using the shell.</p>
+
+<h3 class="function"><a name="stop_worker_process-1">stop_worker_process/1</a></h3>
+<div class="spec">
+<p><tt>stop_worker_process(Conn_pid::pid()) -&gt; ok</tt></p>
+</div><p>Terminate a worker process spawned using
+  spawn_worker_process/2 or spawn_link_worker_process/2. Requests in
+  progress will get the error response <pre>{error, closing_on_request}</pre></p>
 
 <h3 class="function"><a name="terminate-2">terminate/2</a></h3>
 <div class="spec">
@@ -279,6 +354,6 @@ <h3 class="function"><a name="trace_on-2">trace_on/2</a></h3>
 <hr>
 
 <div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
-<p><i>Generated by EDoc, Feb 7 2008, 11:49:30.</i></p>
+<p><i>Generated by EDoc, Mar 27 2008, 01:03:49.</i></p>
 </body>
 </html>

ebin/ibrowse.app

@@ -1,10 +1,11 @@
 {application, ibrowse,
         [{description, "HTTP client application"},
-         {vsn, "1.2.5"},
+         {vsn, "1.4"},
          {modules, [ ibrowse, 
 		     ibrowse_http_client, 
 		     ibrowse_app, 
 		     ibrowse_sup, 
+		     ibrowse_lb, 
 		     ibrowse_lib ]},
          {registered, []},
          {applications, [kernel,stdlib,sasl]},

src/Emakefile.src

@@ -3,4 +3,5 @@
 '../src/ibrowse_app'.
 '../src/ibrowse_sup'.
 '../src/ibrowse_lib'.
+'../src/ibrowse_lb'.
 '../src/ibrowse_test'.

src/Makefile

@@ -3,6 +3,7 @@ ERL_FILES = ibrowse.erl \
 		ibrowse_app.erl \
 		ibrowse_sup.erl \
 		ibrowse_lib.erl \
+		ibrowse_lb.erl \
 		ibrowse_test.erl
 
 

src/ibrowse.app.src

@@ -5,7 +5,8 @@
 		     ibrowse_http_client, 
 		     ibrowse_app, 
 		     ibrowse_sup, 
-		     ibrowse_lib ]},
+		     ibrowse_lib,
+		     ibrowse_lb ]},
          {registered, []},
          {applications, [kernel,stdlib,sasl]},
 	 {env, []},