Multiple pipelines and limiting the number of connections.

Introducing a number of options to the multi interface that
allows for multiple pipelines to the same host, in order to
optimize the balance between the penalty for opening new
connections and the potential pipelining latency.

Two new options for limiting the number of connections:

CURLMOPT_MAX_HOST_CONNECTIONS - Limits the number of running connections
to the same host. When adding a handle that exceeds this limit,
that handle will be put in a pending state until another handle is
finished, so we can reuse the connection.

CURLMOPT_MAX_TOTAL_CONNECTIONS - Limits the number of connections in total.
When adding a handle that exceeds this limit,
that handle will be put in a pending state until another handle is
finished. The free connection will then be reused, if possible, or
closed if the pending handle can't reuse it.

Several new options for pipelining:

CURLMOPT_MAX_PIPELINE_LENGTH - Limits the pipeling length. If a
pipeline is "full" when a connection is to be reused, a new connection
will be opened if the CURLMOPT_MAX_xxx_CONNECTIONS limits allow it.
If not, the handle will be put in a pending state until a connection is
ready (either free or a pipe got shorter).

CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE - A pipelined connection will not
be reused if it is currently processing a transfer with a content
length that is larger than this.

CURLMOPT_CHUNK_LENGTH_PENALTY_SIZE - A pipelined connection will not
be reused if it is currently processing a chunk larger than this.

CURLMOPT_PIPELINING_SITE_BL - A blacklist of hosts that don't allow
pipelining.

CURLMOPT_PIPELINING_SERVER_BL - A blacklist of server types that don't allow
pipelining.

See the curl_multi_setopt() man page for details.

Author: linusnielsen

docs/libcurl/curl_multi_setopt.3

@@ -95,6 +95,112 @@ This option is for the multi handle's use only, when using the easy interface
 you should instead use the \fICURLOPT_MAXCONNECTS\fP option.
 
 (Added in 7.16.3)
+.IP CURLMOPT_MAX_HOST_CONNECTIONS
+Pass a long. The set number will be used as the maximum amount of
+simultaneously open connections to a single host. For each new session to
+a host, libcurl will open a new connection up to the limit set by
+CURLMOPT_MAX_HOST_CONNECTIONS. When the limit is reached, the sessions will
+be pending until there are available connections. If CURLMOPT_PIPELINING is
+1, libcurl will try to pipeline if the host is capable of it.
+
+The default value is 0, which means that there is no limit.
+However, for backwards compatibility, setting it to 0 when CURLMOPT_PIPELINING
+is 1 will not be treated as unlimited. Instead it will open only 1 connection
+and try to pipeline on it.
+
+(Added in 7.30.0)
+.IP CURLMOPT_MAX_PIPELINE_LENGTH
+Pass a long. The set number will be used as the maximum amount of requests
+in a pipelined connection. When this limit is reached, libcurl will use another
+connection to the same host (see CURLMOPT_MAX_HOST_CONNECTIONS), or queue the
+requests until one of the pipelines to the host is ready to accept a request.
+Thus, the total number of requests in-flight is CURLMOPT_MAX_HOST_CONNECTIONS *
+CURLMOPT_MAX_PIPELINE_LENGTH.
+The default value is 5.
+
+(Added in 7.30.0)
+.IP CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE
+Pass a long. If a pipelined connection is currently processing a request
+with a Content-Length larger than CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE, that
+connection will not be considered for additional requests, even if it is
+shorter than CURLMOPT_MAX_PIPELINE_LENGTH.
+The default value is 0, which means that the penalization is inactive.
+
+(Added in 7.30.0)
+.IP CURLMOPT_CHUNK_LENGTH_PENALTY_SIZE
+Pass a long. If a pipelined connection is currently processing a
+chunked (Transfer-encoding: chunked) request with a current chunk length
+larger than CURLMOPT_CHUNK_LENGTH_PENALTY_SIZE, that connection will not be
+considered for additional requests, even if it is shorter than
+CURLMOPT_MAX_PIPELINE_LENGTH.
+The default value is 0, which means that the penalization is inactive.
+
+(Added in 7.30.0)
+.IP CURLMOPT_PIPELINING_SITE_BL
+Pass an array of char *, ending with NULL. This is a list of sites that are
+blacklisted from pipelining, i.e sites that are known to not support HTTP
+pipelining. The array is copied by libcurl.
+
+The default value is NULL, which means that there is no blacklist.
+
+Pass a NULL pointer to clear the blacklist.
+
+Example:
+
+.nf
+  site_blacklist[] =
+  {
+    "www.haxx.se",
+    "www.example.com:1234",
+    NULL
+  };
+
+  curl_multi_setopt(m, CURLMOPT_PIPELINE_SITE_BL, site_blacklist);
+.fi
+
+(Added in 7.30.0)
+.IP CURLMOPT_PIPELINING_SERVER_BL
+Pass an array of char *, ending with NULL. This is a list of server types
+prefixes (in the Server: HTTP header) that are blacklisted from pipelining,
+i.e server types that are known to not support HTTP pipelining. The array is
+copied by libcurl.
+
+Note that the comparison matches if the Server: header begins with the string
+in the blacklist, i.e "Server: Ninja 1.2.3" and "Server: Ninja 1.4.0" can 
+both be blacklisted by having "Ninja" in the backlist.
+
+The default value is NULL, which means that there is no blacklist.
+
+Pass a NULL pointer to clear the blacklist.
+
+Example:
+
+.nf
+  server_blacklist[] =
+  {
+    "Microsoft-IIS/6.0",
+    "nginx/0.8.54",
+    NULL
+  };
+
+  curl_multi_setopt(m, CURLMOPT_PIPELINE_SERVER_BL, server_blacklist);
+.fi
+
+(Added in 7.30.0)
+.IP CURLMOPT_MAX_TOTAL_CONNECTIONS
+Pass a long. The set number will be used as the maximum amount of
+simultaneously open connections in total. For each new session, libcurl
+will open a new connection up to the limit set by
+CURLMOPT_MAX_TOTAL_CONNECTIONS. When the limit is reached, the sessions will
+be pending until there are available connections. If CURLMOPT_PIPELINING is
+1, libcurl will try to pipeline if the host is capable of it.
+
+The default value is 0, which means that there is no limit.
+However, for backwards compatibility, setting it to 0 when CURLMOPT_PIPELINING
+is 1 will not be treated as unlimited. Instead it will open only 1 connection
+and try to pipeline on it.
+
+(Added in 7.30.0)
 .SH RETURNS
 The standard CURLMcode for multi interface error codes. Note that it returns a
 CURLM_UNKNOWN_OPTION if you try setting an option that this version of libcurl

docs/libcurl/libcurl-errors.3

@@ -240,6 +240,9 @@ Mismatch of RTSP Session Identifiers.
 Unable to parse FTP file list (during FTP wildcard downloading).
 .IP "CURLE_CHUNK_FAILED (88)"
 Chunk callback reported error.
+.IP "CURLE_NO_CONNECTION_AVAILABLE (89)"
+(For internal use only, will never be returned by libcurl) No connection
+available, the session will be queued. (added in 7.30.0)
 .IP "CURLE_OBSOLETE*"
 These error codes will never be returned. They were used in an old libcurl
 version and are currently unused.

docs/libcurl/symbols-in-versions

@@ -85,6 +85,7 @@ CURLE_LDAP_SEARCH_FAILED        7.1
 CURLE_LIBRARY_NOT_FOUND         7.1           7.17.0
 CURLE_LOGIN_DENIED              7.13.1
 CURLE_MALFORMAT_USER            7.1           7.17.0
+CURLE_NO_CONNECTION_AVAILABLE   7.30.0
 CURLE_NOT_BUILT_IN              7.21.5
 CURLE_OK                        7.1
 CURLE_OPERATION_TIMEDOUT        7.10.2
@@ -267,8 +268,15 @@ CURLKHTYPE_DSS                  7.19.6
 CURLKHTYPE_RSA                  7.19.6
 CURLKHTYPE_RSA1                 7.19.6
 CURLKHTYPE_UNKNOWN              7.19.6
+CURLMOPT_CHUNK_LENGTH_PENALTY_SIZE 7.30.0
+CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE 7.30.0
+CURLMOPT_MAX_HOST_CONNECTIONS   7.30.0
+CURLMOPT_MAX_PIPELINE_LENGTH    7.30.0
+CURLMOPT_MAX_TOTAL_CONNECTIONS  7.30.0
 CURLMOPT_MAXCONNECTS            7.16.3
 CURLMOPT_PIPELINING             7.16.0
+CURLMOPT_PIPELINING_SERVER_BL   7.30.0
+CURLMOPT_PIPELINING_SITE_BL     7.30.0
 CURLMOPT_SOCKETDATA             7.15.4
 CURLMOPT_SOCKETFUNCTION         7.15.4
 CURLMOPT_TIMERDATA              7.16.0

include/curl/curl.h

@@ -507,6 +507,8 @@ typedef enum {
   CURLE_RTSP_SESSION_ERROR,      /* 86 - mismatch of RTSP Session Ids */
   CURLE_FTP_BAD_FILE_LIST,       /* 87 - unable to parse FTP file list */
   CURLE_CHUNK_FAILED,            /* 88 - chunk callback reported error */
+  CURLE_NO_CONNECTION_AVAILABLE, /* 89 - No connection available, the
+                                    session will be queued */
   CURL_LAST /* never use! */
 } CURLcode;
 

include/curl/multi.h

@@ -338,6 +338,31 @@ typedef enum {
   /* maximum number of entries in the connection cache */
   CINIT(MAXCONNECTS, LONG, 6),
 
+  /* maximum number of (pipelining) connections to one host */
+  CINIT(MAX_HOST_CONNECTIONS, LONG, 7),
+
+  /* maximum number of requests in a pipeline */
+  CINIT(MAX_PIPELINE_LENGTH, LONG, 8),
+
+  /* a connection with a content-length longer than this
+     will not be considered for pipelining */
+  CINIT(CONTENT_LENGTH_PENALTY_SIZE, OFF_T, 9),
+
+  /* a connection with a chunk length longer than this
+     will not be considered for pipelining */
+  CINIT(CHUNK_LENGTH_PENALTY_SIZE, OFF_T, 10),
+
+  /* a list of site names(+port) that are blacklisted from
+     pipelining */
+  CINIT(PIPELINING_SITE_BL, OBJECTPOINT, 11),
+
+  /* a list of server types that are blacklisted from
+     pipelining */
+  CINIT(PIPELINING_SERVER_BL, OBJECTPOINT, 12),
+
+  /* maximum number of open connections in total */
+  CINIT(MAX_TOTAL_CONNECTIONS, LONG, 13),
+
   CURLMOPT_LASTENTRY /* the last unused */
 } CURLMoption;
 

lib/Makefile.inc

@@ -25,7 +25,7 @@ CSOURCES = file.c timeval.c base64.c hostip.c progress.c formdata.c	\
   http_proxy.c non-ascii.c asyn-ares.c asyn-thread.c curl_gssapi.c	\
   curl_ntlm.c curl_ntlm_wb.c curl_ntlm_core.c curl_ntlm_msgs.c		\
   curl_sasl.c curl_schannel.c curl_multibyte.c curl_darwinssl.c		\
-  hostcheck.c bundles.c conncache.c
+  hostcheck.c bundles.c conncache.c pipeline.c
 
 HHEADERS = arpa_telnet.h netrc.h file.h timeval.h qssl.h hostip.h	\
   progress.h formdata.h cookie.h http.h sendf.h ftp.h url.h dict.h	\
@@ -44,4 +44,4 @@ HHEADERS = arpa_telnet.h netrc.h file.h timeval.h qssl.h hostip.h	\
   asyn.h curl_ntlm.h curl_gssapi.h curl_ntlm_wb.h curl_ntlm_core.h	\
   curl_ntlm_msgs.h curl_sasl.h curl_schannel.h curl_multibyte.h		\
   curl_darwinssl.h hostcheck.h bundles.h conncache.h curl_setup_once.h	\
-  multihandle.h setup-vms.h
+  multihandle.h setup-vms.h pipeline.h

lib/README.pipelining

@@ -42,10 +42,3 @@ Details
   still resolve the second one properly to make sure that they actually _can_
   be considered for pipelining. Also, asking for explicit pipelining on handle
   X may be tricky when handle X get a closed connection.
-
-- We need options to control max pipeline length, and probably how to behave
-  if we reach that limit. As was discussed on the list, it can probably be
-  made very complicated, so perhaps we can think of a way to pass all
-  variables involved to a callback and let the application decide how to act
-  in specific situations. Either way, these fancy options are only interesting
-  to work on when everything is working and we have working apps to test with.

lib/hash.h

@@ -104,4 +104,3 @@ void Curl_hash_print(struct curl_hash *h,
 
 
 #endif /* HEADER_CURL_HASH_H */
-

lib/http.c

@@ -73,6 +73,8 @@
 #include "http_proxy.h"
 #include "warnless.h"
 #include "non-ascii.h"
+#include "bundles.h"
+#include "pipeline.h"
 
 #define _MPRINTF_REPLACE /* use our functions only */
 #include <curl/mprintf.h>
@@ -3148,13 +3150,19 @@ CURLcode Curl_http_readwrite_headers(struct SessionHandle *data,
         }
         else if(conn->httpversion >= 11 &&
                 !conn->bits.close) {
+          struct connectbundle *cb_ptr;
 
           /* If HTTP version is >= 1.1 and connection is persistent
              server supports pipelining. */
           DEBUGF(infof(data,
                        "HTTP 1.1 or later with persistent connection, "
                        "pipelining supported\n"));
-          conn->server_supports_pipelining = TRUE;
+          /* Activate pipelining if needed */
+          cb_ptr = conn->bundle;
+          if(cb_ptr) {
+            if(!Curl_pipeline_site_blacklisted(data, conn))
+              cb_ptr->server_supports_pipelining = TRUE;
+          }
         }
 
         switch(k->httpcode) {
@@ -3231,6 +3239,16 @@ CURLcode Curl_http_readwrite_headers(struct SessionHandle *data,
         data->info.contenttype = contenttype;
       }
     }
+    else if(checkprefix("Server:", k->p)) {
+      char *server_name = copy_header_value(k->p);
+
+      /* Turn off pipelining if the server version is blacklisted */
+      if(conn->bundle && conn->bundle->server_supports_pipelining) {
+        if(Curl_pipeline_server_blacklisted(data, server_name))
+          conn->bundle->server_supports_pipelining = FALSE;
+      }
+      Curl_safefree(server_name);
+    }
     else if((conn->httpversion == 10) &&
             conn->bits.httpproxy &&
             Curl_compareheader(k->p,