|Failed to load latest commit information.|
version 0.56, stable, January 10, 2012, svn 589 2006 - 2012 Copyright (C) Robert Iakobashvili, <firstname.lastname@example.org>, Ashdod, Israel Michael Moser, <email@example.com>, Modiin, Israel All rights reserved. To skip bla-bla, please, refer to the QUICK-START file, but mind, that tuning for a high load still is in the FAQs. INTRODUCTION The loader is an open-source community tool written in C-language, simulating application load and application behavior of thousands and tens of thousand HTTP/HTTPS and FTP/FTPS clients, each with its own source IP-address. In contrast to other tools, curl-loader is using real client protocol stacks, namely, HTTP and FTP stacks of libcurl, TLS/SSL of openssl, simulates user behavior and supports login and authentication flavors. The goal of the project is to deliver a powerful and flexible performance loading and testing tool as a reasonable substitute for Spirent Avalanche and IXIA IxLoad. The tool is useful for performance loading of various application services, using HTTP, HTTPS, FTP and FTPS, for testing web and ftp servers and traffic generation. Activities of each virtual client are logged and collected statistics include information about: resolving, connection establishment, sending of requests, receiving responses, headers and data received/sent, errors from network, TLS/SSL and application (HTTP, FTP) level events and errors. Virtual clients are grouped together to so-called batches of clients, performing the same sort of activities, like: - authentication login; - user activity simulation by fetching several URLs with configurable timeouts in between; - logoff. The tool can be easily extended to generate sftp, telnet, tftp, ldap, ssh, scp etc other application protocols, supported by the great libcurl library. FEATURES - Virtual clients number. The tools runs depending on your HW and scenario 5000-25000 and more simultaneously loading clients, all from a single curl-loader process. Actual number of clients may be several times higher (we tried 60 000) and is limited mainly by memory. Each client performs loading from its own source IP-address or, as an option, all clients share the same IP-address; - Rampup of the virtual clients number at loading start in either automatic or manual mode; - IPv4 and IPv6 addresses and URIs; - HTTP 1.1. GET, POST, PUT (including file upload), DELETE, HEAD; - HTTP user authentication login with POST or GET+POST methods. Unique configurable username and password for each virtual client as well as configurable posted string (post-forms) are the options. Another option is loading of users with credentials from a text file; - HTTP POST/GET forms with up to 16 tokens loaded from file; - HTTP user logoff with POST, GET+POST, or GET (cookies); POST logoff with configurable posted string (post-forms); - HTTP multipart form data POST-ing as in RFC1867; - HTTP Web and Proxy Authentication (HTTP 401 and 407 responses) with Basic, Digest (RFC2617) and NTLM supported; - HTTP 3xx redirections with unlimited number of redirections; - HTTP cookies and DNS caches; - FTP passive and active, FTP upload; - Full customization of client request HTTP/FTP headers ; - Transfer limit rate for each client download or upload operation on a per url bases; - URL fetching probability; - TCP connections reuse or re-establishment on a per url bases; - Unlimited configurable number of URLs. Mixing of HTTP, HTTPS, FTP and FTPS urls in a single batch (test plan) configuration; - Thousands URLs configured from a template-URL and static tokens loaded from file; - URLs configured from a template-URL and dynamic tokens fetched from responses; - Connection establishment timers for each URL; - URL completion timers monitoring and enforcement for each client; - Inter/after URL "sleeping" timers, including random timers taken from an interval; - Logfile with tracing activities for each virtual client. The logfile is automatically rewinded, when reaching configurable size preventing disk crashes; - Responses logging (headers and bodies) to files. - Pre-cooked batch configuration (test plan) examples; - Load Status GUI and load status output to file; - Status and statistics per each virtual client logged to file; - Detailed loading statistics at Load Status GUI and to file; LICENSE curl-loader is licensed under GPL2. AUTHORS The tool is written and supported by Robert Iakobashvili and Michael Moser both from Israel, which are the copyright-holders for the project. We are thankful to the great people, specified in THANKS file for their contributions. SUPPORT Please, use the link to the mailing list provided in Support section to contact us and to get support for the tool. Subscribe to the curl-loader-devel list and mail your filled PROBLEM-REPORTING form located in the curl-loader tarball to the list. Your suggestion, ideas and patches would be very much appreciated. The official language of the curl-loader-devel mailing list is English, whereas Russian and German written mails will be also responded. IGNORE_CONTENT_LENGTH - Ignore the Content-Length header on a per url basis. This is useful for Apache 1.x (and similar servers) which will report incorrect content length for files over 2 gigabytes. If this option is used, curl will not be able to accurately report progress, and will simply stop the download when the server ends the connection. E.g. Add the following in the URL Section IGNORE_CONTENT_LENGTH=1 FREQUENTLY ASKED QUESTIONS Questions 1. Basic Terminology 1.1. What is the batch? 1.2. What is the batch configuration file? 2. Hardware Requirements 2.1. What are the minimal HW requirements? 2.2. What is the recommended HW configuration? 3. Building curl-loader 3.1. Which operating systems are supported by curl-loader? 3.2. What are the building pre-requirements? 3.3. How to make (build) curl-loader? 4. Creating Loading Configuration 4.1. How can I create loading configuration file? 4.2. What are the loading configuration file tags and semantics? 4.3. What are the exact meanings of all these tags? 4.4. How does the loader support login, logoff and authentication flavors? 5. What about FTP load and mixed FTP/HTTP load? 6. Running Load 6.1. What are the running environment requirements? 6.2. How I can run the load? 6.3. Which loading modes are supported? 6.4. How I can monitor loading progress status? 6.5. Why I am getting "Connection time-out after 5108 ms" like errors in the log file? 6.6. Where is the detaled log of all virtual clients activities and how to read it? 6.7. Which statistics is collected and how to get to it? 7. Advanced Issues 7.1. What about performance? 7.2. How to run a really big load? 7.3. How to calculate CAPS numbers for a load? QUESTIONS: 1. Basic Terminology 1.1. What is the batch? ^ Batch is a group of clients (virtual clients emulating real clients) with the same characteristics and loading behavior. 1.2. What is the batch configuration file? ^ Batch configuration file is a test plan for a batch of clients. 2. Hardware Requirements 2.1. What are the minimal HW requirements? ^ A PC with Pentium-III and 200 MB may be used for loads with hundreds simultaneous loading clients. 2.2. What is the recommended HW configuration? ^ Note, that each virtual client takes about 30-35 K memory in user-space plus some none-pageable memory in kernel mainly for send and recv buffers. Our PC with Pentium-4 2.4 GHz and 480 MB memory is capable to support 3000-4000 simultaneously loading clients. To reach 10K simultaneously loading clients we would recommend a PC with a single 3.2 GHz Intel or 2.2 GHz AMD and 1.5-2 GB of memory. When loading from a multi-CPU (or multi-core) systems, a loading model with the number of curl-loader threads equal to the number of logical CPUs seen in /proc/cpuinfo may be the most effective and deliver even higher numbers (than 10K) of simultaneously loading clients from a single PC. To run a batch in several threads use the command line option -t <number-of-threads>. If you are thinking, tuning, optimizing and getting our advise - the sky is the limit. 3. Building curl-loader 3.1. Which operating systems are supported by curl-loader? ^ You can use any operating system of your choice, providing that this is linux with kernel 2.4 or 2.6. 3.2. What are the building pre-requirements? ^ General C development environment with bash, gcc, make, etc on a linux machine is required. Building pre-requirements are: 1. openssl binaries; 2. openssl development package with include files (on debian libssl-dev); Adjust Makefile variables to point to the openssl headers and libraries. To specify an openssl development directory with include files (e.g. crypto.h), export environment variable OPENSSLDIR with the value of that directory. For example: $export OPENSSLDIR=the-full-path-to-the-directory Another known issue is libidn.so, which means, that some linux distributions do have some libidn.so.11, but not libidn.so. Resolve it by creating a softlink or in some cases exclude -lidn from the linking string, which requires commenting a string in Makefile and uncommenting another. Tarball of curl is included to the current release. When libcurl or curl-loader has building issues, correct them in the Makefile. 3.3. How to make (build) curl-loader? ^ Run the following commands from your bash linux shell: $tar zxfv curl-loader-<version>.tar.gz $cd curl-loader-<version> $make By default, we are building both libcurl and curl-loader without optimization and with debugging -g option. To build with optimization and without debugging, please, run: $make cleanall $make optimize=1 debug=0 Y can make optional installing of the loader by running as a root: #make install If still any building issues, please, fill you free to contact us for assistance using placed in download tarball PROBLEM-REPORTING form and its instructions. 4. Creating Loading Configuration 4.1. How can I create loading configuration file? ^ To run a load create your configuration file to be passed to curl-loader using -f command line option, e.g. #curl-loader -f ./conf-examples/bulk.conf For more examples, please, look at the files in "conf-examples" directory. You may copy an example file and edit it by using the next FAQ guidelines. 4.2. What are the loading configuration file tags and semantics? ^ Configuration file or "batch configuration file" consists from tag=value strings, grouped into the 2 sections: - General; - URLs; Section General contains common for the batch parameters, whereas section URLs has one or more URL subsections each starting with a certain URL and containing more tags. An example of a simple configuration file to be used for traffic generation client side against e.g. lighttpd web server: ---------------------------------------------------------------------------- ########### GENERAL SECTION ################## BATCH_NAME= 10K-clients CLIENTS_NUM_MAX=10000 CLIENTS_NUM_START=100 CLIENTS_RAMPUP_INC=50 INTERFACE =eth0 NETMASK=255.255.0.0 IP_ADDR_MIN= 192.168.1.1 IP_ADDR_MAX= 192.168.53.255 CYCLES_NUM= -1 URLS_NUM= 1 ########### URLs SECTION ####################### URL=http://localhost/index.html URL_SHORT_NAME="local-index" REQUEST_TYPE=GET TIMER_URL_COMPLETION = 0 TIMER_AFTER_URL_SLEEP = 0 The name of the batch is "10K-clients" with the maximum number of clients to be 10000; load will be started with initial 100 clients and each seconds will be added 50 more clients. Network interface eth0 will be used for the load with unique IP-address for each client taken from 192.168.1.1 up to 192.168.53.255 with netmask 255.255.0.0. Number of cycles is to be performed by each client in not limited (CYCLES_NUM= -1) and load will proceed till user presses Cntl-C to stop it. Each client fetches a single url (URLS_NUM= 1) http://localhost/index.html. The url was given a short name "local-index", and this name will appear at the Load Status GUI. The url will be fetched using HTTP GET request method and the download will not be limited by time (TIMER_URL_COMPLETION = 0). If we would like to limit this operation by time, we could place here some number in milli-seconds, e.g. TIMER_URL_COMPLETION=3000; the timer will be monitored, enforced and reported, if missed, at Load Status GUI and other statistics. By default all missed url-completion timers are considered as errors and added to statistics as T-err. Each client after accomplishing a url, including possible protocol redirections, waits/sleeps for 0 ms (TIMER_AFTER_URL_SLEEP = 0, which means, that actually client does not wait/sleep) and goes to the next URL, if any, to fetch. Yet a one more example of a configuration file used for login to a web-service, perform some activity and logoff, all in cycles, is presented below: ########### GENERAL SECTION ################### # BATCH_NAME=web-service CLIENTS_NUM_MAX=100 CLIENTS_RAMPUP_INC=2 INTERFACE=eth0 NETMASK=24 IP_ADDR_MIN=192.168.0.2 IP_ADDR_MAX=192.168.0.2 URLS_NUM=4 CYCLES_NUM= 200 ########### URLs SECTION ####################### ### Login URL - cycling # GET-part # URL=http://192.168.0.1:8888/vax/root/Admin URL_SHORT_NAME="Login-GET" REQUEST_TYPE=GET TIMER_URL_COMPLETION = 10000 TIMER_AFTER_URL_SLEEP =5000 # POST-part # URL="" URL_SHORT_NAME="Login-POST" URL_USE_CURRENT=1 USERNAME= admin PASSWORD= admin-passphrase REQUEST_TYPE=POST FORM_USAGE_TYPE= SINGLE_USER FORM_STRING= username=%s&password=%s TIMER_URL_COMPLETION = 4000 TIMER_AFTER_URL_SLEEP =1000 ### Cycling URL # URL=http://192.168.0.1:8888/vax/Admin/ServiceList.do URL_SHORT_NAME="Service List" REQUEST_TYPE=GET TIMER_URL_COMPLETION = 3000 TIMER_AFTER_URL_SLEEP =1000 ### Logoff URL - cycling, uses GET and cookies to logoff. # URL=http://192.168.0.1:8888/vax/Admin/Logout.do URL_SHORT_NAME="Logoff" REQUEST_TYPE=GET TIMER_URL_COMPLETION = 0 TIMER_AFTER_URL_SLEEP =1000 The name of the batch is "web-service", maximum number of clients 100, load will started with initial 2 clients and each seconds will be added 2 more clients. Eth0 will be used with the same IP-address for each client 192.168.0.2 with netmask 255.255.255.0 (24 in CIDR notation). If we would like to assign to each client a unique IP, it could be done by providing a range of at least 100 IP-address, e.g. IP_ADDR_MIN=192.168.0.10 and IP_ADDR_MAX=192.168.0.110. Number of cycles is to be performed by each client is limited to 200, and the load stops, when each client accomplishes 200 cycles. A client fetches up to 4 urls, specified in section URLs The first URL operation, which is doing a simulated user/client, is login. Login is performed by GET-ing the page (with 3xx HTTP redirections handled inside) with the first url http://192.168.0.1:8888/vax/root/Admin, called shortly "Login-GET". Maximum time considered appropriate for this operation is 10000 msecs. The 2000 msec "sleep" timer simulates here the time taken by a user to fill username and password to the POST-form received by the GET. The second url is empty and is used for POST-ing to the "current" url, using the form. Such empty string urls are allowed only, when URL_USE_CURRENT=1 is specified. The url is named as "Login-POST". Username "admin" and password "admin-passphrase" are used as the single user type and "filled" to the FORM_STRING= username=%s&password=%s. The resulting form, which is sent to the web-service, is "username=admin&password=admin-passphrase". There are other options available for the tag FORM_STRING, such as unique generated users, unique generated passwords or credentials loaded from a file; these options will be described later. The maximum time allowed for this POST operation is 4000 msecs, and the time of waiting/sleeping till the next url is 1000 msecs. Worth to mention, that when the authentication is accomplished successfully, server sets to client cookies and redirects the client to the service page, which is the next url (note, that all redirections are handled internally). The third url is http://192.168.0.1:8888/vax/Admin/ServiceList.do with a short name "Service List". After the url is fetched a client comes to the forth url http://192.168.0.1:8888/vax/Admin/Logout.do for logoff. This batch (testing plan) does not control and limit time for logoff as indicated by TIMER_URL_COMPLETION = 0. After 1000 msecs sleeping a client comes back to the first url to repeat the sequence till the number of cycles is expired. If we would like, that login and logoff to be done for each client only once, we would add to the first, second and forth url the tag URL_DONT_CYCLE=1 to mark the urls as not-cycled to be fetched only once. Note, that both quoted and non-quoted strings are supported as the tags values. For more examples, please, look at the files in "conf-examples" directory. 4.3. What are the exact meanings of all these tags? ^ Tags of the sections GENERAL: BATCH_NAME requires a string value and is used to name the batch. This name is used for the display while the program is running, and also for the three generated log and statistics files. CLIENTS_NUM_MAX - a maximum number of clients to be used for this load. Any positive number is valid here. Note, that each loading client requires about 30 K of RAM plus some non-pageable memory in kernel. We have already tried loads with up to 60000 clients. Loads above 1000 client is recommended to start with a some lower starting number, managed by the next tag CLIENTS_NUM_START and ramped-up each second by adding CLIENTS_RAMPUP_INC clients to the load. CLIENTS_NUM_START - number of clients to start the load with. If the tag is not present or zero CLIENTS_RAMPUP_INC value will be taken instead. If neither CLIENTS_NUM_START nor CLIENTS_RAMPUP_INC are present or have any positive value, the loader will take CLIENTS_NUM_MAX as the number of clients to start the load. CLIENTS_RAMPUP_INC - number of clients to be added to the load in the auto mode every second till CLIENTS_NUM_MAX number will be reached. For machines with a single CPU we would recommend not to keep the number above 50-100, whereas for dual-CPU machines you can keep it as large as 200-300. INTERFACE - name of the loading network interface. Find your interfaces by running /sbin/ifconfig. NETMASK - netmask for the loading clients IP-addresses (from IP_ADDR_MIN to IP_ADDR_MAX). For IPv4 you can use either quad-dotted netmask string or CIDR number from 0-32. For IPv6 only CIDR values from 0 to 128 are supported. To enable IPv6 change --disable-ipv6 to --enable-ipv6 in Makefile IP-addresses from IP_ADDR_MIN up to IP_ADDR_MAX are the interval to be used for loading clients. The interval of the addresses should be large enough to supply a dedicated IP for each client. Another mode that can be used is "single IP address" for all clients. When IP_ADDR_MIN and IP_ADDR_MAX have the same valid IP-address, it will be used for all clients. If the IP-addresses specified by the tags do not exist, curl-loader adds them as the secondary IPs to the loading network interface. Remember, that you should ensure smooth routing from the addresses used to server and from server back to the client addresses. Another route to ensure, when is appropriate, is to/from DNS server for resolving. Firewalling (netfilter/iptables) settings are also among those to check. CYCLES_NUM is the number of cycles to be performed if the value is positive. A zero or negative number means indefinite number of cycles, but see the RUN_TIME tag below. The default value is zero. Loading can be normally stopped at any moment by pressing Cntl-C; all statistics will be collected properly and files closed to contain necessary information. Note, that cycling relates only to the URLs not containing tag URL_DONT_CYCLE = 1. Curl-loader enables two optional non-cycling areas: the first before the cycling url/s area and the second after the cycling url/s area. DUMP_OPSTATS is the flag indicating whether to write out a fairly voluminous operational statistics file. It takes values "yes" or "no". The default value is "yes". RUN_TIME is the maximum time to run the load. It is specified as up to four numbers: hours, days, minutes and seconds separated by a colon (":"). When less than four numbers are specified, the senior categories are assumed to be omitted. That is 1:1:0:0 means one day and one hour, 1:1 means one minute and 1 second, 10000 means 10000 seconds, etc. A zero or negative value means no limit to run time. As noted above, the curl-loader may stop before the maximum time, if any, is reached if either the number of clycles to be performed runs out or the Cntl-C is pressed on the keyboard. REQ_RATE is the desired fixed request rate per second, sometimes referred to as offered load. It is specified as an integer number. The rate here means that of the effective URL requests, so if, for example, each initial URL is redirected once with a 302 response, the rate of actual requests will be double from the specified one. Note that as many clients as necessary are invoked in order to maintain the offered load. If the total response time to the effective URL exceeds 1 second, the CLIENTS_NUM_MAX value must exceed the REQ_RATE value proportionately. When the number of clients is insufficient for the load, an error message "need free clients (X)" is written to stderr, where X is the number of additional clients required. That number may be used as a guide for increasing the CLIENTS_NUM_MAX value. USER_AGENT provides an option to over-write the default MSIE-6-like HTTP header User-Agent. Place here a quoted string to emulate the browser that you need. The header is entered globally. If you need an option to customize it on a per-URL bases, please, use the tag HEADER="User-Agent: the string of my favorite browser". URLS_NUM is the number of urls to be used in URLs Section. Tags of the sections URLs: URL - can be the first tag of each url-subsection and marks beginning of a new url. Yet another option to start a new url (set of urls) is the tag URL_TEMPLATE. Note, that the tags URL and URL_TEMPLATE are mutually exclusive. URL tag should be either a valid url or an empty string, which is allowed only when URL_USE_CURRENT=1 also presents. "Current url" is used for example, when POST-ing at the url, resulting from the previous GET. HTTP redirections are handled internally and there is no need to use "current url" approach for that purpose. Note, that URL used for file uploading should include a filename and not only a directory. URL_SHORT_NAME - an optional up to 12 ASCII string long name, which will appear on a Load Status GUI. It improves appearance of the current operational status at the console. We are recommending to configure it. URL_USE_CURRENT - used in combination with an empty string URL="" to indicate, that the current url, fetched by the previous operation, should be used. "Current url" is useful e.g., when POST-ing at the url, resulting from the previous GET. Note, that HTTP redirections are handled internally and there is no need to use "current url" approach for that purpose. URL_DONT_CYCLE - marks a url as non-cycling. Curl-loader enables two optional non-cycling areas: the first before the cycling url/s area and the second after it. The first non-cycling area may be used for login, authentication, etc. single time operations, whereas the second non-cycling area is good for a single time logoff, for example. REQUEST_TYPE - HTTP request method to be chosen from GET, POST, PUT, DELETE or HEAD. UPLOAD_FILE - filename, including path, to the file to be used for uploading. The path should be taken from the curl-loader place, e.g. ./conf-examples/bax.conf HEADER - is assisting to customize/add/over-write HTTP/FTP headers. If a header already exits by default, the custom header over-writes it. USER_AGENT tag is for User-Agent header only, whereas by HEADER may be added or over-written any header (including User-Agent). Look for example at ./conf-examples/custom_hdrs.conf. USERNAME and PASSWORD are the tags to provide credentials for the login operations. The strings may be used either as is or as the base-words to append numbers and to generate some unique credentials. The behavior is governed by the tag FORM_USAGE_TYPE. Note, that PASSWORD may be an empty string. FORM_USAGE_TYPE governs user login process. "UNIQUE_USERS_AND_PASSWORDS" is used to generate unique usernames and passwords by appending client sequence numbers (starting from 1) to USERNAME and PASSWORD tags within FORM_STRING template. "UNIQUE_USERS_SAME_PASSWORD" to be used when generating unique users with the same password. "SINGLE_USER" is useful, when all clients are making login using the same USERNAME and PASSWORD. "RECORDS_FROM_FILE" means, that user credentials to be loaded from the file specified by FORM_RECORDS_FILE value. "AS_IS" means, that the FORM_STRING template to be used just AS IS and without any usernames or passwords either from the relevant tags or from file. FORM_STRING is the configurable template form to be used for POST-ing credentials or any other records. To generate multiple unique users with unique passwords FORM_USAGE_TYPE= "UNIQUE_USERS_AND_PASSWORDS", and the FORM_STRING to be a template like "username=%s%d&password=%s%d". First '%s' will be substituted by the value of USERNAME tag and '%d' by the client number. Second '%s' will be substituted by PASSWORD tag value and second '%d' by the same client number. For example, if USERNAME=robert, PASSWORD=stam and FORM_STRING "username=%s%d&password=%s%d", the final POST string, used for the client number 1, will be "username=robert1&password=stam1". In this case USERNAME and PASSWORD strings are used just as base-words for generating unique user credentials by appending a client sequence number. When FORM_USAGE_TYPE="UNIQUE_USERS_SAME_PASSWORD" template of FORM string to be something like "username=%s%d&password=%s", where only username will be unique using the same approach as described above. When FORM_USAGE_TYPE="SINGLE_USER", provide FORM_STRING without %d symbols, e.g. "user=%s&secret=%s". Thus, all clients will have the same POST credentials with the string looking like "user=robert&secret=stam". When FORM_USAGE_TYPE= "RECORDS_FROM_FILE" FORM_STRING to be also without %d symbols, because the credentials uniqueness is supposed to be ensured by the file content. FORM_USAGE_TYPE="AS_IS" means, that FORM_STRING will not be validated and POST-ed indeed AS IS. FORM_RECORDS_FILE specifies the path to the file with credentials or any other tokens. (full-path or relative to curl-loader location). A text file with usernames and passwords with the structure of each string like: <teken1><separator><token2> can be used as an input. According to RFC1738, only reserved ':', '@', '/' and probably ' ' (space) are safe to use as separators between username and password. An example of batch configuration is ./conf-examples/post-form-token-fr-file.conf and an example of credentials is in ./conf-examples/credentials.cred. Next version will support up to 16 tokens (2 tokens now). FORM_RECORDS_FILE_MAX_NUM allows to load not from a records file, specified by tag FORM_RECORDS_FILE not the default number of records being the same as the maximum number of virtual clients CLIENTS_NUM_MAX, but some larger number, required by FORM_RECORDS_RANDOM. FORM_RECORDS_RANDOM allows to use for each virtual client not a pre-defined record, but a randomly chosen record. One can load a 10000 records from a records file using FORM_RECORDS_FILE tag and use the records in a random fashion for 1000 clients. The tag does not ensure uniquickness of the records used for each virtual client. To use the tag properly, specify number of the records to be loaded, using tag FORM_RECORDS_FILE_MAX_NUM WEB_AUTH_METHOD to be configured from the "BASIC", "DIGEST", "GSS_NEGOTIATE", "NTLM" or "ANY". The configured method will be used for authentication on HTTP 401 response. When "ANY" is configured, libcurl will choose a method. To use "GSS_NEGOTIATE" the libcurl should be re-compiled with support for GSS. WEB_AUTH_CREDENTIALS to be provided in the form "user:password". If not configured, the loader uses USERNAME and PASSWORD tags value to synthesize the credentials. PROXY_AUTH_METHOD to be configured from the "BASIC", "DIGEST", "GSS_NEGOTIATE", "NTLM" or "ANY". The configured method will be used for authentication on HTTP 407 response. When "ANY" is configured, libcurl will choose a method. To use "GSS_NEGOTIATE" the libcurl should be re-compiled with support for GSS. PROXY_AUTH_CREDENTIALS to be provided in the form "user:password". If not configured, the loader uses USERNAME and PASSWORD tags value to synthesize the credentials. FRESH_CONNECT is used to define on a per url bases, whether the connection should be re-used or closed and re-connected after request-response cycle. When 1, the TCP connection will be re-established. The system default is to keep the connection and re-use it as much as server and protocol allow it. Still the system default could be changed by the command-line option -r. TIMER_TCP_CONN_SETUP is the time in seconds for DNS resolving and TCP connection setup on a per url bases. The global default is 5 seconds, which can be changed using -c command-line option. TIMER_URL_COMPLETION is the time in milli-seconds to allow a url fetching operation, including all internal re-directions. The legal values are 0, which means no limit on url-completion time, or above 20 msecs. When a value of the tag is above 0, we are monitoring the url-fetching and canceling it, if above the specified milliseconds. The results are presented on the Load Status GUI as the operation "Timeouted" statistics and logged to all statistics files as T-Err number. TIMER_AFTER_URL_SLEEP is the time in milli-seconds for client to sleep after fetching a URL prior to dealing with the next URL. Zero (0) means don't wait and schedule client after the URL immediately. Random timer values could be an option specified as e.g. 0-2000, which means, that a client will sleep for some random time from 0 to 2000 milliseconds. FTP_ACTIVE, when defined as 1, is forcing FTP protocol to use an active mode (the default is passive). LOG_RESP_HEADERS, when defined as 1, logs response headers to a file. Directory <batch-name> is created with subdirs url0, url1, url<n> Headers of responses are logged to the files named: cl-<client-num>-cycle-<cycle-num>.hdr LOG_RESP_BODIES, when defined as 1, logs response headers to a file. Directory <batch-name> is created with subdirs url0, url1... url<n> Headers of responses are logged to the files named: cl-<client-num>-cycle-<cycle-num>.body RESPONSE_STATUS_ERRORS supports changes to the default set of per-url responses considered as errors. By default 4xx without 401 and 407 and all 5xx response codes are treated as errors. Now you can either add a status to the errors set or remove it. Sign + (plus) adds, - (minus) removes. The syntax is a line of tokens separated by ',', where each tokens begins with either + or - and is followed by the response status number from 0 up to 600. For example, the effect of RESPONSE_STATUS_ERRORS="+200,-404" is that 200 responses will be considered for that url as errors, whereas 404 will be considered as success. MULTIPART_FORM_DATA tag adds initial support for multipart form data POST-ing as in RFC1867. Several tags MULTIPART_FORM_DATA can be used for a url to post, e.g. as in ./conf-examples/multipart-formdata-post.conf: MULTIPART_FORM_DATA="yourname=Michael" MULTIPART_FORM_DATA="filedescription=Cool text file with cool text inside" MULTIPART_FORM_DATA="htmlcode=<HTML></HTML>;type=text/html" MULTIPART_FORM_DATA="firstname.lastname@example.org" MULTIPART_FORM_DATA="email@example.com,fil2.txt,fil3.html" The files to be copied are indicated by @ and to be located in the same directory as curl-loader. Currently, the feature uses the strings provided AS_IS and does not generate any unique users, unique passwords or loads tokens from file. To prevent from sending "Expect: 100-continue", pass as a custom HTTP header HEADER="Expect: " TRANSFER_LIMIT_RATE limits client maximum throughput for a url. The value of the tag to be provided as bytes (not bits) per second. FETCH_PROBABILITY allows to fetch a url not as a must, but with a certain run-time probability. The allowed values are in the range from 1 to 100 percents. FETCH_PROBABILITY_ONCE enables for a client to make the decision regarding wether to fetch a URL or not using FETCH_PROBABILITY to be done only once (at the first cycle). 4.4. How does the loader support login, logoff and authentication flavors? ^ curl-loader performs login and logoff operations using the following HTTP methods: - GET+POST (server response to GET provides a post-form to be filled and posted by POST); - POST only; - GET only. The loader also supports multipart form data POST-ing as in RFC1867. URL_TEMPLATE Yet another option to start a new url (set of urls) is the tag URL. Note, that the tags URL and URL_TEMPLATE are mutually exclusive. An example of URL_TEMPLATE usage is: URL_TEMPLATE= http://www.foo.com/user/%s/group/%s. URL_TEMPLATE allows an url to vary from client to client or from cycle to cycle. As in FORM_STRINGs, the %s markers will be replaced with appropriate strings before the URL is fetched. The replacement values can be obtained either statically from file (using URL_TOKEN_FILE) or fetched dynamically by fetching token from HTTP-responses (using RESPONSE_TOKENs and URL_TOKENs). RESPONSE_TOKEN (Example: RESPONSE_TOKEN = user_id) There can be any number of RESPONSE_TOKENs in an URL or URL_TEMPLATE subsection. When curl-loader fetches the url, it will scan the server's response for all the response tokens. If found, curl-loader will save the "value" of each response token for use in constructing later URL_TEMPLATEs (see URL_TOKEN below). For instance, if we specify RESPONSE_TOKEN = user_id, and if the server response contains ... user_id=1234 ..., then we will save the name-value pair "user_id, 1234". Different clients may well receive different responses and save different values, thus constructing different urls from later templates. Note, that order of the RESPONSE_TOKENs in the url subsection is immaterial. Matches and values are collected across response-packet boundaries. Once a value is collected from a particular server response, the scanning for that token stops, and subsequent occurrences of that token in the response will not trigger a new value collection. Finally, a RESPONSE_TOKEN with the same name as one in a previous url will replace any previously collected value for that name. See, the usage example in ./conf-examples/url-template-resp-dynamic.conf URL_TOKEN Such tags may occur in an URL_TEMPLATE subsection. Each URL_TOKEN is taken in order and looked up in this client's bucket of saved RESPONSE_TOKENs. The value obtained is substituted for the next %s marker in the URL_TEMPLATE. Thus the order of the URL_TOKENs matters. There must be the same number of URL_TOKENs as %s makers, and each URL_TOKEN must have a corresponding saved RESPONSE_TOKEN value, or an error results. Note that not all these values need come from the same prior url the only requirement is that the needed value has been obtained from some prior server response by this client. URL_TOKEN is mutually exclusive with URL_TOKEN_FILE tag, where tokens are loaded from file statically. See, the usage example in ./conf-examples/url-template-resp-dynamic.conf URL_TOKEN_FILE - This tag may occur in an URL_TEMPLATE subsection, and is mutually exclusive with URL_TOKEN tag. It specifies a file from which the substitution tokens for the URL_TEMPLATE will be taken. The pathname must be absolute or relative to the curl-loader directory. The syntax for token files is a bit more permissive than the rules for scanning server responses. (1) Each remaining line is parsed to obtain the tokens for constructing one url, so there must be as many tokens on each line as there are %s markers in the URL_TEMPLATE. (2) Tokens are runs of non-whitespace characters, or quoted strings, separated by whitespace. (3) Attention! If there is an extra token remaining in the line, it is saved as a cookie to be sent when the url is fetched. As the load runs, the pre-constructed urls are taken on demand, and if the demand is greater than the supply, we start over from the first url. Thus, if the number of urls is equal to the number of clients, each client will get the same unique url for each cycle. See, the usage example in ./conf-examples/url-template-fr-file.conf URL_RANDOM_RANGE - Specifies a range for random numbers to replace the token specified in URL_RANDOM_TOKEN E.g. URL_RANDOM_RANGE=0-2000 URL_RANDOM_TOKEN - Replace this token in the URL with a random number from the range specified with URL_RANDOM_RANGE. E.g. URL_RANDOM_TOKEN=JUNKSTR For example config look at ./conf-examples/url-randomize.conf RANDOM_SEED - this tags allows setting the random seed to a specified non-negative value. With the same seed the pseudo-random numbers used in various parts of the curl-loader are generated in the same order, which produces more consistent results. If this tag is absent or the value is negative, the random seed is generated based on the current time, i.e. it is different from one run of the curl-loader to another. The same seed is used for all random numbers. (Example, RANDOM_SEED = 10). The loader supports HTTP Web Authentication and Proxy Authentication. The supported authentication methods are Basic, Digest (RFC2617) and NTLM. When responded 401 or 407, libcurl will choose the most safe method from those, supported by the server, unless the batch configuration (test plan) is explicitly indicating the method to be used by the tags WEB_AUTH_METHOD and/or PROXY_AUTH_METHOD. To support GSS Web-Authentication, add in Makefile building of libcurl against appropriate GSS library, see libcurl pages for detailed instructions. 5. What about FTP load and mixed FTP/HTTP load? To generate FTP/FTPS load, please, pass user credentials via ftp-url according to the RFC 1738 like: ftp://username:password@hostname:port/etc-str Tag FTP_ACTIVE serves to switch from the default passive FTP to active. Please, look at the examples in conf-examples directory, e.g. ftp.conf, ftp_http.conf, ftp_upload.conf. 6. Running Load 6.1. What are the running environment requirements? ^ Running hundred and thousand of clients, please, do not forget: - to increase limit of descriptors (sockets) by running e.g. #ulimit -n 100 000; - optionally, to set reuse of sockets in time-wait state: by setting #echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle and/or #echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse; In some cases you may need to increase the system limits for open descriptors (sockets): echo 150 000 > /proc/sys/fs/file-max Note, that your linux distribution may require editing of some other files like /etc/security/limits.conf, etc 6.2. How I can run the load? ^ Usage: run as a root user: #./curl-loader -f <configuration filename> [other options] Other possible options are: -c[onnection establishment timeout, seconds] -e[rror drop client. Client on error doesn't attempt to process the next cycle] -d[etailed logging, hich outputs to logfile headers and bodies of requests/responses] -h[elp] -i[ntermediate (snapshot) statistics time interval (default 3 sec)] -f[ilename of configuration to run (batches of clients)] -l[ogfile max size in MB (default 1024). On the size reached, file pointer is rewinded] -m[ode of loading, 0 - hyper (the default, epoll () based ), 1 - smooth (select () based)] -r[euse connections disabled. Closes TCP-connections and re-open them. Try with and without] -v[erbose output to the logfiles; includes info about headers sent/received. Increase the level of verbosity by using this option twice] -u[rl logging - logs url names to logfile, when -v verbose option is used] -w[arnings skip] -x[set|unset proxy] "<proxy:port>" Connection Reuse Disable Option (-r): The default behavior of curl-loader after HTTP response is to re-use the tcp-connection for the next request. If you are specifying -r command-line option, the TCP connection will be closed and re-opened for the next request. Whether it is appropriate for you to work with -r option or without, it depends on your server support of Keep-Alive and the purpose of your testing. Try with and without -r and see, what you get. Connection reuse (which is the default, without -r option) has advantages due to the decreased consumption of opened descriptors (sockets) and ports. 6.3. Which loading modes are supported? ^ Hyper-mode (command line -m0) is the default mode . The mode is using for event demultiplexing epoll() or /dev/epoll. Another loading mode is called "smoothing" (-m1 command line) is basically the same as hyper, but uses poll() system call for demultiplexing. 6.4. How I can monitor loading progress status? ^ curl-loader outputs to the console loading status and statistics as the Load Status GUI output, where the left column is for the latest interval and the right is for the summary numbers since load start. A copy of the output is also saved in the file <batch-name>.txt 6.5. Why I am getting "Connection time-out after 5108 ms" like errors in the log file? ^ This means, that the connections cannot be established within 5 seconds (which is the default). If some of clients are getting such errors, this means, that the server is overloaded and cannot establish TCP connection within the time. Y may wish to increase the connection timeout globally by e.g. -c 10 or to specify an increased timeout using tag TIMER_TCP_CONN_SETUP=10. When all your clients have such errors, in most cases this is due either to resolving or routing problems or other network problems. Y can set any addresses to your clients in IP_ADDR_MIN and IP_ADDR_MAX tags, providing, that you ensure a smooth route from client and from server for the addresses you use. Please, start with a single client and see that it works, than progress further. When you are specifying any addresses here, they are added by curl-loader initialization to the network interface of your choice and may be seen by the command: #/sbin/ip addr Command ping (man ping) has an option -I to force the ping to be issued from a certain ip-address. To test, that your routing is OK, you may use, e.g.: #ping -I <the client address you wish to use> url-target When it works - it will be OK. Please, also pay attention to the firewalling/NAT settings at your linux loading machine (netfilter/iptables), at the tested servers and through the whole packets route. Y may also wish to read a bit about linux routing/networking/iptables and DNS resolving HOWTOS. 6.6. Where is the detailed log of all virtual clients activities and how to read it? ^ Detailed log is written to the file named: <batch-name>.log: The semantics of logfile output, using command line options -v -v (very verbose) and -u (url print) are as follows. The first line is the time of the load start in milliseconds and in text form: # 1248743647809 Mon Jul 27 18:14:07 2009 The second line identifies log columns: # msec_offset cycle_no url_no client_no (ip) indic info That is the columns are: - time offset from start in milliseconds - cycle number - url number - client number - client ip address in parentheses - indicator, qualifies the message as setup (==), send (>=), receive (<=) or response (!!) - information string Subsequent lines look as follows. 1234 4 3 39 (192.168.0.39) == Trying 10.30.6.42... : eff-url: http://10.30.6.42 :8888/server/Admin/ServiceList.do, url: Which means: 1234 msec from start, cycle 4, url number 3, client number 39 with ipv4 address (192.168.0.39), type of the message is setup, eff-url - is the url, used right now, "url:" is empty, which means, that it is the same as effective. Effective url may be a result of redirection and, thus, "url:" (target url, specified in batch configuration file) will be printed as well. Please, note, that when the logfile reaches 1024 MB size, curl-loader rewinds it and starts to overwrite it from the beginning. Y may tune the rewinding file size by using command line option: -l <log-filesize-in-MB> 6.7. Which statistics is collected and how to get to it? ^ Currently HTTP/HTTPS statistics includes the following counters: - run-time in seconds; - requests num; - 1xx success num; - 2xx success num; - 3xx redirects num; - client 4xx errors num; - server 5xx errors num; - other errors num, like resolving, tcp-connect, server closing or empty responses number (Err); - url completion time expiration errors (T-Err); - average application server Delay (msec), estimated as the time between HTTP request and HTTP response without taking into the account network latency (RTT) (D); - average application server Delay for 2xx (success) HTTP-responses, as above, but only for 2xx responses. The motivation for that is that 3xx redirections and 5xx server errors/rejects may not necessarily provide a true indication of a testing server working functionality (D-2xx); - throughput in, batch average, Bytes/sec (T-In); - throughput out, batch average, Bytes/sec (T-Out); The statistics goes to the screen (both the interval and the current summary statistics for the load) as well as to the file with name <batch_name>.txt When the load completes or when the user presses CTRL-C (sometimes some clients may stall), the final load report is printed at the console as well as to the statistics file. Some strings from the file: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Run-Time,Appl,Clients,Req,1xx,2xx,3xx,4xx,5xx,Err,T-Err,D,D-2xx,T-In,T-Out 2, Appl , 100, 155, 0, 0, 96, 0, 0, 0, 0, 1154, 1154, 2108414, 15538 2, Sec-Appl, 100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 4, Appl, 100, 75, 0, 32, 69, 0, 0, 0, 0, 1267, 1559, 1634656, 8181 4, Sec-Appl, 100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 Cutted here 36, Appl , 39, 98, 0, 35, 58, 0, 0, 0, 0, 869, 851, 1339168, 11392 36, Sec-Appl, 39, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 38, Appl , 3, 91, 0, 44, 62, 0, 0, 0, 0, 530, 587, 1353899, 10136 38, Sec-Appl, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 *, *, *, *, *, *, *, *, *, *, *, * Run-Time,Appl,Clients,Req,1xx,2xx,3xx,4xx,5xx,Err,T-Err,D,D-2xx,T-In,T-Out 38, Appl , 0, 2050, 0, 643, 1407, 0, 213, 0, 0, 725, 812, 1610688, 11706 38, Sec-Appl, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The bottom strings after asterisks are for final averages. At the same time a clients dump file with name <batch_name>.ctx is generated to provide detailed statistics about each client state and statistics counters. One string from the file: 1 (192.168.0.1) ,cycles:124,cstate:1,b-in:22722029,b-out:174605,req:745,2xx:497,3xx:248,4xx:0,5xx:0,err:0 where 1 (192.168.0.1)- is the index of the client and its ip-address; cycles- number of loading cycles done; cstate - is the number of the client state (-1 - error, 0 - init, 1- urls, 2-final-ok); b-in - bytes passed in; b-out - bytes passed out; req- number of requests done; 2xx, 3xx, 4xx, 5xx - number of responses Nxx received; err - number of libcurl errors at the resolving, TCP/IP and TLS/SSL levels; The following conditions are considered as errors: - error at the level of libcurl, which includes resolving, TCP/IP and, when applicable, TLS/SSL errors; - all HTTP 5xx server errors; - most of HTTP 4xx client errors, excluding 401 and 407 authentication responses not considered real errors; When the above error conditions occur, a virtual client is marked as being in the error state. By default we "recover" such client by scheduling it to the next loading cycle, starting from the first operation of the cycle. You may use command line option -e to change the default behavior to another, so that clients once arriving at error state will not be scheduled for load any more. 7. Advanced Issues 7.1. What about performance? ^ HYPER mode (-m 0 command line option) was already used for loads with up to 10000 clients, whereas HW issues, like CPU and memory resource, start to be counting. The mode uses epoll() syscall (via libevent library) for demultiplexing. 7.2. How to run a really big load? ^ 0. Every big load starts with a small load. First, see, that you have a working configuration file. Run it with a 1-2-3 clients and commandline option -v, and look into the l<batch-name>.log logfile. Look into the HW issues, discussed in the FAQs and mind, that each client requires 30-35 K of memory. 1. Compile with optimization; Since you need performance compile with optimization and without debugging. $make cleanall $make optimize=1 debug=0 Y may add to Makefile optimization for your particular processor by -match /-mcpu gcc option directives to OPT_FLAGS. 2. Login as a su; 3. Increase the default number of allowed open descriptors (sockets); Run e.g. #ulimit -n 100 000 When running several instances of curl-loader, consider increase of system limits for open descriptors, if necessary. Take your own account of the socket usage in the system, considering sockets faster recycling (less time in the time-wait state), by setting, optionally, something like this: #echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle and/or #echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse; Correct, if required, the value of CURL_LOADER_FD_SETSIZE (set to 20000 in Makefile) to control the maximum fd, that may be used for select. This is not required to be cared about for the default hyper mode. Increase the maximum number of open descriptors in your linux system, if required, using linux HOWTOS. echo 65535 > /proc/sys/fs/file-max Note, that your linux distribution may require editing of some other files like /etc/security/limits.conf, etc 4. Relax routing checks; Relax routing checks for your loading network interface. When "eth0" used for loading run: echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter 5. Increase memory available for kernel tcp; Read the maximum available TCP memory and sysctl or echo as a root the number to the kernel tcp, e.g.: cat /proc/sys/net/core/wmem_max - the output is 109568. /sbin/sysctl net.ipv4.tcp_mem="109568 109568 109568" or echo "109568 109568 109568" > /proc/sys/net/ipv4/tcp_mem 6. Increase ARP thresholds; echo 1024000 > /proc/sys/net/ipv4/neigh/default/gc_thresh1 echo 1024000 > /proc/sys/net/ipv4/neigh/default/gc_thresh2 echo 1024000 > /proc/sys/net/ipv4/neigh/default/gc_thresh3 Read: http://txt.bitprocessor.be/2011/02/10/curl-loader-and-arp-cache-issues/ 7. Create configuration files for each instance of curl-loader to run. What is important is to give a unique value for tag BATCH_NAME, which is in use by a separate instance of curl-loader. Logfile, report file, etc have name, which are the derivatives of the BATCH_NAME value. Therefore, when several instances of curl-loader are writing to the same file, this is not helpful and may be even "crashful". Please, use in your configuration batch files non-overlapping ranges of IP-addresses, else libcurl virtual clients will compete for the IP-addresses to bind to. Use CLIENTS_RAMPUP_INC tag in smooth or hyper mode to increase number of your clients gradually at start-up in order not boom the server and not to burn out the CPU at your loading machine. Addition of new clients is a CPU-expensive operation, therefore keep CLIENTS_RAMPUP_INC below 100 clients per second. 8. Connections re-use. The default behavior of curl-loader now is to re-use the tcp-connection for the next request. This default decreases consumption of CPU and open sockets. If you are specifying -r command-line option, the connection will be closed and re-opened for the next request. 9. Running curl-loader in multiple threads on several CPUs/cores. SMP-support can be gained by using option -t <threads-num>. We can recommend to use number of threads as number of CPUs from /proc/cpuinfo. Sometimes curl-loader crashes on start with the option used due to suspected bugs in libevent library. Re-run it optionally changing/decreasing number of threads used. Note, that total statistics is written to the file $batch-name_0.txt, whereas logs are per-thread and are written to the files $batch-name_<thread-num>.log. 10. Troubleshooting. Run the first loading attempt with a small number of clients using command-line options -v (verbose) and -u (url in logs). Grep to look for the errors and their reasons. If an error is "Connection timeout", you may try to increase the connection establishment timeout (the default is 5 seconds), using -c command-line option. When all your clients fail to connect with "Connection timeout" error, this may be due to routing or resolving problems. If any assistance required, please, don't hesitate to contact us using PROBLEM-REPORTING form in the download tarball. 11. Logs and statistics. After end of a run, or after SIGINT (Cntl-C), the final results are calculated and printed to the console as well as to the file <batch-name>.txt. Current results are presented in each row, and average summary as the last raws, separated from the rest by asterisks. Pay attention, that <batch-name>.log log file may become huge, particularly, when using verbose output (-v -u). Command-line option -l <maxsize in MB> may be useful, whereas the default policy is to rewind the logfile (writing from the file start), when it reaches 1 GB. Do not use -v and -u options, when you have performance issues. 12. Monitoring of the loading PC. Please, use top or vmstat commands to monitor the memory usage, swapping and CPU. Intensive swapping is a good indication, that your PC is short in user-space memory. If you see "memory pressure" counters in netstat -s output, this is a good indication, that the PC is short in kernel memory for TCP. Zero idle CPU is not the show stopper, but when you see, that Load Status GUI on your console prints its output with delays higher than it should be (the default is 3 seconds and may be adjusted by -i command-line option), you need stronger CPU or run several curl-loader processes on a multi-CPU machine. Note, that for a load with several curl-loader processes you need to arrange different configuration files with different batch-names and not overlapping ranges of IP-addresses for each curl-loader process. 7.3. How to calculate CAPS numbers for a load? ^ When number of clients is defined by CLIENTS_NUM_MAX tag, number of CAPS (call attempts per seconds) is resulting from the clients number and load duration for each cycle, comprising from url time for each url with possible redirections, intervals betwee urls and after url interval. The actions and time intervals are configurable in batch file, whereas url retrival time is server and network dependent and not always easy to predict. The result is that number of clients/requests is a known parameter, and number of CAPS is something to be estimated from the time of test and number of requests. LOAD STATUS GUI presents calculated current and average CAPS. LIMITATIONS: Please, see our TODO file for some of them.