Skip to content

Pathological performance when setting MAX_TOTAL_CONNECTIONS #2369

Closed
@cbay

Description

@cbay

libcurl version: 7.58.0
Operating system: Archlinux (Linux 4.15.5).

I'm trying to use libcurl to do as many requests as possible to a local server (similarly to load testing tools, although that's not my purpose). I've used the multi-uv.c example and slightly simplified it to not save files to disk and accept the number of requests to do as argument (full source).

The local server is nginx serving a tiny static page. It's not the bottleneck.

Doing 1000 requests takes about ~160ms:

$ time ./multi-uv 1000
./multi-uv 1000  0.08s user 0.06s system 89% cpu 0.162 total

For comparison, h2load takes a similar amount of time, although using less CPU:

$ time h2load -c 1000 -n 1000 --h1 http://127.0.0.1:8080/hello.html >/dev/null
h2load -c 1000 -n 1000 --h1 http://127.0.0.1:8080/hello.html  0.02s user 0.07s system 51% cpu 0.166 total

Obviously, reusing connections should improve performance. Confirmed with h2load:

$ time h2load -c 10 -n 1000 --h1 http://127.0.0.1:8080/hello.html >/dev/null
h2load -c 10 -n 1000 --h1 http://127.0.0.1:8080/hello.html > /dev/null  0.01s user 0.02s system 70% cpu 0.037 total

However, adding:

curl_multi_setopt(curl_handle, CURLMOPT_MAX_TOTAL_CONNECTIONS, 10L);

to multi-uv.c actually degrades performance a lot:

$ time ./multi-uv 1000
./multi-uv 1000  0.35s user 0.09s system 90% cpu 0.485 total

Even worse, the performance degradation is not linear at all. Let's do more requests in total (still limiting to 10 connections):

$ time ./multi-uv 2000
./multi-uv 2000  1.38s user 0.18s system 95% cpu 1.629 total
$ time ./multi-uv 5000
./multi-uv 5000  8.28s user 0.99s system 98% cpu 9.367 total

Am I doing something wrong here? missing an obvious optimization?

Here's what profiling with perf returns (truncated):

    81.56%     0.00%  multi-uv     multi-uv                [.] curl_multi_socket_action
            |
            ---curl_multi_socket_action
               |          
                --81.51%--multi_socket
                          |          
                          |--73.87%--multi_runsingle
                          |          |          
                          |          |--65.54%--Curl_connect
                          |          |          |          
                          |          |           --65.10%--create_conn
                          |          |                     |          
                          |          |                     |--11.41%--ConnectionExists
                          |          |                     |          |          
                          |          |                     |          |--6.37%--Curl_conncache_find_bundle
                          |          |                     |          |          |          
                          |          |                     |          |          |--4.86%--hashkey
                          |          |                     |          |          |          |          
                          |          |                     |          |          |           --4.78%--curl_msnprintf
                          |          |                     |          |          |                     |          
                          |          |                     |          |          |                      --4.70%--curl_mvsnprintf
                          |          |                     |          |          |                                |          
                          |          |                     |          |          |                                 --4.44%--dprintf_formatf
                          |          |                     |          |          |                                           |          
                          |          |                     |          |          |                                           |--1.30%--dprintf_Pass1
                          |          |                     |          |          |                                           |          
                          |          |                     |          |          |                                            --1.10%--addbyter
                          |          |                     |          |          |          
                          |          |                     |          |           --1.43%--Curl_hash_pick
                          |          |                     |          |                     |          
                          |          |                     |          |                      --0.89%--Curl_hash_str
                          |          |                     |          |          
                          |          |                     |           --1.97%--extract_if_dead
                          |          |                     |          
                          |          |                     |--7.11%--conn_free
                          |          |                     |          |          
                          |          |                     |          |--2.31%--curl_dofree
                          |          |                     |          |          |          
                          |          |                     |          |           --1.07%--curl_memlog
                          |          |                     |          |          
                          |          |                     |          |--1.58%--Curl_free_primary_ssl_config
                          |          |                     |          |          |          
                          |          |                     |          |           --0.84%--curl_dofree
                          |          |                     |          |          
                          |          |                     |           --0.52%--Curl_ssl_close
                          |          |                     |          
                          |          |                     |--6.40%--Curl_conncache_extract_oldest
                          |          |                     |          |          
                          |          |                     |           --4.44%--Curl_hash_next_element
                          |          |                     |          
                          |          |                     |--5.98%--Curl_conncache_find_bundle
                          |          |                     |          |          
                          |          |                     |          |--4.51%--hashkey
                          |          |                     |          |          |          
                          |          |                     |          |           --4.41%--curl_msnprintf
                          |          |                     |          |                     |          
                          |          |                     |          |                      --4.33%--curl_mvsnprintf
                          |          |                     |          |                                |          
                          |          |                     |          |                                 --4.07%--dprintf_formatf
                          |          |                     |          |                                           |          
                          |          |                     |          |                                           |--1.19%--dprintf_Pass1
                          |          |                     |          |                                           |          
                          |          |                     |          |                                            --0.96%--addbyter
                          |          |                     |          |          
                          |          |                     |           --1.30%--Curl_hash_pick
                          |          |                     |                     |          
                          |          |                     |                      --0.92%--Curl_hash_str
                          |          |                     |          
                          |          |                     |--5.96%--parseurlandfillconn
                          |          |                     |          |          
                          |          |                     |          |--1.99%--findprotocol
                          |          |                     |          |          
                          |          |                     |           --1.08%--Curl_dedotdotify
                          |          |                     |          
                          |          |                     |--4.32%--allocate_conn
                          |          |                     |          
                          |          |                     |--4.01%--create_conn_helper_init_proxy
                          |          |                     |          |          
                          |          |                     |           --2.53%--detect_proxy
                          |          |                     |                     |          
                          |          |                     |                     |--1.06%--Curl_strcasecompare
                          |          |                     |                     |          |          
                          |          |                     |                     |           --0.51%--Curl_raw_toupper
                          |          |                     |                     |          
                          |          |                     |                      --0.77%--curl_getenv
                          |          |                     |          
                          |          |                     |--3.88%--setup_connection_internals
                          |          |                     |          |          
                          |          |                     |          |--2.26%--Curl_free_request_state
                          |          |                     |          |          |          
                          |          |                     |          |           --1.45%--cfree@GLIBC_2.2.5
                          |          |                     |          |          
                          |          |                     |           --1.40%--Curl_http_setup_conn
                          |          |                     |          
                          |          |                     |--3.03%--Curl_clone_primary_ssl_config
                          |          |                     |          |          
                          |          |                     |          |--2.05%--__strlen_avx2
                          |          |                     |          |          
                          |          |                     |           --0.53%--curl_dostrdup
                          |          |                     |          
                          |          |                     |--1.20%--parse_remote_port
                          |          |                     |          
                          |          |                     |--0.88%--set_login
                          |          |                     |          |          
                          |          |                     |           --0.54%--curl_dostrdup
                          |          |                     |          
                          |          |                     |--0.80%--curl_dostrdup
                          |          |                     |          
                          |          |                     |--0.64%--curl_dofree
                          |          |                     |          
                          |          |                     |--0.61%--fix_hostname
                          |          |                     |          
                          |          |                     |--0.60%--cfree@GLIBC_2.2.5
                          |          |                     |          
                          |          |                      --0.55%--__strlen_avx2
                          |          |          
                          |           --4.74%--Curl_multi_process_pending_handles
                          |                     |          
                          |                     |--2.32%--Curl_expire
                          |                     |          |          
                          |                     |           --1.37%--multi_deltimeout
                          |                     |          
                          |                      --0.99%--mstate
                          |                                |          
                          |                                 --0.61%--Curl_infof
                          |          
                          |--2.80%--Curl_splaygetbest
                          |          |          
                          |           --2.24%--Curl_splay
                          |          
                          |--1.85%--add_next_timeout
                          |          
                          |--0.74%--sigpipe_ignore
                          |          
                          |--0.74%--__libc_sigaction
                          |          
                           --0.62%--0xffffffffab800158

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions