CURLOPT_WRITEFUNCTION can't be made to use larger writes

[pprindeville]

I did this

Using the Easy interface, set up an HTTP transfer for a Youtube video (about 77MB long) and added some debugging to the write callback function to dump the buffer pointer and size. Then I added a calll set CURLOPT_BUFFERSIZE.

Running the code, nothing seems to affect the number of bytes sent to the write callback.

Looking at Curl_client_chop_write() at looks like there's no way to send more than CURL_MAX_WRITE_SIZE which is compiled in, regardless of what CURLOPT_BUFFERSIZE is selected.

Question: could the amount of data passed to the write function be controlled by a run-time parameter 1 <= n <= data->set.buffer_size?

I expected the following

Wanted to see fewer writes with more data per write when optimizing for large files over fast connections (1Gb/s and faster).

My call back works best when processing large amounts of data (512K might even be too small) and has time to iterate long enough to saturate the cache.

curl/libcurl version

curl 7.55.1 (x86_64-redhat-linux-gnu) libcurl/7.55.1 OpenSSL/1.1.0g zlib/1.2.11 libidn2/2.0.4 libpsl/0.18.0 (+libidn2/2.0.3) libssh2/1.8.0 nghttp2/1.25.0
Release-Date: 2017-08-14
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy Metalink PSL 

i.e. libcurl-7.55.1-9.fc27.x86_64.

operating system

Fedora 27

[pprindeville]

Somewhat related question: should setting CURLOPT_BUFFERSIZE also cause a setsockopt(fd, SOL_SOCKET, SOL_RCVBUF, ...)? Otherwise, there doesn't seem to be much point to having a large buffer unless the kernel can benefit from the process's ability to receive large amounts of data.

[bagder]

The CURLOPT_WRITEFUNCTION option documentation says The maximum amount of body data that will be passed to the write callback is defined in the curl.h header file: CURL_MAX_WRITE_SIZE. It doesn't say that the size should change based on CURLOPT_BUFFERSIZE... we can't change that behavior without risking breaking existing applications.

cause a setsockopt(fd, SOL_SOCKET, SOL_RCVBUF, ...)?

Maybe. I'm not sure that makes much of a difference in modern systems. Does it?

Otherwise, there doesn't seem to be much point to having a large buffer

There certainly is. You can just run comparisons and see for yourself, especially for SFTP over high latency connections.

[pprindeville]

we can't change that behavior without risking breaking existing applications.

Fair enough. Could we add a new knob as CURLOPT_WRITESIZE?

It would only be used by aware programs, obviously...

I'm not sure that makes much of a difference in modern systems. Does it?

It can. Depends on your bandwidth x delay product.

[pprindeville]

I should explain part of our usage scenario: we're doing passive and active test and measurement in SoHo routers (so uni- or dual-core, 32-bit embedded processors running at 800Mhz or below, typically) and under heavy load, they spend (or should spend) little time in user-space, spending most of their time in the network stack in the soft-irq handling processing packets.

I'm trying to get libcurl to pass large chunks of data (a) from the kernel to the library, and (b) from the library to the application code to optimize processing. At 800MHz, context switches can be expensive and excessive switches increase interrupt processing latency, which translates to higher jitter for transit packets...

[bagder]

I really appreciate getting your insights and feedback and if you can back up your change suggestions with numbers that prove how they actually make a real difference then I'm all for discussing how we can and should proceed to make sure you can cram out as much performance as possible from libcurl. Ideally of course without having to add new options or changing the API, but if that's what's required to boost performance then let's discuss such as well!

[pprindeville]

I'll get you some data on that when I get a chance... right now I'm dealing with a different problem having to do with pausing/unpausing connections.