--write-out values for total_time, download_size, and speed_download do not agree

[jbromley]

I did this

Running on Ubuntu 20.10 using curl 7.76.1 (both from Ubuntu's repo and compiled from source) I was collecting some download speed statistics using the --write-out options with the following command

curl -s -w "%{url_effective},%{time_total},%{size_download},%{speed_download}\n" -o /dev/null -L ${f}

where ${f} is the file being downloaded.

The following shows a sample of the output I received:

http://localhost:8000/html/index.html,0.001196,4682,4682000.000
http://localhost:8000/downloads/index.html,0.001477,469,469000.000
http://localhost:8000/downloads/1k.bin,0.001758,1024,1024000.000
http://localhost:8000/downloads/2k.bin,0.001454,2048,2048000.000
http://localhost:8000/downloads/4k.bin,0.001581,4096,4096000.000
http://localhost:8000/downloads/8k.bin,0.001175,8192,8192000.000

Note that the value reported for speed_download is not equal to the size_download divided by the time_total. Instead it appears that the time_total has been truncated to the millisecond before calculating the speed_download. On my system, clock_getres for the monotonic clock indicates a 1 nanosecond resolution, so I have no reason to believe the printed times (to microsecond resolution) are incorrect.

I expected the following

I expect one of two things depending on whether or not the resolution of time measurements is 1 millisecond or 1 microsecond.

If the resolution for times is 1 millisecond, I would expect times to be printed with no more than three decimal places (i.e. millisecond resolution) and in this case the speed_download number is correct.

If the resolution for times is 1 microsecond, as it appears to be from the six places after the decimal, I expect speed_download to use the full value of the time for its calculation rather than the time truncated to milliseconds.

curl/libcurl version

Ubuntu version:
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh/0.9.3/openssl/zlib nghttp2/1.41.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Compiled from source version:
curl 7.76.1-DEV (x86_64-pc-linux-gnu) libcurl/7.76.1-DEV OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.8.0 nghttp2/1.41.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB PSL SSL TLS-SRP UnixSockets

operating system

Linux yemaya 5.11.0-7614-generic #15~1618626693~20.10~ecb25cd-Ubuntu SMP Thu Apr 22 16:00:45 UTC x86_64 x86_64 x86_64 GNU/Linux

[jbromley]

I did a little more digging and it appears the culprit might be the tvdiff function in tool_util.c. It converts everything to milliseconds when computing the difference of two timeval structs (which themselves have microsecond resolution). This function is used at tool_progress.c line 266 to calculate what seems to be the download speed.

I feel like I could fix this, but I am concerned that perhaps there are many places where it is assumed the time difference is in milliseconds.

[bagder]

I don't think it is at all related to tvdiff. The value for speed_download is extracted from libcurl using CURLINFO_SPEED_DOWNLOAD_T.

In libcurl, that value is calculated here:

curl/lib/progress.c

Lines 386 to 390 in d698c70

	/* The average download speed this far */
	if(dl < CURL_OFF_T_MAX/1000)
	data->progress.dlspeed = (dl * 1000 / (timespent_ms>0?timespent_ms:1));
	else
	data->progress.dlspeed = (dl / (timespent>0?timespent:1));

It does indeed use millisecond precision there.

I'll create a PR with what I believe could be an appropriate fix would be neat if you could try it out!

[jbromley]

I can definitely test out any fixes you might make.

[bagder]

Then check out #7020!