Repository navigation
Description
I did this
Make sure you have a file URL with high download speed; a local disk, and a network SMB resource with high upload speed (either mapped as a disk letter, or accessible via network share URI).
First, try to download the file with --output option pointing onto a file located on the local disk, and mark the speed (might need a few seconds for it to stabilize).
Second, try the same, but with --output pointing to the SMB share, and compare the speed.
In my example, I set up a local HTTP server using python -mhttp.server for a directory with some Linux ISO distributions on my local hard drive (max speed for downloading is 250-300 Mbytes/s); and for the destination I used an SMB share on my locally running CentOS 7 virtual machine (transfer speed varies, but is about 100-250 Mbytes/s). This way I made sure I'm not throttled by my ISP, or the upstream server. But my friend found this issue when downloading a file from Internet via 500 Mbps connection onto his NAS, so it doesn't have to be all local.
My results:
C:\Programs\curl>curl "http://127.0.0.1:8000/ubuntu-22.04.2-desktop-amd64.iso" -o c:\Temp\a.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4699M 100 4699M 0 0 244M 0 0:00:19 0:00:19 --:--:-- 255M
C:\Programs\curl>curl "http://127.0.0.1:8000/ubuntu-22.04.2-desktop-amd64.iso" -o \\192.168.56.101\flint\tmp\a.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4699M 100 4699M 0 0 22.6M 0 0:03:27 0:03:27 --:--:-- 22.7M
In other words, the speed is over 10 times lower when saving onto SMB. To confirm that the SMB itself is not slow, here is the measurement of direct copy of the same file to the same SMB location:
C:\Programs\curl>dir D:\Distribs\OS\ubuntu-20.04.6-live-server-amd64.iso & echo %TIME% & copy D:\Distribs\OS\ubuntu-20.04.6-live-server-amd64.iso \\192.168.56.101\flint\tmp\a.iso & cmd /v:on /c "echo !TIME!"
Volume in drive D has no label.
Volume Serial Number is 2461-9C46
Directory of D:\Distribs\OS
25.07.2023 23:56 1 487 339 520 ubuntu-20.04.6-live-server-amd64.iso
1 File(s) 1 487 339 520 bytes
0 Dir(s) 1 267 068 116 992 bytes free
20:52:03,98
1 file(s) copied.
20:52:15,63
which gives us about 122 Mbytes/s. (And it's not because of caching; I monitored for network activity, and it stopped as soon as the file copying finished.)
I expected the following
The download speed onto SMB should be roughly 100-200 Mbytes/s, as the hardware permits.
curl/libcurl version
Reproduced on several different builds:
- Downloaded from curl.se:
curl 8.5.0 (x86_64-w64-mingw32) libcurl/8.5.0 LibreSSL/3.8.2 (Schannel) zlib/1.3 brotli/1.1.0 zstd/1.5.5 WinIDN libssh2/1.11.0 nghttp2/1.58.0 ngtcp2/1.1.0 nghttp3/1.1.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe UnixSockets zstd
- Preinstalled in Windows 11:
curl 8.4.0 (Windows) libcurl/8.4.0 Schannel WinIDN
Release-Date: 2023-10-11
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI threadsafe Unicode UnixSockets
- Built from source using Visual Studio:
curl 8.5.0 (x86_64-pc-win32) libcurl/8.5.0 OpenSSL/3.2.0 zlib/1.3
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM SSL threadsafe UnixSockets
operating system
Windows 10 22H2 Pro x64 (10.0.19045.3570)
Windows 11 23H2 Pro x64 (10.0.22631.3007)
Activity
jay commentedon Jan 21, 2024
Maybe the fwrite default buffer size is too small? IIRC it's 4k in Windows. However I suspect in most cases microsoft's CRT functions are calling Windows API functions which have their own buffering and I'm sure there would be another layer of buffering for SMB so it didn't try to write 4k chunks repeatedly....
I made a test branch that monitors the cumulative time it takes to fwrite/fflush/fclose the output files in Windows, get it here:
https://github.com/curl/curl/compare/master...jay:curl:win32_measure_fwrite_time?expand=1
Try outputting to a file via SMB with different values (as seen in the example in the commit message)
I used file:// input above to rule out a server issue retrieving the input
CaptainFlint commentedon Jan 22, 2024
I applied the patch on top of 8.5.0, but could not compile it, because in my VS 2019 the internals of the FILE type are completely obscured; the definition is only:
and therefore I get compile errors about
_bufsizbeing not a member of_iobuf.However, since the
_bufsizis only accessed in some precautionary check code that does not affect any functionality, I commented out that wholeifblock and compiled successfully without it. And I confirm that the block size, indeed, does the trick!tool: debug fwrite times in Windows (NOT FOR UPSTREAM)
jay commentedon Jan 23, 2024
Thanks for testing. I have amended the commit to remove the _bufsiz check.
That's quite a difference. I tested with a 1GB file and I also see a difference. 38 seconds vs 11 seconds if I set a 10MB FILE buffer.
I also monitored the Windows API calls to see what the Microsoft CRT is actually calling:
The file is opened as I would expect with CreateFile:
Then it's written:
With default FILE buffer: ~77,000 calls to WriteFile
With 10MB FILE buffer: ~1500 calls to WriteFile
That's staggering. I can't believe Windows doesn't handle this automatically.edit: actually I guess that amount of calls would be expected however I'm surprised Windows doesn't bundle small writes with some larger intermediate buffer before sending over the network.
jay commentedon Jan 25, 2024
Can anyone else in Windows reproduce this?
edmcln commentedon Jan 25, 2024
I did a few tests with the official version of the curl. The results are as follows:
First Method (Using --upload-file):
Second Method (using --output):
Third Method (Using internal Copy command):
The initial results make sense, and I think the SMB protocol version, buffer size, caching, etc. play some role. However, a more in-depth analysis (User+Kernel mode) is required to narrow down the root causes.
Note: The First Method uses SMBv1 (presumably curl project implementation), whereas the other Methods use SMBv2/3 (depending on the OS).
jay commentedon Jan 30, 2024
Thanks for testing. I think we can say that this is a Windows issue not a curl issue. We could mitigate the effect of Windows' smaller SMB buffer by writing larger blocks for network files. However, I'm hesitant to do that because even if we can reliably detect HANDLEs as network files there may be unwanted performance effects on other protocols. Also, it would use a lot more memory per file which would be a problem if many files were open in parallel mode.
edmcln commentedon Jan 31, 2024
I'm not sure what's wrong with the new build (8.6.0), but the write speed is ~3.x times slower than the 8.5.0 or 8.4.0. I monitored the ReadFile and WriteFile API calls, and the buffer size is indeed different. The read buffer is 8192 bytes vs 102400 in previous versions, and write buffer is 4096 bytes vs 16384.
jay commentedon Jan 31, 2024
@icing could this be from your changes to file protocol
bagder commentedon Jan 31, 2024
Probably mostly because of a6c9a33.
bagder commentedon Jan 31, 2024
But I don't see us reverting this change just because SMB on Windows is silly.
jay commentedon Jan 31, 2024
yes but why are we doing that with the smaller read buffer
bagder commentedon Jan 31, 2024
Because we no longer use the download buffer - it is now stack based. Because it does not need to be big to do fast transfers everywhere else. And of course because we did not anticipate this particular thing to do even worse.
CaptainFlint commentedon Jan 31, 2024
I was going to suggest adding an option for changing the buffer size. Basically what @jay implemented earlier for testing (via CURL_OUTFILE_BUFSIZE), only now officially, with a command line option. This way the users who need downloading onto SMB would be able to get full speed, without affecting others. But if you decided to switch buffer located on stack, that may not be possible anymore...
icing commentedon Feb 1, 2024
@bagder we could replace the stack buffers with allocated ones, using
data->set.buffer_size. They need to only exist for the duration of the loops, directly freed again. If you like that, I can make a PR for this.bagder commentedon Feb 1, 2024
Since file:// transfers are also transfers, we could also make use of the download buffer for this...
file, use xfer buf for transfers
icing commentedon Feb 13, 2024
I made #12932 which uses the multi's transfer buffer for file: operations. This should restore the buffer sizes and behaviour exactly as it was in curl 8.5.0.
file: use xfer buf for file:// transfers
bagder commentedon Feb 18, 2024
Fixed via #12932