curl: improve non-blocking STDIN performance

[denandz]

Fix adds an additional call to unpause curl, ensuring the progress function that pauses reading from STDIN is called frequently. Without this call, Curl will eventually only call the progress function once a second-ish which resulted in poor performance.

Additionally, the sleep time has been lowered to significantly improve performance (250MB/s versus 10MB/s). CPU usage remains low, with top reporting 0-1% CPU usage while waiting for input.

Fixes #17553

[denandz]

Some more info on how I arrived at the 1ms sleep time. Using https://github.com/denandz/httpstream2tcp and curl to proxy SSH traffic:

5ms delay:

:~/src/curl/build/src$ scp -o ProxyCommand="./curl -s --no-buffer 127.0.0.1:3000/stream -T . --no-progress-meter --expect100-timeout 0.01" ~/tmp/test.dat 127.0.0.1:weh
test.dat                                                6%  560MB  26.0MB/s   04:53 ETA

2ms delay:

scp -o ProxyCommand="./curl -s --no-buffer 127.0.0.1:3000/stream -T . --no-progress-meter --expect100-timeout 0.01" ~/tmp/test.dat 127.0.0.1:weh
test.dat                                                18% 1481MB 141.8MB/s   00:47 ETA

1ms delay:

scp -o ProxyCommand="./curl -s --no-buffer 127.0.0.1:3000/stream -T . --no-progress-meter --expect100-timeout 0.01" ~/tmp/test.dat 127.0.0.1:weh
test.dat                                                 27% 2269MB 243.5MB/s   00:24 ETA

Top output with a 1ms delay:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND     
 93035 doi       20   0  238880   7328   6304 S   0.0   0.0   0:00.26 curl

[bagder]

I propose an even simpler take: #17569

[denandz]

I propose an even simpler take: #17569

Marvelous. Tested, its even faster.

scp -o ProxyCommand="./curl -s --no-buffer 127.0.0.1:3000/stream -T . --no-progress-meter --expect100-timeout 0.01" ~/tmp/test.dat 127.0.0.1:weh
test.dat                                                36% 2969MB 345.5MB/s   00:15 ETA

Happy to tweak this PR or close and go with #17569 - your call.

[denandz]

I managed to tease out a little more performance by moving to a select() call on non-windows platforms, returning data sooner than the 1ms delay if its available. Not sure if this is worth the additional complexity, though....

test.dat                                                              100% 8192MB 524.2MB/s   00:15

[bagder]

I think doing a select is fine, but this PR now has a merge conflict after the #17572 merge

[denandz]

I think doing a select is fine, but this PR now has a merge conflict after the #17572 merge

All good, have tweaked this PR to address the merge conflict.

[bagder]

"This branch cannot be rebased due to conflicts"

Please rebase onto master and force-push

[denandz]

"This branch cannot be rebased due to conflicts"

Please rebase onto master and force-push

Apologies, had to wrestle rebasing a bit. I think I've got this sorted now, though! :D

[bagder]

Thanks!