Great acceleration on large files, but slower on small files

[etlweather]

Hello, Good work on this.

I tried this out over a very slow link (satellite) of 500-600ms latency. I used it to proxy Windows Shares (CIFS/SMB).

I saw a very good improvement on transferring medium/large files, but on files smaller than 50Mb, and very small files like a Word document of less than 1Mb, I saw speed degradation.

I'm wondering if this is something that could be remedied. I'm haven't fully delved into your code and QUIC.

It may just be due to the CIFS/SMB protocol which sitll obviously occurs despite using Hysteria.

And frankly, ultimately my end goal isn't to proxy CIFS to accelerate it. I'm really on the look out for a fast file transfer mechanism over high latency links. Something that wouldn't suffer the TCP penalties - like you tell it transfer at 50Mbps and it starts right off at that speed instead of starting slow and little by little increase in speed.

The ultimate goal is to have near instant small file copy (files less than 100MB) and fast copy for large files.

So that's how I ran into your project - I thought the QUIC protocol could be a good starting point and was looking for fast file transfer over QUIC. Didn't find much yet, but did find your project which does work in accelerating existing file transfer protocol. But not so much on small files.

[tobyxdd]

I'm not familiar with CIFS, but from what you've described I think maybe it's due to the fact that the flow control window size for each connection takes a while to converge. If that's the case I can work around it by adding an argument for customizing initial flow control window size.

Does CIFS have some kind of connection multiplexing or reuse mechanism? Or it always creates a new connection for every file transfer

[etlweather]

As far as I know, CIFS reuses the connection.

Note: I am not stuck on CIFS, that's just how things are done and I was looking to see if I could proxy CIFS through Hysteria - and it worked pretty smoothly.

Do you have experience with transferring small files over other protocols with Hysteria acceleration - like HTTP? Are small files benefiting from the acceleration?

[etlweather]

Also, maybe a bit off topic but do you think it could be workable to create a file transfer mechanism over QUIC relatively easily? Would small files benefit? Maybe a web server and client using QUIC would be a good approach?

You seem to have good experience on the subject since you produced a workable and good software around it, hence my questions... if you don't mind.

[tobyxdd]

By speed degradation do you mean that it becomes slower with Hysteria, or it's just not as fast as transferring large files? Would you mind providing some details?

[etlweather]

It was slower to transfer small files when I ran CIFS over Hysteria, compared to straight CIFS. For large files, it was faster CIFS-over-Hysteria than not.

I unfortunately didn't keep good notes of the metrics. But for larger files, I got a factor of at least 3X the speed when using CIFS over Hysteria (files of about 100MB). This is over the 500-600ms latency link.

But when transferring small files, like a Word document, or a photo (<4MB) then it was slower when I ran CIFS over Hysteria.

I have a need for both large and small files.

[tobyxdd]

From my experience a problem with many protocols (not just SMB, also FTP/SFTP, etc.) that cause transferring large numbers of tiny files to be slow is that they often spend multiple RTTs to negotiate information back and forth before a transfer actually begins, introducing noticeable delays between each file transfer and reduce the overall speed. Not sure if that's the case here or why it's even slower with Hysteria, but a temporary solution could be packing and sending the small files as one large archive.
I'll look into this when I have time, and perhaps build a tool that's optimized for transferring/syncing folders with large numbers of tiny files in the future.

[tobyxdd]

have you ever tried rsync btw?

[etlweather]

Yes. rsync tends to stall over high latency. Also, in my case, it's not "a lot of small files". It's a small file, then later, another small file, etc. So the initial connection time of rsync is killing the speed, much like this CIFS problem but maybe even worst. I was hoping to circumvent that since Hysteria sort of keeps a live connection between the client and server.

I need to try HTTP. Maybe it's going to be better.

Thanks for checking and exchanging ideas.

And I realized I could rerun my tests easily so here is some data on the transfer speed I got over this satellite link:

Small 42KB

benchcopy.exe c:\temp\sample.docx \local-hysteria-relay\test$
Took 5.3064539 seconds
0.06Mbps

benchcopy.exe c:\temp\sample.docx \remote-direct\test$
Took 7.7516866 seconds
0.04Mbps

Medium: 18.4KB

benchcopy.exe c:\temp\sample.pdf \local-hysteria-relay\test$
Took 19.5618926 seconds
7.36Mbps

benchcopy.exe c:\temp\sample.pdf \remote-direct\test$
Took 116.5437549 seconds
1.23Mbps

---

So far, Hysteria is the best WAN acceleration product I found that does not cost $100,000 and even those... aren't particularly all that good for CIFS when transferring binary files that are already compressed - they mostly "accelerate" by compressing and caching which isn't useful in my case.