Support for streaming large files with lookahead

[lars18th]

Hi,

The current cache and lookahead functionality is good. However, for large files in streaming it can be improved. Please, consider this:

• When playing a file from a drive mounted with the sshfs-win the current behaviour is like this: every time a small chunk is readed by the player, then a new small chunk is requestest to the remote. However, this reading way desn't saturate the SSH link. As a result, you can't smooth play the file... except if your latency is small and your bandwith is large.

You can check this comparing a COPY of a remote file with a PLAY of the same file. In the first case, your link will saturate, but in the second case the link is used, more or less, at the bitrate speed of the file. And this generates pauses with large network delays.

So my suggestion is to provide the option to:

• Enhance the lookahead buffer to large sizes (up to 4GB for example).
• And if it's necessary, use a local file cache to store parts of the file.

I hope you agree to consider it.
Regards.

[billziss-gh]

@lars18th thanks for your suggestion.

In general both operating systems and file systems implement parts of the read-ahead policy. On Windows the read-ahead policy is implemented in the operating system and its exact mechanics are not fully documented -- however it appears that in modern Windows OS'es the read-ahead policy is adapted based on how the file is used.

The file system can influence the operating system read-ahead policy by using the kernel mode CcSetReadAheadGranularityEx DDI (or the older CcSetReadAheadGranularity DDI). At this time WinFsp (the underlying kernel mode driver that SSHFS-Win uses) does not use any of these DDI's (because the newer DDI is available only on Win8+ and because the older DDI did not appear terribly useful). Perhaps this should be rectified and I will open an enhancement request on WinFsp for this.

The file system can also has its own read-ahead policy that is separate from the operating system. AFAIK the upstream SSHFS project (on which SSHFS-Win is based on) does not implement read-ahead. You may also want to open an issue there asking for read-ahead to be implemented.

[AgentOak]

The missing readahead causes really severe throughput problems (pretty much unusable) for me with media playback.

I'm trying to play from a remote sshfs mount, ~30mbps throughput and ~55ms ping.
Windows Explorer can achieve ~2000 KiB/s, which is ok but not nearly saturating the connection. mpv does much smaller reads, and as a result only reads about ~60 KiB/s from my sshfs mount. Playback is absolutely impossible. Same results with VLC.

When I configure mpv to use a 2MB buffer it will do bigger reads and can go up to ~2600 KiB/s which is a good result, but the increased buffed size introduces a lot of lag when seeking in the file. Playing several files at the same time is not working; one link will starve the other link of bandwidth, even if their combined bitrate would easily fit the bandwidth budget.

I originally reported this problem to mpv because they recently changed their cache implementation (see mpv-player/mpv#6802), but I think this is better fixed in sshfs(-win) since almost all applications are affected.

[billziss-gh]

@Marco01809 @lars18th please add your votes to winfsp/winfsp#243. This increases the chances that I will look into this enhancement sooner rather than later :)