[feat] Getting a URL to resources (assets, media, videos, files, etc) without loading the entire file into memory; streaming playback

[reivilibre]

Describe the problem

I'd like to make an application that can play videos from the filesystem. In some cases, the videos might be larger than the available system memory, so we don't want to load them as a Uint8Array, Blob or anything like that.

My reading of the code first introduced in #2104 AFAICT and now living at https://github.com/tauri-apps/tauri/blob/3e702cf8b15762cdca43c8d7ff6f6e8ee9670244/core/tauri/src/manager.rs#L640-L651= suggests that, when using the asset:// protocol (e.g. through convertFileSrc) the entire file will be read into memory as a Vec<u8> upon request (unless there's a Range header, in which case it will read part of the file).

Describe the solution you'd like

Naïvely, I'd like the asset:// protocol to stream the file from disk rather than buffer it all into memory.

It doesn't have to be the asset protocol; any mechanism where the Rust backend can expose a single file on the filesystem and have it streamed into a video tag on the page would be sufficient for my needs.

Alternatives considered

A workaround might be to expose the file over HTTP and stream it using a http:// URI. This kind of entirely bypasses Tauri though?

Additional context

No response

[FabianLars]

Did you see our streaming example? https://github.com/tauri-apps/tauri/tree/dev/examples/streaming
Sounds exactly like what you're looking for. The problem with that one is that it doesn't work on linux because webkitgtk didn't support headers for custom protocols until the latest release, but it's not widespread enough to depend on it yet (also, in first tests it looked like it still doesn't work correctly iirc)

[reivilibre]

I must admit I didn't come across the examples before, so thanks for linking that one and for mentioning the current status.

However I still don't think that's right.

When there's no Range header, it uses read_to_end as far as I can see, buffering the entire file into memory.

content.read_to_end(&mut buf)?;

My experience with streaming HTTP requests and video playback (specifically Firefox to be honest; if other browsers behave differently then I haven't had experience with those) is that the video will be played back in one HTTP connection unless the user seeks to a different position (at which point a new request with a Range header is sent).

In my experience, the server-side code usually reads 'small' chunks of the file into a buffer and writes them out to the stream whenever possible. I always imagined that TCP applies backpressure so that the server ends up writing only as much as the client is able to cope with; after all, neither client nor server are holding the entire file in memory.

I realise that you're not using HTTP here, but I would have expected the principle to be the same: only read the file in chunks and write out to the response (tokio::io::copy is useful for this when using Tokio).

Have I got the wrong end of the stick?

[FabianLars]

However I still don't think that's right.

Well you can modify it however you want, it's just an example.

When there's no Range header, it uses read_to_end

That's correct, but at least in the context of videos (via the video&source html tags) the Range header is present (just comment out the read_to_end line and observe that it still works without increasing mem by 150mb at once).

I thought this read_to_end fallback is for linux (which as mentioned doesn't have/had header support), not sure about that.

Edit: Because we're talking about memory consumption, note that on Windows custom protocols are leaking memory, but a MS employee is working on it: https://togithub.com/MicrosoftEdge/WebView2Feedback/issues/2171#issuecomment-1121332830

[reivilibre]

Well you can modify it however you want, it's just an example.

The problem is that the API doesn't actually permit streaming, from what I can see (You can see that https://docs.rs/tauri/1.0.0-rc.0/tauri/http/struct.Response.html expects a Vec<u8> as the body. An API that supports streaming needs to give you a way to write to a stream rather than returning the entire body at once. One example is Rocket's Stream::chunked: https://api.rocket.rs/v0.4/rocket/response/struct.Stream.html. Axum has something else.).

The example as shown does not implement what would normally be called 'streaming' IMO. If you try to play a 10 GB video with it, assuming you have less than 10GB RAM + swap, the application will crash (on Linux [since it doesn't support headers according to your message] — I later noticed the part where the example shortens the requested ranges).

That's correct, but at least in the context of videos (via the video&source html tags) the Range header is present

I'm sorry, you're right. When I looked at the network requests last time, I wasn't using a <video> tag but instead just had the video itself open.

On a small example page (https://www.w3schools.com/tags/tryit.asp?filename=tryhtml5_video), I see that there is a header of Range: bytes=0-.
However, doesn't this Range mean 'the entire file'? I checked https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range to be sure I'm not spouting nonsense.

However I now notice the example will shorten the range to 400 kiB. I'm surprised this works; I don't see on MDN anywhere that mentions that you're allowed to send a shorter range than the requested range? (If you have a reference about this being allowable, I'd be interested in seeing it!)

---

I realise I have crossed my wires a bit here, so I will sum up that I think there are kind of 2 different concepts going on here:

• Range headers (in which the browser can request a smaller portion of the file) — I call these 'partial responses'.
  • If the web view requests small enough ranges for playing the video, then we may not even care about streaming responses
  • Playback of a video involves many requests for small portions of the file.
  • <video> tags seem to do this automatically (personally tested in Firefox)
• Streaming responses (which is what I came here to talk about)
  • This is where you have only 1 request but the server doesn't have to have the entire response ready at once.

I still think streaming responses are valuable to be honest (do the webviews actually support them?) — for a start, it would avoid problems with custom headers not being supported.
Another application feature I had in mind involved letting you download/upload attachments (which could potentially be large). In this case, the browser might not even use Range headers to download the file.
(Though for that case, I believe you can get access to the file dialogue anyway and probably just do it all in Rust backend code, so I suppose it's not important)

[reivilibre]

For the WebkitGtk implementation, this code is relevant:

https://github.com/tauri-apps/wry/blob/494a11057f9ddd2bf4bcecdc96b43ed95c5bd08e/src/webview/webkitgtk/web_context.rs#L243-L244=

To support streamed responses, it looks like we'd have to use a different kind of InputStream other than MemoryInputStream. The good news is that this definitely seems possible. It probably needs a bit of massaging into the API though.

[Bandonker]

I've run into this exact streaming issue building audio apps in Tauri! The memory consumption with large files was insane.

I ended up creating a library to handle called desktop-audio-proxy. It basically sets up a streaming proxy that handles range requests properly so you can stream huge files without loading them into memory.

Here's how I use it:

import { startProxyServer, TauriAudioService } from 'desktop-audio-proxy';

// Start the streaming server
const proxyServer = await startProxyServer({
  port: 3001,
  enableStreaming: true  // This handles range requests for seeking
});

const audioService = new TauriAudioService({
  proxyUrl: 'http://localhost:3001'
});

// Now I can stream any size file
const streamUrl = await audioService.getStreamableUrl('/path/to/massive-audio.wav');
audioElement.src = streamUrl; // Plays immediately, no memory issues

The key thing is it supports HTTP range requests, so the browser only loads chunks as needed. I've tested it with 4GB+ audio files and memory usage stays constant around 10-15MB. Seeking is instant too since it doesn't need to re-download.

It automatically detects you're in Tauri and handles the file protocol conversion stuff internally. I've used it for podcasts, long recordings, music libraries, etc.

The proxy handles all the streaming complexity while your audio/video elements just get normal HTTP URLs that work exactly like you'd expect in a browser.

DAP if you want to try.

Hope this helps with the large file streaming use case!

[chrox]

Nice plugin. However, Tauri 2.0 natively supports random access of big files via @tauri-apps/plugin-fs and the asset protocol also supports HTTP range requests. See my NativeFile and RemoteFile implementation using both of the approaches: https://github.com/readest/readest/blob/main/apps/readest-app/src/utils/file.ts

[rupisaini123]

Nice plugin. However, Tauri 2.0 natively supports random access of big files via @tauri-apps/plugin-fs and the asset protocol also supports HTTP range requests. See my NativeFile and RemoteFile implementation using both of the approaches: https://github.com/readest/readest/blob/main/apps/readest-app/src/utils/file.ts

Hi @chrox,
I don't think RemoteFile will be able to handle cors issue, right?

[chrox]

Hi @chrox,
I don't think RemoteFile will be able to handle cors issue, right?

No, it cannot if you use window.fetch but there is no CORS issues if you fetch with the plugin-http.