Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate file-reading with streams #26759

Open
gterzian opened this issue Jun 3, 2020 · 0 comments
Open

Integrate file-reading with streams #26759

gterzian opened this issue Jun 3, 2020 · 0 comments

Comments

@gterzian
Copy link
Member

@gterzian gterzian commented Jun 3, 2020

Similar to #26743 (although perhaps smaller in scope and a bit easier).

When reading a file, we currently just loop and read chunks from the file "as fast as possible" and send each chunk to script over IPC.

pub fn fetch_file_in_chunks(

Instead, we could use a mechanism as described in the linked-to issue, where the stream in script would "pull" chunks of the file from net, based on it's own(and possibly user provided) queuing and high-watermark strategies.

That way, one could imagine a situation where a file is read from disk, chunks are pulled into script by a readable stream, then perhaps put through a transform stream of some kind, and then transmitted over the network as the body of a request(another stream). Then, as bytes of the request body would be transmitted over the network, this would result in more chunks being "pulled" from the file on disk(once the internal buffer of a stream would go below some desired size).

(One can also imagine a fast path where if it's just the native underlying source for the blob stream that is directly feeding data into another native underlying source for the stream of the request body, we wouldn't have to go through SpiderMonkey, and perhaps we wouldn't event have to send file data to script over IPC at all, although it might be hard to know in advance how the stream will end-up being used in script, if at all)

Currently we'd just have one thread n net reading the file as fast as possible, sending bytes to script over IPC where they would accumulate in the buffer of the blob stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.