New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read file URL in chunks #21560

Merged
merged 3 commits into from Sep 7, 2018

Conversation

Projects
None yet
5 participants
@JacksonCoder
Contributor

JacksonCoder commented Aug 31, 2018

This is a very straightforward PR: essentially, I replaced a read_to_end call that occurs when processing a file URL fetch with a loop that reads the file in chunks (with a FILE_CHUNK_SIZE constant that specifies the chunk size, which I've put at 32KB), and then calls target.process_response_chunk to process the chunk. The chunk is then appended to the fetch result, and once the end of the file is reached, the result is returned.


  • ./mach build -d does not report any errors
  • ./mach test-tidy does not report any errors
  • These changes fix #21466.
  • There are tests for these changes OR
  • These changes do not require tests because there isn't really a way to observe this.

This change is Reviewable

@highfive

This comment has been minimized.

highfive commented Aug 31, 2018

Thanks for the pull request, and welcome! The Servo team is excited to review your changes, and you should hear from @Manishearth (or someone else) soon.

@highfive

This comment has been minimized.

highfive commented Aug 31, 2018

Heads up! This PR modifies the following files:

@highfive

This comment has been minimized.

highfive commented Aug 31, 2018

warning Warning warning

  • These commits modify net code, but no tests are modified. Please consider adding a test!
let mut buffer = reader.fill_buf().unwrap().to_vec();
let buffer_len = buffer.len();
bytes.append(&mut buffer);
target.process_response_chunk(buffer);

This comment has been minimized.

@jdm

jdm Aug 31, 2018

Member

We actually have a system in place that requires less manual work, so let's make use of it. wait_for_response handles both synchronous and asynchronous responses; we can make this one asynchronous by:

  1. spawning a new thread before we open the file
  2. creating a new channel and storing it in done_chan (
    // We're about to spawn a thread to be waited on here
    let (done_sender, done_receiver) = channel();
    *done_chan = Some((done_sender.clone(), done_receiver));
    )
  3. setting the body to ResponseBody::Receiving
  4. sending the file chunks across the channel
  5. setting the body to ResponseBody::Done once the file is completely read (
    if cancellation_listener.lock().unwrap().cancelled() {
    *res_body.lock().unwrap() = ResponseBody::Done(vec![]);
    let _ = done_sender.send(Data::Cancelled);
    return;
    }
    match read_block(&mut res) {
    Ok(Data::Payload(chunk)) => {
    if let ResponseBody::Receiving(ref mut body) = *res_body.lock().unwrap() {
    body.extend_from_slice(&chunk);
    let _ = done_sender.send(Data::Payload(chunk));
    }
    },
    Ok(Data::Done) | Err(_) => {
    let mut body = res_body.lock().unwrap();
    let completed_body = match *body {
    ResponseBody::Receiving(ref mut body) => {
    mem::replace(body, vec![])
    },
    _ => vec![],
    };
    *body = ResponseBody::Done(completed_body);
    let _ = done_sender.send(Data::Done);
    break;
    }
    Ok(Data::Cancelled) => unreachable!() // read_block doesn't return Data::Cancelled
    )

To ensure everything still works, try loading a page from your local machine (not from a web server) that loads an image >32kb using a relative URL.

This comment has been minimized.

@JacksonCoder

JacksonCoder Sep 1, 2018

Contributor

Would thread::spawn work for making a new thread?

This comment has been minimized.

@jdm

jdm Sep 2, 2018

Member

Yes.

This comment has been minimized.

@JacksonCoder

JacksonCoder Sep 3, 2018

Contributor

Should I still call target.process_response_chunk?

This comment has been minimized.

@jdm

jdm Sep 3, 2018

Member

That should not be necessary in the model that I described.

This comment has been minimized.

@JacksonCoder

JacksonCoder Sep 4, 2018

Contributor

I had some time to work on this today, I'm building it to see if I can load files properly. I'll push the commit if it works.

@jdm

jdm approved these changes Sep 5, 2018

@jdm

This comment has been minimized.

Member

jdm commented Sep 5, 2018

Checking files for tidiness...
./components/net/fetch/methods.rs:31: use statement is not in alphabetical order
	expected: std::sync::mpsc::channel
	found: std::str
let buffer_len = buffer.len();
if let ResponseBody::Receiving(ref mut body) = *res_body.lock().unwrap() {
body.extend_from_slice(&buffer);
let _ = done_sender.send(Data::Payload(buffer.clone()));

This comment has been minimized.

@jdm

jdm Sep 5, 2018

Member

Why is this clone necessary?

This comment has been minimized.

@JacksonCoder

JacksonCoder Sep 6, 2018

Contributor

It isn't, I forgot to remove it because I was doing some testing earlier. It's fixed now.

@jdm

This comment has been minimized.

Member

jdm commented Sep 5, 2018

Apart from those small nits, these changes look great! Thanks for working on this!

@JacksonCoder

This comment has been minimized.

Contributor

JacksonCoder commented Sep 6, 2018

No problem, I loved working on this and hope to get more involved in Servo development in the future.

@jdm

This comment has been minimized.

Member

jdm commented Sep 6, 2018

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 6, 2018

📌 Commit 430ba86 has been approved by jdm

@highfive highfive assigned jdm and unassigned Manishearth Sep 6, 2018

@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 6, 2018

⌛️ Testing commit 430ba86 with merge 1343c7d...

bors-servo added a commit that referenced this pull request Sep 6, 2018

Auto merge of #21560 - JacksonCoder:chunk-file, r=jdm
Read file URL in chunks

<!-- Please describe your changes on the following line: -->
This is a very straightforward PR: essentially, I replaced a `read_to_end` call that occurs when processing a file URL fetch with a loop that reads the file in chunks (with a `FILE_CHUNK_SIZE` constant that specifies the chunk size, which I've put at 32KB), and then calls `target.process_response_chunk` to process the chunk. The chunk is then appended to the fetch result, and once the end of the file is reached, the result is returned.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix #21466.

<!-- Either: -->
- [ ] There are tests for these changes OR
- [X] These changes do not require tests because there isn't really a way to observe this.

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/21560)
<!-- Reviewable:end -->
@bors-servo

This comment has been minimized.

Contributor

bors-servo commented Sep 7, 2018

@bors-servo bors-servo merged commit 430ba86 into servo:master Sep 7, 2018

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
homu Test successful
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment