Add file backend support for Blob and related #11221

izgzhen · 2016-05-17T06:58:49Z

./mach build -d does not report any errors
./mach test-tidy --faster does not report any errors
These changes fix Add file handling functionality to Blob object #10851, related to Implement File API and related support #11131
These changes do not require tests because the implementation is partial and can't work alone

Major changes

Add new backend to Blob and a BlobImpl struct to abstract multiple backends
Rewrite most interfaces of Blob to accommodate the change
Change the read behaviour of FileReader, considering the case when blob is file-backed and not cached

The design is still immature, welcome comments!

Problems to resolve

I used DOMRefCell to cache the bytes in BlobImpl, is it sound?
The interfaces (like BlobImpl::get_bytes) handle requests in a default-to-empty way when the inner DataSlice is not cached. It might be possible to handle this condition better.

This change is

highfive · 2016-05-17T06:58:53Z

Heads up! This PR modifies the following files:

@KiChjang: components/script/dom/testbinding.rs, components/script/dom/file.rs, components/script/dom/xmlhttprequest.rs, components/net_traits/filemanager_thread.rs, components/net_traits/filemanager_thread.rs, components/script/dom/htmlformelement.rs, components/script/dom/filereader.rs, components/script/dom/websocket.rs, components/script/dom/blob.rs, components/script/dom/formdata.rs

highfive · 2016-05-17T06:58:54Z

Warning

These commits modify script code, but no tests are modified. Please consider adding a test!

izgzhen · 2016-05-17T07:01:25Z

@Manishearth I refined the design a bit and I think we can push the progress of this thread in parallel with #11189

Manishearth · 2016-05-17T22:18:57Z

components/script/dom/blob.rs

    typeString: String,
    isClosed_: Cell<bool>,
 }

+#[derive(Clone, JSTraceable)]
+pub struct BlobImpl {
+    slice: DOMRefCell<Option<DataSlice>>,


Perhaps this should be an enum?

I am thinking about the possibility of caching here. If it is file-backed, we can put the cached content inside the other DataSlice field.

I'd rather have an enum here and within the FileId variant have an option for caching. As it stands slice = Some can mean cached or an in-memory slice, a clear boundary would be nicer and more Rusty.

highfive · 2016-05-18T06:15:18Z

New code was committed to pull request.

izgzhen · 2016-05-20T05:12:05Z

@Manishearth In order to get port to filemanger thread inside blob (by GlobalRef), I need to attach the filemanager_thread to worker (in workerglobalscope.rs). Currently the worker only has access to core_resource_thread (renamed from original resource_thread), rather than the combined ResourceThreads from which I can access the file manager. However I don't know if storage API (storage_thread I think) can be exposed to worker as well.

jdm · 2016-05-20T05:20:24Z

Exposing it won't hurt anything; the storage APIs aren't accessible from workers, so there hasn't been any need to make the storage thread available.

Manishearth · 2016-05-20T12:00:42Z

Any particular reason for workers not being able to use storage? If there's a security reason we should not even give workers an unused storage chan (sandboxing).

Manishearth · 2016-05-20T12:03:40Z

Seems like it used to be for security reasons but now it's just for "threading issues".

izgzhen · 2016-05-20T12:06:26Z

I didn't see any Exposed or similar in this spec http://www.w3.org/TR/2016/REC-webstorage-20160419/#the-storage-interface

Manishearth · 2016-05-20T12:11:56Z

Oh, sure, the spec doesn't let you, the question is: If a sandboxed worker was compromised with an RCE vulnerability and got access to storage data, is that an escalation of access?

Assuming that we have checks in place in the resource/storage thread that ensure that no process gets data from a different origin (we don't, but we should eventually), I don't think this is an issue. Also, I don't see us putting workers in separate processes, so if a worker thread is compromised the script thread would be compromised too anyway.

Manishearth · 2016-05-20T13:42:15Z

components/script/dom/blob.rs

+    let file_manager: IpcSender<FileManagerThreadMsg> = unimplemented!();
+    let (chan, recv) = ipc::channel().unwrap();
+    let _ = file_manager.send(FileManagerThreadMsg::ReadFile(chan, id));
+    let result = recv.recv().unwrap();


Should we be handling/propagating errors here?

I think read_file can be made Option<DataSlice> and we can throw a script-level exception or something at the dom API where it is used (like Slice)

Result<DataSlice,()> and use try! or the ? operator, but yeah

Manishearth · 2016-05-20T13:46:49Z

Almost done!

highfive · 2016-05-20T16:22:46Z

New code was committed to pull request.

bors-servo · 2016-05-21T09:12:11Z

☔ The latest upstream changes (presumably #10961) made this pull request unmergeable. Please resolve the merge conflicts.

Manishearth · 2016-05-23T05:45:25Z

components/script/dom/htmlformelement.rs

@@ -308,12 +308,17 @@ impl HTMLFormElement {
                        DispositionParam::Filename(Charset::Ext(String::from(charset.clone())),
                                                   None,
                                                   f.name().clone().into()));
+                    /// XXX: unwrap


replace it with an expect

though in this case unwrap_or(text/plain) might be better?

Not sure, can't find related spec, maybe an empty ""?

Section 4.4, it defaults to text/plain

https://tools.ietf.org/html/rfc7578#section-4.4

bors-servo · 2016-05-23T09:16:14Z

☔ The latest upstream changes (presumably #11225) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2016-05-24T09:59:59Z

☔ The latest upstream changes (presumably #11326) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2016-05-25T04:31:35Z

☔ The latest upstream changes (presumably #11368) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2016-06-01T06:09:11Z

⌛ Testing commit 43ad4ba with merge e83fb45...

Add file backend support for Blob and related - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy --faster` does not report any errors - [x] These changes fix #10851, related to #11131 - [x] These changes do not require tests because the implementation is partial and can't work alone 1. Add new backend to `Blob` and a `BlobImpl` struct to abstract multiple backends 2. Rewrite most interfaces of `Blob` to accommodate the change 3. Change the `read` behaviour of `FileReader`, considering the case when blob is file-backed and not cached The design is still immature, welcome comments! - [x] I used `DOMRefCell` to cache the bytes in `BlobImpl`, is it sound? - [x] The interfaces (like `BlobImpl::get_bytes`) handle requests in a default-to-empty way when the inner `DataSlice` is not cached. It might be possible to handle this condition better.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11221)

bors-servo · 2016-06-01T06:59:11Z

💔 Test failed - mac-rel-css

highfive · 2016-06-01T06:59:13Z

  ▶ TIMEOUT [expected PASS] /css21_dev/html4/at-charset-074.htm

  ▶ TIMEOUT [expected PASS] /css21_dev/html4/at-charset-072.htm
  │ 
  └ Shutting down the Constellation after generating an output file or exit flag specified

  ▶ TIMEOUT [expected PASS] /css21_dev/html4/attribute-value-selector-001.htm

Manishearth · 2016-06-01T07:01:49Z

@bors-servo try

Manishearth · 2016-06-01T07:01:55Z

@bors-servo retry

Intermittent timeout in /css21_dev/html4/at-charset-074.htm , /css21_dev/html4/at-charset-072.htm #11541

bors-servo · 2016-06-01T07:01:58Z

⌛ Trying commit 43ad4ba with merge decbabd...

Add file backend support for Blob and related - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy --faster` does not report any errors - [x] These changes fix #10851, related to #11131 - [x] These changes do not require tests because the implementation is partial and can't work alone 1. Add new backend to `Blob` and a `BlobImpl` struct to abstract multiple backends 2. Rewrite most interfaces of `Blob` to accommodate the change 3. Change the `read` behaviour of `FileReader`, considering the case when blob is file-backed and not cached The design is still immature, welcome comments! - [x] I used `DOMRefCell` to cache the bytes in `BlobImpl`, is it sound? - [x] The interfaces (like `BlobImpl::get_bytes`) handle requests in a default-to-empty way when the inner `DataSlice` is not cached. It might be possible to handle this condition better.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11221)

bors-servo · 2016-06-01T08:18:22Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel, mac-dev-unit, mac-rel-css, mac-rel-wpt, windows

bors-servo · 2016-06-01T08:18:23Z

⌛ Testing commit 43ad4ba with merge 7293702...

Add file backend support for Blob and related - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy --faster` does not report any errors - [x] These changes fix #10851, related to #11131 - [x] These changes do not require tests because the implementation is partial and can't work alone 1. Add new backend to `Blob` and a `BlobImpl` struct to abstract multiple backends 2. Rewrite most interfaces of `Blob` to accommodate the change 3. Change the `read` behaviour of `FileReader`, considering the case when blob is file-backed and not cached The design is still immature, welcome comments! - [x] I used `DOMRefCell` to cache the bytes in `BlobImpl`, is it sound? - [x] The interfaces (like `BlobImpl::get_bytes`) handle requests in a default-to-empty way when the inner `DataSlice` is not cached. It might be possible to handle this condition better.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11221)

bors-servo · 2016-06-01T09:03:47Z

💔 Test failed - mac-rel-css

Manishearth · 2016-06-01T09:36:20Z

@bors retry try-

interrupt

Manishearth · 2016-06-01T10:01:14Z

@bors-servo retry try-

Add file backend support for Blob and related - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy --faster` does not report any errors - [x] These changes fix #10851, related to #11131 - [x] These changes do not require tests because the implementation is partial and can't work alone 1. Add new backend to `Blob` and a `BlobImpl` struct to abstract multiple backends 2. Rewrite most interfaces of `Blob` to accommodate the change 3. Change the `read` behaviour of `FileReader`, considering the case when blob is file-backed and not cached The design is still immature, welcome comments! - [x] I used `DOMRefCell` to cache the bytes in `BlobImpl`, is it sound? - [x] The interfaces (like `BlobImpl::get_bytes`) handle requests in a default-to-empty way when the inner `DataSlice` is not cached. It might be possible to handle this condition better.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11221)

bors-servo · 2016-06-01T10:09:22Z

⌛ Testing commit 43ad4ba with merge 3d7b176...

bors-servo · 2016-06-01T10:58:54Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel, mac-dev-unit, mac-rel-css, mac-rel-wpt, windows

highfive assigned mbrubeck May 17, 2016

highfive added the S-awaiting-review There is new code that needs to be reviewed. label May 17, 2016

mbrubeck assigned Manishearth and unassigned mbrubeck May 17, 2016

Manishearth reviewed May 17, 2016
View reviewed changes

Manishearth reviewed May 20, 2016
View reviewed changes

highfive added the S-needs-rebase There are merge conflict errors. label May 21, 2016

Manishearth reviewed May 23, 2016
View reviewed changes

highfive added the S-needs-rebase There are merge conflict errors. label May 23, 2016

highfive added the S-needs-rebase There are merge conflict errors. label May 24, 2016

izgzhen mentioned this pull request May 24, 2016

Make Waterfall Green Again #11365

Closed

jdm removed the S-needs-rebase There are merge conflict errors. label May 24, 2016

jdm removed the S-blocked-on-external Something, somewhere else, needs to happen before this PR can be merged. label Jun 1, 2016

highfive removed the S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. label Jun 1, 2016

highfive added the S-tests-failed The changes caused existing tests to fail. label Jun 1, 2016

highfive added S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. and removed S-tests-failed The changes caused existing tests to fail. labels Jun 1, 2016

highfive added S-tests-failed The changes caused existing tests to fail. and removed S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. labels Jun 1, 2016

highfive added S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. and removed S-tests-failed The changes caused existing tests to fail. labels Jun 1, 2016

bors-servo merged commit 43ad4ba into servo:master Jun 1, 2016

highfive removed the S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. label Jun 1, 2016

bors-servo mentioned this pull request Jun 1, 2016

implement related service worker interface and register method #11114

Merged

Manishearth mentioned this pull request Jun 1, 2016

Intermittent timeout in /css21_dev/html4/at-charset-074.htm , /css21_dev/html4/at-charset-072.htm #11541

Closed

Add file backend support for Blob and related #11221

Add file backend support for Blob and related #11221

Conversation

izgzhen commented May 17, 2016 • edited

Major changes

Problems to resolve

highfive commented May 17, 2016

highfive commented May 17, 2016

izgzhen commented May 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

highfive commented May 18, 2016

izgzhen commented May 20, 2016

jdm commented May 20, 2016

Manishearth commented May 20, 2016

Manishearth commented May 20, 2016

izgzhen commented May 20, 2016

Manishearth commented May 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Manishearth commented May 20, 2016

highfive commented May 20, 2016

bors-servo commented May 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors-servo commented May 23, 2016

bors-servo commented May 24, 2016

bors-servo commented May 25, 2016

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

highfive commented Jun 1, 2016

Manishearth commented Jun 1, 2016

Manishearth commented Jun 1, 2016 • edited

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

Manishearth commented Jun 1, 2016

Manishearth commented Jun 1, 2016

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

izgzhen commented May 17, 2016 •

edited

Manishearth commented Jun 1, 2016 •

edited