Add file backend support for Blob and related #11221

izgzhen · 2016-05-17T06:58:49Z

./mach build -d does not report any errors
./mach test-tidy --faster does not report any errors
These changes fix #10851, related to #11131
These changes do not require tests because the implementation is partial and can't work alone

Major changes

Add new backend to Blob and a BlobImpl struct to abstract multiple backends
Rewrite most interfaces of Blob to accommodate the change
Change the read behaviour of FileReader, considering the case when blob is file-backed and not cached

The design is still immature, welcome comments!

Problems to resolve

I used DOMRefCell to cache the bytes in BlobImpl, is it sound?
The interfaces (like BlobImpl::get_bytes) handle requests in a default-to-empty way when the inner DataSlice is not cached. It might be possible to handle this condition better.

This change is

highfive · 2016-05-17T06:58:53Z

Heads up! This PR modifies the following files:

@KiChjang: components/script/dom/testbinding.rs, components/script/dom/file.rs, components/script/dom/xmlhttprequest.rs, components/net_traits/filemanager_thread.rs, components/net_traits/filemanager_thread.rs, components/script/dom/htmlformelement.rs, components/script/dom/filereader.rs, components/script/dom/websocket.rs, components/script/dom/blob.rs, components/script/dom/formdata.rs

highfive · 2016-05-17T06:58:54Z

Warning

These commits modify script code, but no tests are modified. Please consider adding a test!

izgzhen · 2016-05-17T07:01:25Z

@Manishearth I refined the design a bit and I think we can push the progress of this thread in parallel with #11189

Manishearth · 2016-05-17T22:18:57Z

components/script/dom/blob.rs

    typeString: String,
    isClosed_: Cell<bool>,
 }

+#[derive(Clone, JSTraceable)]
+pub struct BlobImpl {
+    slice: DOMRefCell<Option<DataSlice>>,


Perhaps this should be an enum?

I am thinking about the possibility of caching here. If it is file-backed, we can put the cached content inside the other DataSlice field.

I'd rather have an enum here and within the FileId variant have an option for caching. As it stands slice = Some can mean cached or an in-memory slice, a clear boundary would be nicer and more Rusty.

Manishearth · 2016-05-17T22:19:46Z

components/script/dom/blob.rs

+    }
+
+    pub fn get_bytes(&self) -> Vec<u8> {
+        match *self.slice.borrow() {


what if it's a file?

Currently I don't put any read operations in blob.rs, which means that the user needs to check if it is cached and then cache if necessary. Or, if the file is not cached, a empty slice or stuff with similar semantics will be returned.

I feel like blob.rs should handle get_bytes either way. Of course it shouldn't be the one doing the reading, but it should talk to the resource task and fetch the reads and handle caching.

It am not sure if it is possible to fetch a reference to the window within Blob itself, which means that I can access script_thread stuff.

blob.global() should work

highfive · 2016-05-18T06:15:18Z

New code was committed to pull request.

izgzhen · 2016-05-20T05:12:05Z

@Manishearth In order to get port to filemanger thread inside blob (by GlobalRef), I need to attach the filemanager_thread to worker (in workerglobalscope.rs). Currently the worker only has access to core_resource_thread (renamed from original resource_thread), rather than the combined ResourceThreads from which I can access the file manager. However I don't know if storage API (storage_thread I think) can be exposed to worker as well.

jdm · 2016-05-20T05:20:24Z

Exposing it won't hurt anything; the storage APIs aren't accessible from workers, so there hasn't been any need to make the storage thread available.

Manishearth · 2016-05-20T12:00:42Z

Any particular reason for workers not being able to use storage? If there's a security reason we should not even give workers an unused storage chan (sandboxing).

Manishearth · 2016-05-20T12:03:40Z

Seems like it used to be for security reasons but now it's just for "threading issues".

izgzhen · 2016-05-20T12:06:26Z

I didn't see any Exposed or similar in this spec http://www.w3.org/TR/2016/REC-webstorage-20160419/#the-storage-interface

Manishearth · 2016-05-20T12:11:56Z

Oh, sure, the spec doesn't let you, the question is: If a sandboxed worker was compromised with an RCE vulnerability and got access to storage data, is that an escalation of access?

Assuming that we have checks in place in the resource/storage thread that ensure that no process gets data from a different origin (we don't, but we should eventually), I don't think this is an issue. Also, I don't see us putting workers in separate processes, so if a worker thread is compromised the script thread would be compromised too anyway.

Manishearth · 2016-05-20T13:42:15Z

components/script/dom/blob.rs

+    let file_manager: IpcSender<FileManagerThreadMsg> = unimplemented!();
+    let (chan, recv) = ipc::channel().unwrap();
+    let _ = file_manager.send(FileManagerThreadMsg::ReadFile(chan, id));
+    let result = recv.recv().unwrap();


Should we be handling/propagating errors here?

I think read_file can be made Option<DataSlice> and we can throw a script-level exception or something at the dom API where it is used (like Slice)

Result<DataSlice,()> and use try! or the ? operator, but yeah

Manishearth · 2016-05-20T13:43:37Z

components/script/dom/htmlformelement.rs

@@ -313,7 +313,8 @@ impl HTMLFormElement {
                        content_disposition,
                        content_type));

-                    result.push_str(from_utf8(&f.upcast::<Blob>().get_data().get_bytes()).unwrap());
+                    let slice = f.upcast::<Blob>().get_slice();
+                    result.push_str(from_utf8(&slice.get_bytes()).unwrap());


Not necessarily utf8 here? The charset can change.

We should handle errors too.

Manishearth · 2016-05-20T13:46:49Z

Almost done!

highfive · 2016-05-20T16:22:46Z

New code was committed to pull request.

bors-servo · 2016-05-21T09:12:11Z

☔ The latest upstream changes (presumably #10961) made this pull request unmergeable. Please resolve the merge conflicts.

Manishearth · 2016-05-23T05:45:25Z

components/script/dom/htmlformelement.rs

@@ -308,12 +308,17 @@ impl HTMLFormElement {
                        DispositionParam::Filename(Charset::Ext(String::from(charset.clone())),
                                                   None,
                                                   f.name().clone().into()));
+                    /// XXX: unwrap


replace it with an expect

though in this case unwrap_or(text/plain) might be better?

Not sure, can't find related spec, maybe an empty ""?

Section 4.4, it defaults to text/plain

https://tools.ietf.org/html/rfc7578#section-4.4

bors-servo · 2016-05-23T09:16:14Z

☔ The latest upstream changes (presumably #11225) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2016-06-01T06:09:11Z

⌛ Testing commit 43ad4ba with merge e83fb45...


        Auto merge of #11221 - izgzhen:blob-file-backend, r=Manishearth

Add file backend support for Blob and related - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy --faster` does not report any errors - [x] These changes fix #10851, related to #11131 - [x] These changes do not require tests because the implementation is partial and can't work alone 1. Add new backend to `Blob` and a `BlobImpl` struct to abstract multiple backends 2. Rewrite most interfaces of `Blob` to accommodate the change 3. Change the `read` behaviour of `FileReader`, considering the case when blob is file-backed and not cached The design is still immature, welcome comments! - [x] I used `DOMRefCell` to cache the bytes in `BlobImpl`, is it sound? - [x] The interfaces (like `BlobImpl::get_bytes`) handle requests in a default-to-empty way when the inner `DataSlice` is not cached. It might be possible to handle this condition better.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11221)

bors-servo · 2016-06-01T06:59:11Z

💔 Test failed - mac-rel-css

highfive · 2016-06-01T06:59:13Z

  ▶ TIMEOUT [expected PASS] /css21_dev/html4/at-charset-074.htm

  ▶ TIMEOUT [expected PASS] /css21_dev/html4/at-charset-072.htm
  │ 
  └ Shutting down the Constellation after generating an output file or exit flag specified

  ▶ TIMEOUT [expected PASS] /css21_dev/html4/attribute-value-selector-001.htm

Manishearth · 2016-06-01T07:01:49Z

@bors-servo try

Manishearth · 2016-06-01T07:01:55Z

@bors-servo retry

#11541

bors-servo · 2016-06-01T07:01:58Z

⌛ Trying commit 43ad4ba with merge decbabd...


        Auto merge of #11221 - izgzhen:blob-file-backend, r=<try>

Add file backend support for Blob and related - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy --faster` does not report any errors - [x] These changes fix #10851, related to #11131 - [x] These changes do not require tests because the implementation is partial and can't work alone 1. Add new backend to `Blob` and a `BlobImpl` struct to abstract multiple backends 2. Rewrite most interfaces of `Blob` to accommodate the change 3. Change the `read` behaviour of `FileReader`, considering the case when blob is file-backed and not cached The design is still immature, welcome comments! - [x] I used `DOMRefCell` to cache the bytes in `BlobImpl`, is it sound? - [x] The interfaces (like `BlobImpl::get_bytes`) handle requests in a default-to-empty way when the inner `DataSlice` is not cached. It might be possible to handle this condition better.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11221)

bors-servo · 2016-06-01T08:18:22Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel, mac-dev-unit, mac-rel-css, mac-rel-wpt, windows

bors-servo · 2016-06-01T08:18:23Z

⌛ Testing commit 43ad4ba with merge 7293702...


        Auto merge of #11221 - izgzhen:blob-file-backend, r=Manishearth

Add file backend support for Blob and related - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy --faster` does not report any errors - [x] These changes fix #10851, related to #11131 - [x] These changes do not require tests because the implementation is partial and can't work alone 1. Add new backend to `Blob` and a `BlobImpl` struct to abstract multiple backends 2. Rewrite most interfaces of `Blob` to accommodate the change 3. Change the `read` behaviour of `FileReader`, considering the case when blob is file-backed and not cached The design is still immature, welcome comments! - [x] I used `DOMRefCell` to cache the bytes in `BlobImpl`, is it sound? - [x] The interfaces (like `BlobImpl::get_bytes`) handle requests in a default-to-empty way when the inner `DataSlice` is not cached. It might be possible to handle this condition better.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11221)

bors-servo · 2016-06-01T09:03:47Z

💔 Test failed - mac-rel-css

Manishearth · 2016-06-01T09:36:20Z

@bors retry try-

interrupt

Manishearth · 2016-06-01T10:01:14Z

@bors-servo retry try-


        Auto merge of #11221 - izgzhen:blob-file-backend, r=Manishearth

Add file backend support for Blob and related - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy --faster` does not report any errors - [x] These changes fix #10851, related to #11131 - [x] These changes do not require tests because the implementation is partial and can't work alone 1. Add new backend to `Blob` and a `BlobImpl` struct to abstract multiple backends 2. Rewrite most interfaces of `Blob` to accommodate the change 3. Change the `read` behaviour of `FileReader`, considering the case when blob is file-backed and not cached The design is still immature, welcome comments! - [x] I used `DOMRefCell` to cache the bytes in `BlobImpl`, is it sound? - [x] The interfaces (like `BlobImpl::get_bytes`) handle requests in a default-to-empty way when the inner `DataSlice` is not cached. It might be possible to handle this condition better.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11221)

bors-servo · 2016-06-01T10:09:22Z

⌛ Testing commit 43ad4ba with merge 3d7b176...

bors-servo · 2016-06-01T10:58:54Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel, mac-dev-unit, mac-rel-css, mac-rel-wpt, windows

highfive assigned mbrubeck May 17, 2016

highfive added the S-awaiting-review label May 17, 2016

mbrubeck assigned Manishearth and unassigned mbrubeck May 17, 2016

Manishearth reviewed May 17, 2016
View changes

izgzhen force-pushed the izgzhen:blob-file-backend branch from 87adc31 to 3c194a1 May 18, 2016

Manishearth reviewed May 20, 2016
View changes

izgzhen force-pushed the izgzhen:blob-file-backend branch from 3c194a1 to 2579570 May 20, 2016

highfive added the S-needs-rebase label May 21, 2016

izgzhen force-pushed the izgzhen:blob-file-backend branch from 2579570 to bc888d1 May 21, 2016

Manishearth reviewed May 23, 2016
View changes

izgzhen force-pushed the izgzhen:blob-file-backend branch from bc888d1 to 4bf9943 May 23, 2016

highfive added the S-needs-rebase label May 23, 2016

izgzhen force-pushed the izgzhen:blob-file-backend branch from 4bf9943 to 13fb790 May 23, 2016

jdm removed the S-blocked-on-external label Jun 1, 2016

highfive removed the S-awaiting-merge label Jun 1, 2016

highfive added the S-tests-failed label Jun 1, 2016

highfive added S-awaiting-merge and removed S-tests-failed labels Jun 1, 2016

highfive added S-tests-failed and removed S-awaiting-merge labels Jun 1, 2016

highfive added S-awaiting-merge and removed S-tests-failed labels Jun 1, 2016

bors-servo merged commit 43ad4ba into servo:master Jun 1, 2016
2 of 3 checks passed

2 of 3 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

highfive removed the S-awaiting-merge label Jun 1, 2016

bors-servo mentioned this pull request Jun 1, 2016

implement related service worker interface and register method #11114

Merged

Manishearth mentioned this pull request Jun 1, 2016

Intermittent timeout in /css21_dev/html4/at-charset-074.htm , /css21_dev/html4/at-charset-072.htm #11541

Closed

servo / servo

Join GitHub today

Add file backend support for Blob and related #11221

Add file backend support for Blob and related #11221

Conversation

izgzhen commented May 17, 2016 • edited

Major changes

Problems to resolve

highfive commented May 17, 2016

highfive commented May 17, 2016

izgzhen commented May 17, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

izgzhen May 18, 2016 • edited

This comment has been minimized.

highfive commented May 18, 2016

izgzhen commented May 20, 2016

jdm commented May 20, 2016

Manishearth commented May 20, 2016

Manishearth commented May 20, 2016

izgzhen commented May 20, 2016

Manishearth commented May 20, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Manishearth May 20, 2016 • edited

Manishearth commented May 20, 2016

highfive commented May 20, 2016

bors-servo commented May 21, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors-servo commented May 23, 2016

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

highfive commented Jun 1, 2016

Manishearth commented Jun 1, 2016

Manishearth commented Jun 1, 2016 • edited

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

Manishearth commented Jun 1, 2016

Manishearth commented Jun 1, 2016

bors-servo commented Jun 1, 2016

bors-servo commented Jun 1, 2016

izgzhen commented May 17, 2016 •

edited

izgzhen May 18, 2016 •

edited

Manishearth May 20, 2016 •

edited

Manishearth commented Jun 1, 2016 •

edited