Properly set origin of fetch requests #16508

fnune · 2017-04-18T05:15:45Z

These changes aim to fix #15247

./mach build -d does not report any errors
./mach test-tidy does not report any errors
These changes fix Fetch is not setting origin correctly. #15247 (github issue number if applicable).

There are tests for these changes
These changes do not require tests because cors is already tested with different origins

These changes require changes in tests, but I need help with that (see comments below).

This change is

highfive · 2017-04-18T05:15:51Z

Thanks for the pull request, and welcome! The Servo team is excited to review your changes, and you should hear from @metajack (or someone else) soon.

highfive · 2017-04-18T05:15:52Z

Heads up! This PR modifies the following files:

@fitzgen: components/script/dom/globalscope.rs, components/script/fetch.rs
@KiChjang: components/script/dom/globalscope.rs, components/script/fetch.rs

highfive · 2017-04-18T05:15:53Z

Warning

These commits modify script code, but no tests are modified. Please consider adding a test!

metajack · 2017-04-18T05:26:03Z

@bors-servo r+

bors-servo · 2017-04-18T05:26:03Z

📌 Commit af91972 has been approved by metajack

bors-servo · 2017-04-18T05:26:07Z

⌛ Testing commit af91972 with merge 43e4d17...

Fetch set origin  These changes are a WIP, aiming to fix #15247 ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [ ] These changes fix #15247 (github issue number if applicable).  - [ ] There are tests for these changes OR - [x] These changes do not require tests because cors is already tested with different origins    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/16508)

fnune · 2017-04-18T05:36:48Z

@metajack Hey! Thanks for the review but I think the PR needs some more work, I updated the description

metajack · 2017-04-18T05:38:01Z

@bors-servo r-

metajack · 2017-04-18T05:38:53Z

I'm in Tokyo at the CSS WG meeting, so probably one of @jdm or @asajeffrey or @Manishearth should provide the feedback you are asking for.

fnune · 2017-04-18T05:40:51Z

@metajack That's fine, thanks! Have a nice one.

bors-servo · 2017-04-18T05:44:13Z

💔 Test failed - mac-rel-wpt1

metajack · 2017-04-18T06:17:59Z

@brainlessdeveloper You'll also want to update the test results since presumably this made something pass now :)

KiChjang · 2017-04-18T06:42:57Z

components/script/dom/globalscope.rs

@@ -263,6 +263,11 @@ impl GlobalScope {
        unreachable!();
    }

+    /// Get the origin URL for the current global object, shorthand for GlobalScope::current().get_url()
+    pub fn origin() -> ServoUrl {


This function is a misnomer then -- it's not really getting the origin of an environment settings object, since it returns a ServoUrl. In view of this, I think you should disregard my earlier suggestion and remove this function altogether.

You're right. Won't save that much typing anyway either. Can you help me with my feedback requests in the PR description?

fnune · 2017-04-19T21:24:08Z

After carefully reading the spec, I have two concerns to present to you. Maybe I am just extremely confused, but it seems to me that we can't just pass a ServoUrl to RequestInit.

I drew this little flow chart, I hope it doesn't confuse you:

When setting origin here, we instantiate a RequestInit from a Request. About this step, the spec says:

A request has an associated origin, which is "client" or an origin. Unless stated otherwise it is "client". "client" is changed to an origin during fetching. It provides a convenient way for standards to not have to set request’s origin. Request’s origin can be changed during redirects too.

Changed to an origin during fetching means we need to pass the enum ImmutableOrigin to RequestInit. As per a part of the spec, an origin can be either an opaque origin, or a tuple of a schema, a host, and a port (and a domain that's always null). This is reflected neatly by the ImmutableOrigin tuple. As far as I've understood, the origin will be an opaque origin if it's generated from a file path. The spec says that an opaque origin will always be a string containing null.

The ImmutableOrigin struct contains a couple of handy methods that deal with the logic to figure out if the origin is an opaque origin or if it's a tuple. Another method, the unicode_serialization method, returns a string from an ImmutableOrigin. This string will be either "null" if it's an opaque origin or contain the schema, host, and port if it's a tuple.

In my doodle, there's Q1 (in blue), representing my first concern: I can instantiate a Url struct from a tuple origin, and then create a ServoUrl from it, and pass it to RequestInit's origin field. But I can't do this with an opaque origin, because neither Url or ServoUrl will take a string "null" to be instantiated.

About my second concern (Q2 in blue in the doodle): the spec says a request's "client" is changed to an origin during fetching. But there's still the possibility that the Request from which we're trying to instantiate a RequestInit contains an Origin of type Client.

I think RequestInit should not take an origin of type ServoUrl, I think it should take... an origin of type Origin! And it should deal with transforming it to a string afterwards. The problem is, if you grep for RequestInit { in the project, it appears 105 times, and I'm quite sure all of them serve an origin in a different way, and none of them account for the fact that the origin should be "null" in some cases. So if I'm right, there's quite a bit of refactoring work.

Please, tell me if I've missed something. To me it seems just passing GlobalScope::current().get_url() blindly does not comply with the spec, and will lead to a lot of tears in the future.

jdm · 2017-04-20T10:32:36Z

You are correct, and that's what the TODO in request_init_from_request is referencing. Going back to my suggestion of adding an origin method to GlobalScope, we can have the Window implementation return the associated document's immutable origin, and for workers return an origin based on get_url().origin(). I suspect the 105 uses you see are overcounting; here's the list of the important ones:

components/gfx/font_cache_thread.rs
215:                let request = RequestInit {

components/net_traits/request.rs
164:    fn default() -> RequestInit {

components/script/dom/dedicatedworkerglobalscope.rs
176:            let request = RequestInit {

components/script/dom/eventsource.rs
333:    pub fn request(&self) -> RequestInit {
358:        let mut request = RequestInit {

components/script/dom/htmlimageelement.rs
261:        let request = RequestInit {

components/script/dom/htmlmediaelement.rs
542:            let request = RequestInit {

components/script/dom/htmlscriptelement.rs
240:    let request = RequestInit {

components/script/dom/serviceworkerglobalscope.rs
161:            let request = RequestInit {

components/script/dom/workerglobalscope.rs
206:            let request = NetRequestInit {

components/script/dom/xmlhttprequest.rs
582:        let mut request = RequestInit {

components/script/fetch.rs
44:    NetTraitsRequestInit {

components/script/layout_image.rs
71:    let request = FetchRequestInit {

components/script/script_thread.rs
2125:        let request = RequestInit {

components/script/stylesheet_loader.rs
240:        let request = RequestInit {

Then there are some in tests/unit/net/http_loader.rs as well, but those shouldn't require as much care.

fnune · 2017-04-20T11:24:52Z

So in summary I should change RequestInit to take only an Origin in the origin field, and then, in the uses that already there, instantiate an Origin from the URL each use case has implemented. ~~The problem is the following: in some cases, that request may have been created from the browser console, in which case its origin field should be the string "null".~~ If I instantiate an Origin from the URLs that are currently there, the case where the origin has to be "null" will be omitted.

Edit: I was wrong about that, read the following comments.

jdm · 2017-04-20T22:20:18Z

I'm not sure I understand the connection to the browser console here. If such a request occurred, I would expect it to use the origin of the page associated with the console, not null.

fnune · 2017-04-20T22:58:24Z

I'm sorry, I was wrong about the browser console, and you're right: if I open my console right now and go fetch('http://somewebsite.com'), the request's origin will be set to https://github.com.

What I was trying to explain was the possibility of the request origin being "null". It can happen in (AFAIK) two situations:

The request was not done from a hosted website but from a file in the user's computer.
If a cross-origin resource redirects to another resource at a new origin, (the value of origin should be set to "null" after redirecting).

Currently, all the uses of RequestInit neglect this possibility.

jdm · 2017-04-20T23:19:16Z

The redirection happens as part of the fetch implementation, so the value for RequestInit is not affected by that. file:// URLs are the easiest example of null origins, but it could also happen from blob:// URLs too I believe. If we encapsulate the global origin within the origin() method, we can fix those for free later on.

fnune · 2017-04-21T11:22:02Z

Seems good, I'll be working on it tonight. Thanks for your help!

fnune · 2017-04-21T18:07:27Z

Should I use the origin method from the document (this line), or should I implement a new origin method for GlobalScope ?

I suppose in the case of implementing a new method for GlobalScope, I would actually return the document's origin, something like this: window.Document().origin

...which makes it sort of futile to implement it, because I could just get the document and use its origin. What do you think?

asajeffrey · 2017-07-16T16:42:58Z

Good point, it looks like what's happening is that the same-origin check is failing because the request origin is an opaque origin, rather than the origin that originated the request. Tracking this down, it looks like the culprit is https://github.com/brainlessdeveloper/servo/blob/5dc8fc99a89478a4827677fc256664b689a024fa/components/script/script_thread.rs#L2289:

    fn pre_page_load(&self, incomplete: InProgressLoad, load_data: LoadData) {
        let id = incomplete.pipeline_id.clone();
        let mut req_init = RequestInit {
            url: load_data.url.clone(),
            method: load_data.method,
            destination: Destination::Document,
            credentials_mode: CredentialsMode::Include,
            use_url_credentials: true,
            pipeline_id: Some(id),
            referrer_url: load_data.referrer_url,
            referrer_policy: load_data.referrer_policy,
            headers: load_data.headers,
            body: load_data.data,
            redirect_mode: RedirectMode::Manual,
            // TODO: Add a proper origin to pre_page_load
            // https://github.com/servo/servo/issues/17238
            .. RequestInit::default()
        };
    ...

Adding:

            origin: incomplete.origin.immutable().clone(),

Seems to do the trick.

@brainlessdeveloper: do you want to make this change and we'll see if this fixes everything?

asajeffrey · 2017-07-16T17:46:15Z

I ran the wpt test suite locally. Results at https://gist.github.com/asajeffrey/fec16fc305de171b758eafe42c111333.

Eyeballing it, that looks like quite a few PASS [expected FAIL], the usual intermittents, and the issue discussed above in #16508 (comment).

So this is all looking good with that one-line change. (Usual story: days of debugging to find one line to change, sigh.)

fnune · 2017-07-16T18:38:31Z

@asajeffrey on it right now! :)

asajeffrey · 2017-07-16T18:55:32Z

Thanks!

fnune · 2017-07-16T19:10:33Z

So, all the PASS, expected FAIL tests I see are in the quirks mode module (presumably nothing to do with this) and then the intermittent tests we dug up some days ago. The only new thing I see is this:

  ▶ Unexpected subtest result in /cors/allow-headers.htm:
  └ PASS [expected FAIL] Allow origin: _http://web-platform.test:8000___[tab]_

I guess I should change the expectation for this one?

fnune · 2017-07-16T19:57:05Z

Ok, the cors tests pass with no unexpected results, as well as referrer-policy and fetch/api (in this last one, excluding the intermittent ones ofc). So I think we should be good to go, hopefully.

asajeffrey · 2017-07-16T20:17:19Z

@bors-servo: try

bors-servo · 2017-07-16T20:17:23Z

⌛ Trying commit 6032940 with merge 006e36e...

Properly set origin of fetch requests  These changes aim to fix #15247 ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [x] These changes fix #15247 (github issue number if applicable).  - [ ] There are tests for these changes - [x] These changes do not require tests because cors is already tested with different origins These changes require changes in tests, but I need help with that (see comments below).    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/16508)

bors-servo · 2017-07-16T21:22:13Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel-css, linux-rel-wpt, mac-dev-unit, mac-rel-css1, mac-rel-css2, mac-rel-wpt1, mac-rel-wpt2, mac-rel-wpt3, mac-rel-wpt4, windows-msvc-dev
State: approved= try=True

fnune · 2017-07-16T21:23:25Z

Finally! :D

asajeffrey · 2017-07-16T21:45:50Z

I'll have a last look at it when I'm in front of a computer, I'm not expecting any issues, we should be good to merge! Thanks for your patience.

fnune · 2017-07-16T21:47:41Z

Thanks for your patience too, especially considering the last failures were caused by things I typed, and I didn't manage to debug it 🙈

asajeffrey · 2017-07-17T15:29:20Z

OK, we are good to go! @bors-servo r+

bors-servo · 2017-07-17T15:29:21Z

📌 Commit 6032940 has been approved by asajeffrey

bors-servo · 2017-07-17T15:29:25Z

⌛ Testing commit 6032940 with merge 2bb4f65...

Properly set origin of fetch requests  These changes aim to fix #15247 ---  - [x] `./mach build -d` does not report any errors - [x] `./mach test-tidy` does not report any errors - [x] These changes fix #15247 (github issue number if applicable).  - [ ] There are tests for these changes - [x] These changes do not require tests because cors is already tested with different origins These changes require changes in tests, but I need help with that (see comments below).    --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/16508)

bors-servo · 2017-07-17T16:17:53Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel-css, linux-rel-wpt, mac-dev-unit, mac-rel-css1, mac-rel-css2, mac-rel-wpt1, mac-rel-wpt2, mac-rel-wpt3, mac-rel-wpt4, windows-msvc-dev
Approved by: asajeffrey
Pushing 2bb4f65 to master...

asajeffrey · 2017-07-17T16:41:22Z

Woo hoo! Just shy of the three month anniversary. Thanks @brainlessdeveloper!

fnune · 2017-07-17T17:11:57Z

Thank you for your help! :)

KiChjang · 2017-07-17T22:07:33Z

I feel a bit guilty about the last one here because I've mentioned it in the linked issue here but I did not speak up about the probability of the root cause being that. Sorry.

asajeffrey · 2017-07-17T22:54:28Z

@KiChjang np, it was a good exercise in figuring out how our fetch code works!

highfive assigned metajack Apr 18, 2017

highfive added the S-awaiting-review There is new code that needs to be reviewed. label Apr 18, 2017

highfive added S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. and removed S-awaiting-review There is new code that needs to be reviewed. labels Apr 18, 2017

fnune changed the title ~~Fetch set origin~~ Properly set origin of fetch requests Apr 18, 2017

highfive added S-tests-failed The changes caused existing tests to fail. and removed S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. labels Apr 18, 2017

KiChjang changed the title ~~Properly set origin of fetch requests~~ [WIP] Properly set origin of fetch requests Apr 18, 2017

KiChjang reviewed Apr 18, 2017

View reviewed changes

Change RequestInit origin type to ImmutableOrigin

6032940

fnune force-pushed the fetch-set-origin branch from 5dc8fc9 to 6032940 Compare July 16, 2017 19:44

highfive added S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. and removed S-awaiting-review There is new code that needs to be reviewed. labels Jul 17, 2017

bors-servo merged commit 6032940 into servo:master Jul 17, 2017

highfive removed the S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. label Jul 17, 2017

bors-servo mentioned this pull request Jul 17, 2017

Set navigation start value according to navigation timing spec #17735

Merged

3 tasks

fnune deleted the fetch-set-origin branch July 17, 2017 17:49

Properly set origin of fetch requests #16508

Properly set origin of fetch requests #16508

Conversation

fnune commented Apr 18, 2017 • edited Loading

highfive commented Apr 18, 2017

highfive commented Apr 18, 2017

highfive commented Apr 18, 2017

metajack commented Apr 18, 2017

bors-servo commented Apr 18, 2017

bors-servo commented Apr 18, 2017

fnune commented Apr 18, 2017

metajack commented Apr 18, 2017

metajack commented Apr 18, 2017

fnune commented Apr 18, 2017

bors-servo commented Apr 18, 2017

metajack commented Apr 18, 2017

KiChjang Apr 18, 2017 • edited Loading

Choose a reason for hiding this comment

fnune Apr 18, 2017

Choose a reason for hiding this comment

fnune commented Apr 19, 2017 • edited Loading

jdm commented Apr 20, 2017 • edited Loading

fnune commented Apr 20, 2017 • edited Loading

jdm commented Apr 20, 2017

fnune commented Apr 20, 2017 • edited Loading

jdm commented Apr 20, 2017

fnune commented Apr 21, 2017

fnune commented Apr 21, 2017 • edited Loading

asajeffrey commented Jul 16, 2017

asajeffrey commented Jul 16, 2017

fnune commented Jul 16, 2017

asajeffrey commented Jul 16, 2017

fnune commented Jul 16, 2017

fnune commented Jul 16, 2017

asajeffrey commented Jul 16, 2017

bors-servo commented Jul 16, 2017

bors-servo commented Jul 16, 2017

fnune commented Jul 16, 2017

asajeffrey commented Jul 16, 2017

fnune commented Jul 16, 2017

asajeffrey commented Jul 17, 2017

bors-servo commented Jul 17, 2017

bors-servo commented Jul 17, 2017

bors-servo commented Jul 17, 2017

asajeffrey commented Jul 17, 2017

fnune commented Jul 17, 2017

KiChjang commented Jul 17, 2017

asajeffrey commented Jul 17, 2017

fnune commented Apr 18, 2017 •

edited

Loading

KiChjang Apr 18, 2017 •

edited

Loading

fnune commented Apr 19, 2017 •

edited

Loading

jdm commented Apr 20, 2017 •

edited

Loading

fnune commented Apr 20, 2017 •

edited

Loading

fnune commented Apr 20, 2017 •

edited

Loading

fnune commented Apr 21, 2017 •

edited

Loading