Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update referrer computation #27104

Merged
merged 3 commits into from Jul 3, 2020

Conversation

utsavoza
Copy link
Contributor

@utsavoza utsavoza commented Jun 28, 2020

The PR updates the request's referrer computation in consideration with the recent changes in w3c/webappsec-referrer-policy#135.


@highfive
Copy link

highfive commented Jun 28, 2020

Heads up! This PR modifies the following files:

  • @asajeffrey: components/script/dom/globalscope.rs
  • @KiChjang: components/script/dom/globalscope.rs, components/net/fetch/methods.rs, components/net/tests/http_loader.rs, components/net/http_loader.rs

@jdm
Copy link
Member

jdm commented Jun 28, 2020

@bors-servo try=wpt

@bors-servo
Copy link
Contributor

bors-servo commented Jun 28, 2020

Trying commit 627e88e with merge 4fb586e...

bors-servo added a commit that referenced this issue Jun 28, 2020
Update referrer computation

The PR updates the request's referrer computation in consideration with the recent changes in [w3c/webappsec-referrer-policy#135](w3c/webappsec-referrer-policy#135).

---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #27030
- [x] There are tests for these changes
@bors-servo
Copy link
Contributor

bors-servo commented Jun 28, 2020

💔 Test failed - status-taskcluster

Copy link
Member

@gterzian gterzian left a comment

Looks great, just one question and some docs that can be added.

@@ -242,56 +239,67 @@ fn is_schemelessy_same_site(site_a: &ImmutableOrigin, site_b: &ImmutableOrigin)
}

/// <https://w3c.github.io/webappsec-referrer-policy/#strip-url>
fn strip_url(mut referrer_url: ServoUrl, origin_only: bool) -> Option<ServoUrl> {
fn strip_url_for_use_as_referrer(mut url: ServoUrl, origin_only: bool) -> Option<ServoUrl> {
const MAX_REFERRER_URL_LENGTH: usize = 4096;
Copy link
Member

@gterzian gterzian Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering why url always seems to be non-null, where are we dealing with Step 1?

Copy link
Contributor Author

@utsavoza utsavoza Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, it doesn't seem clear to me from the spec when a referrerSource can be null

Copy link
Member

@gterzian gterzian Jun 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks for checking, I have filed w3c/webappsec-referrer-policy#139

Copy link
Contributor Author

@utsavoza utsavoza Jul 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks like step 1 here is redundant.

url.set_username("").unwrap();
url.set_password(None).unwrap();
url.set_fragment(None);
if origin_only || url.as_str().len() > MAX_REFERRER_URL_LENGTH {
Copy link
Member

@gterzian gterzian Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a note that the length check is Step 6 of https://w3c.github.io/webappsec-referrer-policy/#determine-requests-referrer.

@@ -97,6 +97,11 @@ impl ServoUrl {
scheme == "https" || scheme == "wss"
}

pub fn is_local_scheme(&self) -> bool {
Copy link
Member

@gterzian gterzian Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the spec link \\\ <https://fetch.spec.whatwg.org/#local-scheme>

@@ -169,6 +174,20 @@ impl ServoUrl {
Ok(Self::from_url(Url::from_file_path(path)?))
}

// https://w3c.github.io/webappsec-secure-contexts/#potentially-trustworthy-url
Copy link
Member

@gterzian gterzian Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I see this hasn't been done consistently elsewhere in this file, and I do think it's better to document this as \\\ <https://w3c.github.io/webappsec-secure-contexts/#potentially-trustworthy-url>

Copy link
Member

@gterzian gterzian left a comment

Ok, just wondering about one extra change in the new commit which might require a spec number.

Also please squash into one commit. Should be good to go after!

@@ -240,6 +240,7 @@ pub fn main_fetch(
let referrer_url = match mem::replace(&mut request.referrer, Referrer::NoReferrer) {
Referrer::NoReferrer => None,
Referrer::ReferrerUrl(referrer_source) | Referrer::Client(referrer_source) => {
request.headers.remove(header::REFERER);
Copy link
Member

@gterzian gterzian Jun 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this change intended? Could you please add a spec number if possible?

Copy link
Contributor Author

@utsavoza utsavoza Jun 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that was an existing step I'd earlier omitted by mistake. The step doesn't correspond to the spec, but I believe it removes Referer headers that were set in past redirects or preflights.

Copy link
Member

@gterzian gterzian Jun 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

@utsavoza
Copy link
Contributor Author

utsavoza commented Jun 30, 2020

Squashed into one commit

@gterzian
Copy link
Member

gterzian commented Jun 30, 2020

Thanks!

Looking at the test expectations update, besides a lot of progress, it also seems to include some new FAIL. Do you know what is causing those?

They seem to relate to the "no-refferer" being too aggressive?

Is there perhaps a follow-up issue that could be filed for them?

It could also be that the tests are outdated.

@utsavoza
Copy link
Contributor Author

utsavoza commented Jul 1, 2020

Looking at the test expectations update, besides a lot of progress, it also seems to include some new FAIL. Do you know what is causing those?

I didn't get a chance to take a look at this earlier, but afaict the referrer policy isn't getting propagated properly during about:srcdoc page load, specifically with Metadata::default(url) step:

let mut meta = Metadata::default(url);
meta.set_content_type(Some(&mime::TEXT_HTML));

Maybe I should update the PR to include the change and see if it fixes the current failing tests.

Edit: I have updated the PR with the change, but we'll need to see if it breaks any other tests unexpectedly.

@utsavoza
Copy link
Contributor Author

utsavoza commented Jul 1, 2020

I think this PR should also close #14505

@gterzian
Copy link
Member

gterzian commented Jul 2, 2020

I think this PR should also close #14505

Awesome!

@gterzian
Copy link
Member

gterzian commented Jul 2, 2020

Thanks!

@bors-servo r+

@bors-servo
Copy link
Contributor

bors-servo commented Jul 2, 2020

Trying commit b9adf52 with merge b063dab...

bors-servo added a commit that referenced this issue Jul 2, 2020
Update referrer computation

The PR updates the request's referrer computation in consideration with the recent changes in [w3c/webappsec-referrer-policy#135](w3c/webappsec-referrer-policy#135).

---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #27030
- [x] There are tests for these changes
@bors-servo
Copy link
Contributor

bors-servo commented Jul 2, 2020

💔 Test failed - status-taskcluster

@CYBAI
Copy link
Member

CYBAI commented Jul 2, 2020

Looks like same failures again. Or maybe we can say this PR makes the test failure non-intermittent now 😆?

  â–¶ Unexpected subtest result in /webdriver/tests/get_current_url/get.py:
  │ ERROR [expected PASS] test_get_current_url_after_modified_location
  │   → teardown error
  │ 
  │ capabilities = {}
  │ configuration = {'capabilities': {}, 'host': '127.0.0.1', 'port': 57833}
  │ request = <SubRequest 'session' for <Function 'test_get_current_url_after_modified_location'>>
  │ 
  │     @pytest.fixture(scope="function")
  │     def session(capabilities, configuration, request):
  │         """Create and start a session for a test that does not itself test session creation.
  │     
  │         By default the session will stay open after each test, but we always try to start a
  │         new one and assume that if that fails there is already a valid session. This makes it
  │         possible to recover from some errors that might leave the session in a bad state, but
  │         does not demand that we start a new session per test."""
  │         global _current_session
  │     
  │         # Update configuration capabilities with custom ones from the
  │         # capabilities fixture, which can be set by tests
  │         caps = copy.deepcopy(configuration["capabilities"])
  │         caps.update(capabilities)
  │         caps = {"alwaysMatch": caps}
  │     
  │         # If there is a session with different capabilities active, end it now
  │         if _current_session is not None and (
  │                 caps != _current_session.requested_capabilities):
  │             _current_session.end()
  │             _current_session = None
  │     
  │         if _current_session is None:
  │             _current_session = webdriver.Session(
  │                 configuration["host"],
  │                 configuration["port"],
  │                 capabilities=caps)
  │         try:
  │             _current_session.start()
  │         except webdriver.error.SessionNotCreatedException:
  │             if not _current_session.session_id:
  │                 raise
  │     
  │         # Enforce a fixed default window size and position
  │         _current_session.window.size = defaults.WINDOW_SIZE
  │         _current_session.window.position = defaults.WINDOW_POSITION
  │     
  │         yield _current_session
  │     
  │ >       cleanup_session(_current_session)
  │ 
  │ capabilities = {}
  │ caps       = {'alwaysMatch': {}}
  │ configuration = {'capabilities': {}, 'host': '127.0.0.1', 'port': 57833}
  │ request    = <SubRequest 'session' for <Function 'test_get_current_url_after_modified_location'>>
  │ 
  │ tests/wpt/web-platform-tests/webdriver/tests/support/fixtures.py:164: 
  │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  │ tests/wpt/web-platform-tests/webdriver/tests/support/helpers.py:77: in cleanup_session
  │     _restore_timeouts(session)
  │ tests/wpt/web-platform-tests/webdriver/tests/support/helpers.py:15: in inner
  │     return f(*args, **kwargs)
  │ tests/wpt/web-platform-tests/webdriver/tests/support/helpers.py:49: in _restore_timeouts
  │     session.timeouts.implicit = defaults.IMPLICIT_WAIT_TIMEOUT
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:65: in implicit
  │     return self._set("implicit", secs)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:40: in _set
  │     self.session.send_session_command("POST", "timeouts", body)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:514: in send_session_command
  │     return self.send_command(method, url, body, timeout)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:469: in send_command
  │     session=self, timeout=timeout)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py:182: in send
  │     response = self._request(method, uri, payload, headers, timeout=None)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py:199: in _request
  │     self.connection.request(method, url, payload, headers)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1053: in request
  │     self._send_request(method, url, body, headers)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1093: in _send_request
  │     self.endheaders(body)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1049: in endheaders
  │     self._send_output(message_body)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:893: in _send_output
  │     self.send(msg)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:855: in send
  │     self.connect()
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:832: in connect
  │     self.timeout, self.source_address)
  │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  │ 
  │ address = ('127.0.0.1', 57833), timeout = <object object at 0x10ae43390>
  │ source_address = None
  │ 
  │     def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
  │                           source_address=None):
  │         """Connect to *address* and return the socket object.
  │     
  │         Convenience function.  Connect to *address* (a 2-tuple ``(host,
  │         port)``) and return the socket object.  Passing the optional
  │         *timeout* parameter will set the timeout on the socket instance
  │         before attempting to connect.  If no *timeout* is supplied, the
  │         global default timeout setting returned by :func:`getdefaulttimeout`
  │         is used.  If *source_address* is set it must be a tuple of (host, port)
  │         for the socket to bind as a source address before making the connection.
  │         An host of '' or port 0 tells the OS to use the default.
  │         """
  │     
  │         host, port = address
  │         err = None
  │         for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  │             af, socktype, proto, canonname, sa = res
  │             sock = None
  │             try:
  │                 sock = socket(af, socktype, proto)
  │                 if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
  │                     sock.settimeout(timeout)
  │                 if source_address:
  │                     sock.bind(source_address)
  │                 sock.connect(sa)
  │                 return sock
  │     
  │             except error as _:
  │                 err = _
  │                 if sock is not None:
  │                     sock.close()
  │     
  │         if err is not None:
  │ >           raise err
  │ E           error: [Errno 61] Connection refused
  │ 
  │ _          = error(61, 'Connection refused')
  │ address    = ('127.0.0.1', 57833)
  │ af         = 2
  │ canonname  = ''
  │ err        = error(61, 'Connection refused')
  │ host       = '127.0.0.1'
  │ port       = 57833
  │ proto      = 6
  │ res        = (2, 1, 6, '', ('127.0.0.1', 57833))
  │ sa         = ('127.0.0.1', 57833)
  │ sock       = <socket._socketobject object at 0x114373360>
  │ socktype   = 1
  │ source_address = None
  │ timeout    = <object object at 0x10ae43390>
  │ 
  â”” /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py:575: error

  â–¶ Unexpected subtest result in /webdriver/tests/get_current_url/get.py:
  │ FAIL [expected PASS] test_get_current_url_after_modified_location
  │   → error: [Errno 61] Connection refused
  │ 
  │ session = <Session 1ace95da-5f88-4b7c-8712-ac92fb009e60>
  │ 
  │     def test_get_current_url_after_modified_location(session):
  │         start = get_current_url(session)
  │         session.execute_script("window.location.href = 'about:blank#wd_test_modification'")
  │         Poll(session, message="URL did not change").until(
  │ >           lambda s: get_current_url(s).body["value"] != start.body["value"])
  │ 
  │ session    = <Session 1ace95da-5f88-4b7c-8712-ac92fb009e60>
  │ start      = <Responsetatus=200 body={"value": "file:///Users/worker/tasks/task_1593661525/repo/tests/wpt/web-platform-tests"}>
  │ 
  │ tests/wpt/web-platform-tests/webdriver/tests/get_current_url/get.py:79: 
  │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  │ tests/wpt/web-platform-tests/webdriver/tests/support/sync.py:117: in until
  │     rv = condition(self.session)
  │ tests/wpt/web-platform-tests/webdriver/tests/get_current_url/get.py:79: in <lambda>
  │     lambda s: get_current_url(s).body["value"] != start.body["value"])
  │ tests/wpt/web-platform-tests/webdriver/tests/get_current_url/get.py:16: in get_current_url
  │     "GET", "session/{session_id}/url".format(**vars(session)))
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py:182: in send
  │     response = self._request(method, uri, payload, headers, timeout=None)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py:199: in _request
  │     self.connection.request(method, url, payload, headers)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1053: in request
  │     self._send_request(method, url, body, headers)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1093: in _send_request
  │     self.endheaders(body)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1049: in endheaders
  │     self._send_output(message_body)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:893: in _send_output
  │     self.send(msg)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:855: in send
  │     self.connect()
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:832: in connect
  │     self.timeout, self.source_address)
  │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  │ 
  │ address = ('127.0.0.1', 57833), timeout = <object object at 0x10ae43390>
  │ source_address = None
  │ 
  │     def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
  │                           source_address=None):
  │         """Connect to *address* and return the socket object.
  │     
  │         Convenience function.  Connect to *address* (a 2-tuple ``(host,
  │         port)``) and return the socket object.  Passing the optional
  │         *timeout* parameter will set the timeout on the socket instance
  │         before attempting to connect.  If no *timeout* is supplied, the
  │         global default timeout setting returned by :func:`getdefaulttimeout`
  │         is used.  If *source_address* is set it must be a tuple of (host, port)
  │         for the socket to bind as a source address before making the connection.
  │         An host of '' or port 0 tells the OS to use the default.
  │         """
  │     
  │         host, port = address
  │         err = None
  │         for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  │             af, socktype, proto, canonname, sa = res
  │             sock = None
  │             try:
  │                 sock = socket(af, socktype, proto)
  │                 if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
  │                     sock.settimeout(timeout)
  │                 if source_address:
  │                     sock.bind(source_address)
  │                 sock.connect(sa)
  │                 return sock
  │     
  │             except error as _:
  │                 err = _
  │                 if sock is not None:
  │                     sock.close()
  │     
  │         if err is not None:
  │ >           raise err
  │ E           error: [Errno 61] Connection refused
  │ 
  │ _          = error(61, 'Connection refused')
  │ address    = ('127.0.0.1', 57833)
  │ af         = 2
  │ canonname  = ''
  │ err        = error(61, 'Connection refused')
  │ host       = '127.0.0.1'
  │ port       = 57833
  │ proto      = 6
  │ res        = (2, 1, 6, '', ('127.0.0.1', 57833))
  │ sa         = ('127.0.0.1', 57833)
  │ sock       = <socket._socketobject object at 0x113fcae50>
  │ socktype   = 1
  │ source_address = None
  │ timeout    = <object object at 0x10ae43390>
  │ 
  â”” /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py:575: error

  â–¶ Unexpected subtest result in /webdriver/tests/get_current_url/get.py:
  │ ERROR [expected PASS] test_get_current_url_nested_browsing_contexts
  │   → setup error
  │ 
  │ capabilities = {}
  │ configuration = {'capabilities': {}, 'host': '127.0.0.1', 'port': 57833}
  │ request = <SubRequest 'session' for <Function 'test_get_current_url_nested_browsing_contexts'>>
  │ 
  │     @pytest.fixture(scope="function")
  │     def session(capabilities, configuration, request):
  │         """Create and start a session for a test that does not itself test session creation.
  │     
  │         By default the session will stay open after each test, but we always try to start a
  │         new one and assume that if that fails there is already a valid session. This makes it
  │         possible to recover from some errors that might leave the session in a bad state, but
  │         does not demand that we start a new session per test."""
  │         global _current_session
  │     
  │         # Update configuration capabilities with custom ones from the
  │         # capabilities fixture, which can be set by tests
  │         caps = copy.deepcopy(configuration["capabilities"])
  │         caps.update(capabilities)
  │         caps = {"alwaysMatch": caps}
  │     
  │         # If there is a session with different capabilities active, end it now
  │         if _current_session is not None and (
  │                 caps != _current_session.requested_capabilities):
  │             _current_session.end()
  │             _current_session = None
  │     
  │         if _current_session is None:
  │             _current_session = webdriver.Session(
  │                 configuration["host"],
  │                 configuration["port"],
  │                 capabilities=caps)
  │         try:
  │             _current_session.start()
  │         except webdriver.error.SessionNotCreatedException:
  │             if not _current_session.session_id:
  │                 raise
  │     
  │         # Enforce a fixed default window size and position
  │ >       _current_session.window.size = defaults.WINDOW_SIZE
  │ 
  │ capabilities = {}
  │ caps       = {'alwaysMatch': {}}
  │ configuration = {'capabilities': {}, 'host': '127.0.0.1', 'port': 57833}
  │ request    = <SubRequest 'session' for <Function 'test_get_current_url_nested_browsing_contexts'>>
  │ 
  │ tests/wpt/web-platform-tests/webdriver/tests/support/fixtures.py:159: 
  │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:19: in inner
  │     return func(self, *args, **kwargs)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:263: in size
  │     self.session.send_session_command("POST", "window/rect", body)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:514: in send_session_command
  │     return self.send_command(method, url, body, timeout)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:469: in send_command
  │     session=self, timeout=timeout)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py:182: in send
  │     response = self._request(method, uri, payload, headers, timeout=None)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py:199: in _request
  │     self.connection.request(method, url, payload, headers)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1053: in request
  │     self._send_request(method, url, body, headers)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1093: in _send_request
  │     self.endheaders(body)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1049: in endheaders
  │     self._send_output(message_body)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:893: in _send_output
  │     self.send(msg)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:855: in send
  │     self.connect()
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:832: in connect
  │     self.timeout, self.source_address)
  │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  │ 
  │ address = ('127.0.0.1', 57833), timeout = <object object at 0x10ae43390>
  │ source_address = None
  │ 
  │     def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
  │                           source_address=None):
  │         """Connect to *address* and return the socket object.
  │     
  │         Convenience function.  Connect to *address* (a 2-tuple ``(host,
  │         port)``) and return the socket object.  Passing the optional
  │         *timeout* parameter will set the timeout on the socket instance
  │         before attempting to connect.  If no *timeout* is supplied, the
  │         global default timeout setting returned by :func:`getdefaulttimeout`
  │         is used.  If *source_address* is set it must be a tuple of (host, port)
  │         for the socket to bind as a source address before making the connection.
  │         An host of '' or port 0 tells the OS to use the default.
  │         """
  │     
  │         host, port = address
  │         err = None
  │         for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  │             af, socktype, proto, canonname, sa = res
  │             sock = None
  │             try:
  │                 sock = socket(af, socktype, proto)
  │                 if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
  │                     sock.settimeout(timeout)
  │                 if source_address:
  │                     sock.bind(source_address)
  │                 sock.connect(sa)
  │                 return sock
  │     
  │             except error as _:
  │                 err = _
  │                 if sock is not None:
  │                     sock.close()
  │     
  │         if err is not None:
  │ >           raise err
  │ E           error: [Errno 61] Connection refused
  │ 
  │ _          = error(61, 'Connection refused')
  │ address    = ('127.0.0.1', 57833)
  │ af         = 2
  │ canonname  = ''
  │ err        = error(61, 'Connection refused')
  │ host       = '127.0.0.1'
  │ port       = 57833
  │ proto      = 6
  │ res        = (2, 1, 6, '', ('127.0.0.1', 57833))
  │ sa         = ('127.0.0.1', 57833)
  │ sock       = <socket._socketobject object at 0x1142564b0>
  │ socktype   = 1
  │ source_address = None
  │ timeout    = <object object at 0x10ae43390>
  │ 
  â”” /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py:575: error

  â–¶ Unexpected subtest result in /webdriver/tests/get_current_url/get.py:
  │ ERROR [expected PASS] test_get_current_url_nested_browsing_context
  │   → setup error
  │ 
  │ capabilities = {}
  │ configuration = {'capabilities': {}, 'host': '127.0.0.1', 'port': 57833}
  │ request = <SubRequest 'session' for <Function 'test_get_current_url_nested_browsing_context'>>
  │ 
  │     @pytest.fixture(scope="function")
  │     def session(capabilities, configuration, request):
  │         """Create and start a session for a test that does not itself test session creation.
  │     
  │         By default the session will stay open after each test, but we always try to start a
  │         new one and assume that if that fails there is already a valid session. This makes it
  │         possible to recover from some errors that might leave the session in a bad state, but
  │         does not demand that we start a new session per test."""
  │         global _current_session
  │     
  │         # Update configuration capabilities with custom ones from the
  │         # capabilities fixture, which can be set by tests
  │         caps = copy.deepcopy(configuration["capabilities"])
  │         caps.update(capabilities)
  │         caps = {"alwaysMatch": caps}
  │     
  │         # If there is a session with different capabilities active, end it now
  │         if _current_session is not None and (
  │                 caps != _current_session.requested_capabilities):
  │             _current_session.end()
  │             _current_session = None
  │     
  │         if _current_session is None:
  │             _current_session = webdriver.Session(
  │                 configuration["host"],
  │                 configuration["port"],
  │                 capabilities=caps)
  │         try:
  │             _current_session.start()
  │         except webdriver.error.SessionNotCreatedException:
  │             if not _current_session.session_id:
  │                 raise
  │     
  │         # Enforce a fixed default window size and position
  │ >       _current_session.window.size = defaults.WINDOW_SIZE
  │ 
  │ capabilities = {}
  │ caps       = {'alwaysMatch': {}}
  │ configuration = {'capabilities': {}, 'host': '127.0.0.1', 'port': 57833}
  │ request    = <SubRequest 'session' for <Function 'test_get_current_url_nested_browsing_context'>>
  │ 
  │ tests/wpt/web-platform-tests/webdriver/tests/support/fixtures.py:159: 
  │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:19: in inner
  │     return func(self, *args, **kwargs)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:263: in size
  │     self.session.send_session_command("POST", "window/rect", body)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:514: in send_session_command
  │     return self.send_command(method, url, body, timeout)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/client.py:469: in send_command
  │     session=self, timeout=timeout)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py:182: in send
  │     response = self._request(method, uri, payload, headers, timeout=None)
  │ tests/wpt/web-platform-tests/tools/webdriver/webdriver/transport.py:199: in _request
  │     self.connection.request(method, url, payload, headers)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1053: in request
  │     self._send_request(method, url, body, headers)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1093: in _send_request
  │     self.endheaders(body)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:1049: in endheaders
  │     self._send_output(message_body)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:893: in _send_output
  │     self.send(msg)
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:855: in send
  │     self.connect()
  │ /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py:832: in connect
  │     self.timeout, self.source_address)
  │ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  │ 
  │ address = ('127.0.0.1', 57833), timeout = <object object at 0x10ae43390>
  │ source_address = None
  │ 
  │     def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
  │                           source_address=None):
  │         """Connect to *address* and return the socket object.
  │     
  │         Convenience function.  Connect to *address* (a 2-tuple ``(host,
  │         port)``) and return the socket object.  Passing the optional
  │         *timeout* parameter will set the timeout on the socket instance
  │         before attempting to connect.  If no *timeout* is supplied, the
  │         global default timeout setting returned by :func:`getdefaulttimeout`
  │         is used.  If *source_address* is set it must be a tuple of (host, port)
  │         for the socket to bind as a source address before making the connection.
  │         An host of '' or port 0 tells the OS to use the default.
  │         """
  │     
  │         host, port = address
  │         err = None
  │         for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  │             af, socktype, proto, canonname, sa = res
  │             sock = None
  │             try:
  │                 sock = socket(af, socktype, proto)
  │                 if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
  │                     sock.settimeout(timeout)
  │                 if source_address:
  │                     sock.bind(source_address)
  │                 sock.connect(sa)
  │                 return sock
  │     
  │             except error as _:
  │                 err = _
  │                 if sock is not None:
  │                     sock.close()
  │     
  │         if err is not None:
  │ >           raise err
  │ E           error: [Errno 61] Connection refused
  │ 
  │ _          = error(61, 'Connection refused')
  │ address    = ('127.0.0.1', 57833)
  │ af         = 2
  │ canonname  = ''
  │ err        = error(61, 'Connection refused')
  │ host       = '127.0.0.1'
  │ port       = 57833
  │ proto      = 6
  │ res        = (2, 1, 6, '', ('127.0.0.1', 57833))
  │ sa         = ('127.0.0.1', 57833)
  │ sock       = <socket._socketobject object at 0x114342fa0>
  │ socktype   = 1
  │ source_address = None
  │ timeout    = <object object at 0x10ae43390>
  │ 
  â”” /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py:575: error

utsavoza added 2 commits Jul 2, 2020
Update unit tests for determine_requests_referrer

Update wpt metadata

Add missing spec links
@utsavoza
Copy link
Contributor Author

utsavoza commented Jul 2, 2020

Based on the panic trace I observed while running the test locally:

0:08.57 INFO STDOUT: ============================================================================================= test session starts ==============================================================================================
 0:08.57 INFO STDOUT: platform darwin -- Python 2.7.16, pytest-unknown, py-1.5.2, pluggy-0.5.3.dev -- /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
 0:08.57 INFO STDOUT: rootdir: /Users/utsavoza/dev/workspace/servo/servo, inifile:
 0:08.57 INFO STDOUT: collecting ...
 0:08.60 INFO STDOUT: collected 2 items
 0:08.60 INFO STDOUT: tests/wpt/web-platform-tests/webdriver/tests/get_current_url/get.py::test_get_current_url_file_protocol
 0:09.81 pid:46667 Full command: /Users/utsavoza/dev/workspace/servo/servo/target/release/servo --webdriver=59533 --hard-fail --headless
pid:46667 [2020-07-02T19:18:07Z ERROR net::http_loader] url: https://servo.org/
 0:09.81 pid:46667 [2020-07-02T19:18:07Z ERROR net::http_loader] url: https://servo.org/
 0:09.81 pid:46667 [2020-07-02T19:18:07Z ERROR net::http_loader] url: https://servo.org/
 0:10.86 INFO STDOUT: PASSED
 0:11.37 INFO STDOUT: tests/wpt/web-platform-tests/webdriver/tests/get_current_url/get.py::test_get_current_url_after_modified_location
 0:12.41 pid:46667 [2020-07-02T19:18:09Z ERROR net::http_loader] url: file:///Users/utsavoza/dev/workspace/servo/servo/tests/wpt/web-platform-tests
 0:12.41 pid:46667 called `Result::unwrap()` on an `Err` value: () (thread <unnamed>, at components/net/http_loader.rs:252)
 0:12.43 pid:46667    0: <servo::backtrace::Print as core::fmt::Debug>::fmt
 0:12.43 pid:46667    1: core::fmt::write
 0:12.43 pid:46667    2: std::io::Write::write_fmt
 0:12.43 pid:46667    3: servo::backtrace::print
 0:12.43 pid:46667    4: servo::main::{{closure}}
 0:12.43 pid:46667    5: std::panicking::rust_panic_with_hook
 0:12.43 pid:46667    6: _rust_begin_unwind
 0:12.43 pid:46667    7: core::panicking::panic_fmt
 0:12.43 pid:46667    8: core::result::unwrap_failed
 0:12.43 pid:46667    9: net::http_loader::strip_url_for_use_as_referrer
 0:12.43 pid:46667   10: net::http_loader::determine_requests_referrer
 0:12.43 pid:46667   11: net::fetch::methods::main_fetch
 0:12.44 pid:46667   12: net::fetch::methods::fetch_with_cors_cache
 0:12.44 pid:46667   13: net::fetch::methods::fetch
 0:12.44 pid:46667   14: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
 0:12.44 pid:46667   15: <rayon_core::job::HeapJob<BODY> as rayon_core::job::Job>::execute
 0:12.44 pid:46667   16: rayon_core::registry::WorkerThread::wait_until_cold
 0:12.44 pid:46667   17: rayon_core::registry::ThreadBuilder::run
 0:12.44 pid:46667   18: std::sys_common::backtrace::__rust_begin_short_backtrace
 0:12.44 pid:46667   19: core::ops::function::FnOnce::call_once{{vtable.shim}}
 0:12.44 pid:46667   20: std::sys::unix::thread::Thread::new::thread_start
 0:12.44 pid:46667   21: __pthread_start
 0:12.44 pid:46667 [2020-07-02T19:18:09Z ERROR servo] called `Result::unwrap()` on an `Err` value: ()
 0:12.44 pid:46667 Rayon: detected unexpected panic; aborting
 0:12.44 pid:46667 Redirecting call to abort() to mozalloc_abort
 0:12.44 pid:46667
 0:12.44 pid:46667 Stack trace
 0:12.44 pid:46667    0: <servo::backtrace::Print as core::fmt::Debug>::fmt
 0:12.44 pid:46667    1: core::fmt::write
 0:12.44 pid:46667    2: std::io::Write::write_fmt
 0:12.44 pid:46667    3: servo::backtrace::print
 0:12.44 pid:46667    4: servo::install_crash_handler::handler
 0:12.44 pid:46667    5: __sigtramp
 0:12.44 pid:46667    6: _mozalloc_abort.cold.1
 0:12.44 pid:46667    7: _mozalloc_abort
 0:12.44 pid:46667    8: _abort
 0:12.44 pid:46667    9: std::sys::unix::abort_internal
 0:12.44 pid:46667   10: std::process::abort
 0:12.44 pid:46667   11: rayon_core::registry::Registry::handle_panic
 0:12.44 pid:46667   12: <rayon_core::job::HeapJob<BODY> as rayon_core::job::Job>::execute
 0:12.44 pid:46667   13: rayon_core::registry::WorkerThread::wait_until_cold
 0:12.44 pid:46667   14: rayon_core::registry::ThreadBuilder::run
 0:12.44 pid:46667   15: std::sys_common::backtrace::__rust_begin_short_backtrace
 0:12.44 pid:46667   16: core::ops::function::FnOnce::call_once{{vtable.shim}}
 0:12.44 pid:46667   17: std::sys::unix::thread::Thread::new::thread_start
 0:12.44 pid:46667   18: __pthread_start
 0:12.60 INFO STDOUT: FAILED
 0:12.67 INFO STDOUT: tests/wpt/web-platform-tests/webdriver/tests/get_current_url/get.py::test_get_current_url_after_modified_location
 0:12.67 INFO STDOUT: ERROR

it looks like strip_url_for_use_as_referrer implementation panics at:

url.set_username("").unwrap();
url.set_password(None).unwrap();

because the underlying implementation (for set_username and set_password) returns an error for urls with file as their scheme.

Earlier we skipped the algorithm for all urls that didn't have http or https scheme which is why we didn't see any similar errors/panics.

I am not sure about the correct behavior in this case (that aligns with the spec), however returning None in the case
of error:

url.set_username("").ok()?;
url.set_password(None).ok()?;

or ignoring the returned result might resolve the test failure.

let _ = url.set_username("");
let _ = url.set_password(None);

What do you think? @gterzian @CYBAI @jdm

@CYBAI
Copy link
Member

CYBAI commented Jul 3, 2020

because the underlying implementation (for set_username and set_password) returns an error for urls with file as their scheme.

I just checked the spec of Strip url for use as a referrer and it says "If url’s scheme is a local scheme, then return no referrer." in step 2.

I'm not pretty familiar with referrer policy though, file scheme sounds pretty similar to local scheme 🤔 is it possible to be a missing piece in the step 2 in spec?

WDYT? @gterzian @jdm

(btw, might be an off-topic question though, why doesn't local scheme defined in fetch spec contain file scheme 🤔? what does the local mean?)

@gterzian
Copy link
Member

gterzian commented Jul 3, 2020

@utsavoza Thanks for looking into this.

Since the purpose of " Strip url for use as a referrer " is to remove some information from the url, I'd say if the information cannot be present in the first place, for example password/username with file urls, then we should not unwrap on those errors, since the "stripping" is actually unnecessary and the information not present in the first place.

And the Url implementation in Rust in such a case "does nothing and returns an error" https://docs.rs/url/2.0.0/url/struct.Url.html#method.set_password

So instead of unwrapping, we should just assume success(since there is actually nothing to strip in the first place) and continue with the algorithm I think.

That is also what the spec does, for example at https://url.spec.whatwg.org/#dom-url-username

Where the setter returns early in case of https://url.spec.whatwg.org/#cannot-have-a-username-password-port

I'm not pretty familiar with referrer policy though, file scheme sounds pretty similar to local scheme 🤔 is it possible to be a missing piece in the step 2 in spec?

Local scheme is defined at https://fetch.spec.whatwg.org/#local-scheme

@CYBAI
Copy link
Member

CYBAI commented Jul 3, 2020

Where the setter returns early in case of https://url.spec.whatwg.org/#cannot-have-a-username-password-port

Thanks for pointing to this spec link! Didn't find this one. Does this mean, the step 2 should be "If url’s scheme is a local scheme or file, then return no referrer." ?

I'm not pretty familiar with referrer policy though, file scheme sounds pretty similar to local scheme 🤔 is it possible to be a missing piece in the step 2 in spec?

Local scheme is defined at https://fetch.spec.whatwg.org/#local-scheme

Sorry for being off-topic to this PR >_< I did find the spec link of local scheme but what I'm not clear is, what does the local mean? 🤔

@gterzian
Copy link
Member

gterzian commented Jul 3, 2020

Does this mean, the step 2 should be "If url’s scheme is a local scheme or file, then return no referrer."

I don't think it means we should always return no referrer in the file case. Instead, we should simply let step 3 and 4 basically do nothing, and continue with the other steps.

what does the local mean?

I think it just means "a scheme that is "about", "blob", or "data"".

@jdm
Copy link
Member

jdm commented Jul 3, 2020

@bors-servo try=wpt

@bors-servo
Copy link
Contributor

bors-servo commented Jul 3, 2020

Trying commit 1b9e84b with merge 0c09ec9...

bors-servo added a commit that referenced this issue Jul 3, 2020
Update referrer computation

The PR updates the request's referrer computation in consideration with the recent changes in [w3c/webappsec-referrer-policy#135](w3c/webappsec-referrer-policy#135).

---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix #27030 and fix #14505
- [x] There are tests for these changes
@bors-servo
Copy link
Contributor

bors-servo commented Jul 3, 2020

☀️ Test successful - status-taskcluster
State: approved= try=True

Copy link
Member

@gterzian gterzian left a comment

Thanks, awesome work!

@gterzian
Copy link
Member

gterzian commented Jul 3, 2020

@bors-servo r+

@bors-servo
Copy link
Contributor

bors-servo commented Jul 3, 2020

📌 Commit 1b9e84b has been approved by gterzian

@bors-servo
Copy link
Contributor

bors-servo commented Jul 3, 2020

Testing commit 1b9e84b with merge 8800452...

@utsavoza
Copy link
Contributor Author

utsavoza commented Jul 3, 2020

Thanks for your help! @gterzian @CYBAI @jdm

@bors-servo
Copy link
Contributor

bors-servo commented Jul 3, 2020

☀️ Test successful - status-taskcluster
Approved by: gterzian
Pushing 8800452 to master...

@bors-servo bors-servo merged commit 8800452 into servo:master Jul 3, 2020
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants