[MRG] Referrer policies in RefererMiddleware #2306

redapple · 2016-10-05T15:26:13Z

I've based the tests and implementation on https://www.w3.org/TR/referrer-policy/

By default, with this change, RefererMiddleware does not send a Referer header (referrer value):

when the source response is file:// or s3://,
nor from https:// response to an http:// request.

However, it does send a referrer value for any https:// to another https:// (as browsers do, "no-referrer-when-downgrade" being the default policy)
This needs discussion.

User can change the policy per-request, using a new referrer_policy meta key, with values from "no-referrer" / "no-referrer-when-downgrade" / "same-origin" / "origin" / "origin-when-cross-origin" / "unsafe-url".

~~Still missing:~~

Try to use urlparse_cached()
Make default policy customizable through a REFERER_DEFAULT_POLICY setting
Test custom policies
Docs updates
Add tests for policy given by response headers (Referrer-Policy) -- is this even used in practice by web servers?
Handle referrers during redirects

To handle redirects, I added a signal handler on request_scheduled. I did not find a way to test both redirect middleware and referer middleware (I did not search too much though).
Suggestions to do that are welcome.

codecov-io · 2016-10-05T15:37:09Z

Codecov Report

Merging #2306 into master will decrease coverage by -0.6%.
The diff coverage is 94.44%.

@@            Coverage Diff            @@
##           master    #2306     +/-   ##
=========================================
- Coverage   83.93%   83.34%   -0.6%     
=========================================
  Files         161      161             
  Lines        8920     9060    +140     
  Branches     1316     1290     -26     
=========================================
+ Hits         7487     7551     +64     
- Misses       1176     1240     +64     
- Partials      257      269     +12

Impacted Files	Coverage Δ
scrapy/utils/url.py	`100% <100%> (ø)`	✅
scrapy/settings/default_settings.py	`98.63% <100%> (ø)`	✅
scrapy/spidermiddlewares/referer.py	`93% <93.98%> (+8.39%)`	✅
scrapy/core/downloader/handlers/s3.py	`62.9% <0%> (-32.26%)`	❌
scrapy/utils/boto.py	`46.66% <0%> (-26.67%)`	❌
scrapy/utils/gz.py	`85.29% <0%> (-14.71%)`	❌
scrapy/link.py	`86.36% <0%> (-13.64%)`	❌
scrapy/core/downloader/tls.py	`80% <0%> (-8.58%)`	❌
scrapy/_monkeypatches.py	`50% <0%> (-7.15%)`	❌
scrapy/utils/iterators.py	`95.45% <0%> (-4.55%)`	❌
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b49b9c...fad499a. Read the comment docs.

kmike · 2016-10-18T14:18:08Z

scrapy/spidermiddlewares/referer.py

+        if stripped is not None:
+            return urlunparse(stripped)
+
+    def strip_url(self, req_or_resp, origin_only=False):


What do you think about making it an utility function? It can be more reusable and easier to test this way (even doctests could be enough). I'd make it accept URL or ParseResult instead of scrapy Request or Response.

Good idea @kmike , I even had a similar use-case for https://github.com/umrashrf/scrapy/pull/1/files#diff-c1b844ba341115f81f50b33fd691fcbbR14

@kmike , are you thinking of an addition to w3lib.url?
Also, the current output is a ParseResult tuple, mainly for comparing origins.
Does it make sense to have the utility function serialize/urlunparse on demand, with some argument?
Or should we just compare origins as strings? (which is probably the cleanest as I don't really like returning different non-None types)

@kmike , I moved some of it to scrapy.utils.url. What do you think?

kmike · 2016-10-21T08:30:26Z

scrapy/utils/url.py

@@ -103,3 +103,35 @@ def guess_scheme(url):
        return any_to_uri(url)
    else:
        return add_http_if_no_scheme(url)
+
+
+def strip_url_credentials(url, origin_only=False, keep_fragments=False):


Could you please add a docstring to this function?

The function is named 'strip credentials', but it also strips port, I think this is not intuitive.

Right, it strips standard HTTP and HTTPS port numbers. This is mainly for comparing origins.
Docstring is needed indeed.
Does it make sense to have this feature as switchable through an argument?

Yeah, it should be either an argument or a separate function. If it becomes an argument then I think it is better to rename the function (not sure about the new name).

@kmike , code updated with new name for helper strip_url()

redapple · 2017-01-17T13:35:40Z

I'm adding 2 policies that appeared in the working draft:

redapple · 2017-01-18T13:58:19Z

Alright, @kmike , @eliasdorneles , @lopuhin , @dangra , I think this is ready for review.
Thanks in advance!

redapple · 2017-01-18T14:01:53Z

(I'm not sure why I use ftps in some tests. It's not supported by Scrapy)

kmike · 2017-01-18T14:03:29Z

docs/topics/spider-middleware.rst

+using ``file://`` or ``s3://`` scheme.
+
+.. warning::
+    By default, Scrapy's default referrer policy, just like `"no-referrer-when-downgrade"`_,


"By default" looks redundant

kmike · 2017-01-18T15:48:56Z

docs/topics/spider-middleware.rst

+.. warning::
+    Scrapy's default referrer policy, just like `"no-referrer-when-downgrade"`_,
+    will send a non-empty "Referer" header from any ``https://`` to any ``https://`` URL,
+    even if the domain is different.


If that's what browsers do by default then I think it makes sense to note that, to make a warning less scary.

kmike · 2017-01-18T16:57:04Z

docs/topics/spider-middleware.rst

+
+- a path to a ``scrapy.spidermiddlewares.referer.ReferrerPolicy`` subclass,
+  either a custom one or one of the built-in ones
+  (see ``scrapy.spidermiddlewares.referer``),


Is there a page in the docs we can reference to?

What kind of reference do you have in mind? built-in classes?

We could also define an interface for policies, so people know what to implement.

@kmike , an alternative is to use a table:

$ git diff diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 9010ae7..46a84d0 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -346,21 +346,21 @@ This setting accepts: - a path to a ``scrapy.spidermiddlewares.referer.ReferrerPolicy`` subclass, either a custom one or one of the built-in ones (see ``scrapy.spidermiddlewares.referer``), -- or one of the standard W3C-defined string values: - - - `"no-referrer"`_, - - `"no-referrer-when-downgrade"`_ - (the W3C-recommended default, used by major web browsers), - - `"same-origin"`_, - - `"origin"`_, - - `"strict-origin"`_, - - `"origin-when-cross-origin"`_, - - `"strict-origin-when-cross-origin"`_, - - or `"unsafe-url"`_ - (not recommended). - -It can also be the non-standard value ``"scrapy-default"`` to use -Scrapy's default referrer policy. +- or one of the standard W3C-defined string values + +======================================= ======================================================================== ======================================================= +String value Class name +======================================= ======================================================================== ======================================================= +`"no-referrer"`_ ``'scrapy.spidermiddlewares.referer.NoReferrerPolicy'`` +`"no-referrer-when-downgrade"`_ ``'scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy'`` the W3C-recommended default, used by major web browsers +`"same-origin"`_ ``'scrapy.spidermiddlewares.referer.SameOriginPolicy'`` +`"origin"`_ ``'scrapy.spidermiddlewares.referer.OriginPolicy'`` +`"strict-origin"`_ ``'scrapy.spidermiddlewares.referer.StrictOriginPolicy'`` +`"origin-when-cross-origin"`_ ``'scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy'`` +`"strict-origin-when-cross-origin"`_ ``'scrapy.spidermiddlewares.referer.StrictOriginWhenCrossOriginPolicy'`` +`"unsafe-url"`_ ``'scrapy.spidermiddlewares.referer.UnsafeUrlPolicy'`` NOT recommended +``"scrapy-default"`` ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'`` Scrapy's default policy (see below) +======================================= ======================================================================== ======================================================= Scrapy's default referrer policy is a variant of `"no-referrer-when-downgrade"`_, with the addition that "Referer" is not sent if the parent request was

I like such table.

Also, adding autodocs for these policy classes looks helpful - they already have docstrings, it is just a matter of putting several autoclass labels to scrapy rst docs.

as for "see scrapy.spidermiddlewares.referer", the question was how would user find this scrapy.spidermiddlewares.referer - there is no link or anything.

Documenting policy interface sounds like too much for me; I'm not sure this is something users may want to extend.

kmike · 2017-01-18T16:59:35Z

docs/topics/spider-middleware.rst

+
+Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'``
+
+.. reqmeta:: referrer_policy


It could make sense use the same spelling everywhere in Scrapy. Currently option name (REFERER_POLICY) has one 'r' while meta key name has 'rr'.

aha, I see, we already have REFERER_ENABLED

I'm ok with changing to REFERRER_POLICY. (It's a shame about that spelling mistake in the RFCs.)

We're loosing either way - now option names are inconsistent :)

I guess the way to go is to rename the other option and a middleware itself as well; of course, it can be done later.

kmike · 2017-01-18T17:00:45Z

docs/topics/spider-middleware.rst

+Scrapy's default referrer policy.
+
+Scrapy's default referrer policy is a variant of `"no-referrer-when-downgrade"`_,
+with the addition that "Referrer" is not sent if the parent request was


header name is Referer

kmike · 2017-01-18T17:01:59Z

docs/topics/spider-middleware.rst

+using ``file://`` or ``s3://`` scheme.
+
+.. warning::
+    Scrapy's default referrer policy—just like `"no-referrer-when-downgrade"`_,


a nitpick - there should be spaces before and after a dash :)

redapple · 2017-02-21T16:21:59Z

@kmike , I rebased, updated the docs with autoclass directives and renamed the setting to REFERRER_POLICY.

redapple · 2017-02-21T16:25:26Z

Another thing could be to move the policies to scrapy/extensions, similar to HTTP cache policies (with scrapy/scrapy/extensions/httpcache.py). What do you think?

kmike · 2017-02-22T17:30:25Z

scrapy/spidermiddlewares/referer.py

+                    try:
+                        self.default_policy = _policy_classes[policy.lower()]
+                    except:
+                        raise NotConfigured("Unknown referrer policy name %r" % policy)


I think it is better to raise an error and stop the crawl instead of silently disabling the middleware if a policy is unknown.

Good point!

kmike · 2017-02-22T17:31:47Z

scrapy/spidermiddlewares/referer.py

+            policy_name = to_native_str(
+                response.headers.get('Referrer-Policy', '').decode('latin1'))
+
+        cls = _policy_classes.get(policy_name.lower(), self.default_policy)


What do you think about warning for unknown policies? They can be typos.

Makes sense.

kmike · 2017-02-22T17:40:30Z

scrapy/spidermiddlewares/referer.py

+
+    NOREFERRER_SCHEMES = LOCAL_SCHEMES
+
+    def referrer(self, response, request):


It seems the API requires response and request (not response_url and request_url) only because of urlparse_cached - you wanted to avoid duplicate computations. It leads to some ugly code when a fake Response is created. Is this correct, or are there cases response and requestis indeed needed?

I wonder how much is the effect, given that urlparse has its own internal cache (using urls as keys, not response/requests).

Right @kmike , that was the reason.
I cannot think right now of a real need for handling request/responses directly. Meta information and HTTP headers ("Referrer-Policy") are to be interpreted at middleware level, and not at policy level I believe.

Indeed the fake response is quite ugly.
Let me clean it up with URLs all along.

redapple · 2017-03-01T16:57:43Z

@kmike , I think I have addressed all your comments. (and rebased)

kmike · 2017-03-01T17:31:35Z

@redapple tests are failing - could you please check it?

redapple · 2017-03-02T15:34:54Z

@kmike , done. Sorry about that.

kmike · 2017-03-02T21:10:06Z

scrapy/spidermiddlewares/referer.py

+                         strip_default_port=True,
+                         origin_only=origin_only)
+
+    def origin(self, r):


Is there a reason some of the arguments are named r while other are named url?

Not really. I kept request/response when the 2 arguments are needed. But I can use "url" in others.

Hm, what's the reason arguments are still named request/response, and not something like request_url? Maybe I got a bit too used to type hints :)

Most probably laziness and readability. I can change that too if you prefer.

Could you please change it? When looking at code I expect scrapy Request and Response objects to be passed to .referrer method, to figure out that they are urls you need to find calling code. I agree that request_url looks worse, but imho it can help readability here.

kmike · 2017-03-02T23:05:23Z

Looks great, thanks @redapple!

redapple added enhancement security labels Oct 5, 2016

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ Referrer policies in RefererMiddleware Oct 12, 2016

kmike reviewed Oct 18, 2016

View changes

kmike reviewed Oct 21, 2016

View changes

redapple changed the title ~~Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Nov 16, 2016

redapple added this to the v1.3 milestone Nov 16, 2016

redapple changed the title ~~[MRG] Referrer policies in RefererMiddleware~~ [WIP] Referrer policies in RefererMiddleware Jan 17, 2017

redapple force-pushed the redapple:referrer-policy branch from 41eb742 to 31dd441 Jan 17, 2017

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Jan 17, 2017

redapple changed the title ~~[MRG] Referrer policies in RefererMiddleware~~ [WIP] Referrer policies in RefererMiddleware Jan 17, 2017

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Jan 17, 2017

kmike reviewed Jan 18, 2017

View changes

docs/topics/spider-middleware.rst Outdated

using ``file://`` or ``s3://`` scheme.

.. warning::

By default, Scrapy's default referrer policy, just like `"no-referrer-when-downgrade"`_,

This comment has been minimized.

Sign in to view

kmike Jan 18, 2017
Member

"By default" looks redundant

kmike reviewed Jan 18, 2017

View changes

redapple force-pushed the redapple:referrer-policy branch from c4e8297 to f43be28 Jan 18, 2017

kmike reviewed Jan 18, 2017

View changes

redapple changed the title ~~[MRG] Referrer policies in RefererMiddleware~~ [WIP] Referrer policies in RefererMiddleware Feb 21, 2017

redapple force-pushed the redapple:referrer-policy branch from ebc2f8f to 4e78d19 Feb 21, 2017

kmike reviewed Feb 22, 2017

View changes

redapple added 15 commits Jan 17, 2017

Update StrictOriginPolicy

ebcacd3

Fix tests

b6c761d

Update docs with "strict-..." policies

c86f568

Update docs

e249abc

Update docs for new "referrer_policy" Request.meta key

03ff19d

Reword warning on no-referrer-when-downgrade policy

eb07285

Edit text

605935f

Use table for referrer policy options

3dc09ee

Add autoclass directives to document built-in policies

537683f

Rename setting to REFERRER_POLICY (with 2 Rs)

bc200d1

Pass URLs around instead of Request/Responses

04e4d08

Update tests

d2aa51c

Add test for Referer header on HTTP redirections

8226e77

Warn or fail with exception on unknown policies

6916dd6

Add tests for policy fallback on unknown policies from meta and headers

Loading status checks…

efa5003

redapple force-pushed the redapple:referrer-policy branch from c3c69d9 to efa5003 Mar 1, 2017

Fix strip_url() tests

Loading status checks…

2d55d83

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Mar 2, 2017

kmike reviewed Mar 2, 2017

View changes

redapple changed the title ~~[MRG] Referrer policies in RefererMiddleware~~ [WIP] Referrer policies in RefererMiddleware Mar 2, 2017

redapple added 3 commits Mar 2, 2017

Rename arguments

Loading status checks…

c2c5031

Remove Legacy Policy which is equivalent to UnsafeUrl Policy

db176f8

Rename arguments (bis)

Loading status checks…

fad499a

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Mar 2, 2017

kmike merged commit 7e8453c into scrapy:master Mar 2, 2017
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details


		Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'``

		.. reqmeta:: referrer_policy


		NOREFERRER_SCHEMES = LOCAL_SCHEMES

		def referrer(self, response, request):

scrapy / scrapy

[MRG] Referrer policies in RefererMiddleware #2306

[MRG] Referrer policies in RefererMiddleware #2306

Conversation

redapple commented Oct 5, 2016 • edited

codecov-io commented Oct 5, 2016 • edited

Codecov Report

This comment has been minimized.

kmike Oct 18, 2016 • edited Member

This comment has been minimized.

redapple Oct 18, 2016 Author Contributor

This comment has been minimized.

redapple Oct 18, 2016 Author Contributor

This comment has been minimized.

redapple Oct 19, 2016 Author Contributor

This comment has been minimized.

kmike Oct 21, 2016 Member

This comment has been minimized.

redapple Oct 21, 2016 Author Contributor

This comment has been minimized.

kmike Oct 21, 2016 Member

This comment has been minimized.

redapple Oct 25, 2016 Author Contributor

redapple commented Jan 17, 2017

redapple commented Jan 18, 2017

redapple commented Jan 18, 2017

This comment has been minimized.

kmike Jan 18, 2017 Member

This comment has been minimized.

kmike Jan 18, 2017 Member

This comment has been minimized.

kmike Jan 18, 2017 Member

This comment has been minimized.

redapple Jan 26, 2017 Author Contributor

This comment has been minimized.

redapple Jan 26, 2017 Author Contributor

This comment has been minimized.

kmike Feb 21, 2017 Member

This comment has been minimized.

kmike Feb 21, 2017 Member

This comment has been minimized.

kmike Feb 21, 2017 Member

This comment has been minimized.

kmike Jan 18, 2017 Member

This comment has been minimized.

kmike Jan 18, 2017 Member

This comment has been minimized.

redapple Jan 26, 2017 Author Contributor

This comment has been minimized.

kmike Feb 22, 2017 Member

This comment has been minimized.

kmike Feb 22, 2017 Member

This comment has been minimized.

kmike Jan 18, 2017 Member

This comment has been minimized.

kmike Jan 18, 2017 Member

redapple commented Feb 21, 2017

redapple commented Feb 21, 2017

This comment has been minimized.

kmike Feb 22, 2017 Member

This comment has been minimized.

redapple Feb 22, 2017 Author Contributor

This comment has been minimized.

kmike Feb 22, 2017 Member

This comment has been minimized.

redapple Feb 22, 2017 Author Contributor

This comment has been minimized.

kmike Feb 22, 2017 • edited Member

This comment has been minimized.

redapple Feb 22, 2017 Author Contributor

redapple commented Mar 1, 2017

kmike commented Mar 1, 2017

redapple commented Mar 2, 2017

This comment has been minimized.

kmike Mar 2, 2017 • edited Member

This comment has been minimized.

redapple Mar 2, 2017 Author Contributor

This comment has been minimized.

kmike Mar 2, 2017 Member

This comment has been minimized.

redapple commented Oct 5, 2016 •

edited

codecov-io commented Oct 5, 2016 •

edited

kmike Oct 18, 2016 •

edited

Member

redapple Oct 18, 2016
Author Contributor

redapple Oct 18, 2016
Author Contributor

redapple Oct 19, 2016
Author Contributor

kmike Oct 21, 2016
Member

redapple Oct 21, 2016
Author Contributor

kmike Oct 21, 2016
Member

redapple Oct 25, 2016
Author Contributor

kmike Jan 18, 2017
Member

kmike Jan 18, 2017
Member

kmike Jan 18, 2017
Member

redapple Jan 26, 2017
Author Contributor

redapple Jan 26, 2017
Author Contributor

kmike Feb 21, 2017
Member

kmike Feb 21, 2017
Member

kmike Feb 21, 2017
Member

kmike Jan 18, 2017
Member

kmike Jan 18, 2017
Member

redapple Jan 26, 2017
Author Contributor

kmike Feb 22, 2017
Member

kmike Feb 22, 2017
Member

kmike Jan 18, 2017
Member

kmike Jan 18, 2017
Member

kmike Feb 22, 2017
Member

redapple Feb 22, 2017
Author Contributor

kmike Feb 22, 2017
Member

redapple Feb 22, 2017
Author Contributor

kmike Feb 22, 2017 •

edited

Member

redapple Feb 22, 2017
Author Contributor

kmike Mar 2, 2017 •

edited

Member

redapple Mar 2, 2017
Author Contributor

kmike Mar 2, 2017
Member

redapple Mar 2, 2017
Author Contributor

kmike Mar 2, 2017
Member

redapple Mar 2, 2017
Author Contributor