[MRG] Referrer policies in RefererMiddleware #2306

redapple · 2016-10-05T15:26:13Z

I've based the tests and implementation on https://www.w3.org/TR/referrer-policy/

By default, with this change, RefererMiddleware does not send a Referer header (referrer value):

when the source response is file:// or s3://,
nor from https:// response to an http:// request.

However, it does send a referrer value for any https:// to another https:// (as browsers do, "no-referrer-when-downgrade" being the default policy)
This needs discussion.

User can change the policy per-request, using a new referrer_policy meta key, with values from "no-referrer" / "no-referrer-when-downgrade" / "same-origin" / "origin" / "origin-when-cross-origin" / "unsafe-url".

~~Still missing:~~

Try to use urlparse_cached()
Make default policy customizable through a REFERER_DEFAULT_POLICY setting
Test custom policies
Docs updates
Add tests for policy given by response headers (Referrer-Policy) -- is this even used in practice by web servers?
Handle referrers during redirects

To handle redirects, I added a signal handler on request_scheduled. I did not find a way to test both redirect middleware and referer middleware (I did not search too much though).
Suggestions to do that are welcome.

codecov-io · 2016-10-05T15:37:09Z

Codecov Report

Merging #2306 into master will decrease coverage by -0.6%.
The diff coverage is 94.44%.

@@            Coverage Diff            @@
##           master    #2306     +/-   ##
=========================================
- Coverage   83.93%   83.34%   -0.6%     
=========================================
  Files         161      161             
  Lines        8920     9060    +140     
  Branches     1316     1290     -26     
=========================================
+ Hits         7487     7551     +64     
- Misses       1176     1240     +64     
- Partials      257      269     +12

Impacted Files	Coverage Δ
scrapy/utils/url.py	`100% <100%> (ø)`	✅
scrapy/settings/default_settings.py	`98.63% <100%> (ø)`	✅
scrapy/spidermiddlewares/referer.py	`93% <93.98%> (+8.39%)`	✅
scrapy/core/downloader/handlers/s3.py	`62.9% <0%> (-32.26%)`	❌
scrapy/utils/boto.py	`46.66% <0%> (-26.67%)`	❌
scrapy/utils/gz.py	`85.29% <0%> (-14.71%)`	❌
scrapy/link.py	`86.36% <0%> (-13.64%)`	❌
scrapy/core/downloader/tls.py	`80% <0%> (-8.58%)`	❌
scrapy/_monkeypatches.py	`50% <0%> (-7.15%)`	❌
scrapy/utils/iterators.py	`95.45% <0%> (-4.55%)`	❌
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b49b9c...fad499a. Read the comment docs.

kmike · 2016-10-18T14:18:08Z

scrapy/spidermiddlewares/referer.py

+        if stripped is not None:
+            return urlunparse(stripped)
+
+    def strip_url(self, req_or_resp, origin_only=False):


What do you think about making it an utility function? It can be more reusable and easier to test this way (even doctests could be enough). I'd make it accept URL or ParseResult instead of scrapy Request or Response.

Good idea @kmike , I even had a similar use-case for https://github.com/umrashrf/scrapy/pull/1/files#diff-c1b844ba341115f81f50b33fd691fcbbR14

@kmike , are you thinking of an addition to w3lib.url?
Also, the current output is a ParseResult tuple, mainly for comparing origins.
Does it make sense to have the utility function serialize/urlunparse on demand, with some argument?
Or should we just compare origins as strings? (which is probably the cleanest as I don't really like returning different non-None types)

@kmike , I moved some of it to scrapy.utils.url. What do you think?

kmike · 2016-10-21T08:30:26Z

scrapy/utils/url.py

@@ -103,3 +103,35 @@ def guess_scheme(url):
        return any_to_uri(url)
    else:
        return add_http_if_no_scheme(url)
+
+
+def strip_url_credentials(url, origin_only=False, keep_fragments=False):


Could you please add a docstring to this function?

The function is named 'strip credentials', but it also strips port, I think this is not intuitive.

Right, it strips standard HTTP and HTTPS port numbers. This is mainly for comparing origins.
Docstring is needed indeed.
Does it make sense to have this feature as switchable through an argument?

Yeah, it should be either an argument or a separate function. If it becomes an argument then I think it is better to rename the function (not sure about the new name).

@kmike , code updated with new name for helper strip_url()

redapple · 2017-01-17T13:35:40Z

I'm adding 2 policies that appeared in the working draft:

redapple · 2017-01-18T13:58:19Z

Alright, @kmike , @eliasdorneles , @lopuhin , @dangra , I think this is ready for review.
Thanks in advance!

redapple · 2017-01-18T14:01:53Z

(I'm not sure why I use ftps in some tests. It's not supported by Scrapy)

kmike · 2017-01-18T14:03:29Z

docs/topics/spider-middleware.rst

+using ``file://`` or ``s3://`` scheme.
+
+.. warning::
+    By default, Scrapy's default referrer policy, just like `"no-referrer-when-downgrade"`_,


"By default" looks redundant

kmike · 2017-01-18T15:48:55Z

docs/topics/spider-middleware.rst

+.. warning::
+    Scrapy's default referrer policy, just like `"no-referrer-when-downgrade"`_,
+    will send a non-empty "Referer" header from any ``https://`` to any ``https://`` URL,
+    even if the domain is different.


If that's what browsers do by default then I think it makes sense to note that, to make a warning less scary.

kmike · 2017-01-18T16:57:04Z

docs/topics/spider-middleware.rst

+
+- a path to a ``scrapy.spidermiddlewares.referer.ReferrerPolicy`` subclass,
+  either a custom one or one of the built-in ones
+  (see ``scrapy.spidermiddlewares.referer``),


Is there a page in the docs we can reference to?

What kind of reference do you have in mind? built-in classes?

We could also define an interface for policies, so people know what to implement.

@kmike , an alternative is to use a table:

$ git diff diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 9010ae7..46a84d0 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -346,21 +346,21 @@ This setting accepts: - a path to a ``scrapy.spidermiddlewares.referer.ReferrerPolicy`` subclass, either a custom one or one of the built-in ones (see ``scrapy.spidermiddlewares.referer``), -- or one of the standard W3C-defined string values: - - - `"no-referrer"`_, - - `"no-referrer-when-downgrade"`_ - (the W3C-recommended default, used by major web browsers), - - `"same-origin"`_, - - `"origin"`_, - - `"strict-origin"`_, - - `"origin-when-cross-origin"`_, - - `"strict-origin-when-cross-origin"`_, - - or `"unsafe-url"`_ - (not recommended). - -It can also be the non-standard value ``"scrapy-default"`` to use -Scrapy's default referrer policy. +- or one of the standard W3C-defined string values + +======================================= ======================================================================== ======================================================= +String value Class name +======================================= ======================================================================== ======================================================= +`"no-referrer"`_ ``'scrapy.spidermiddlewares.referer.NoReferrerPolicy'`` +`"no-referrer-when-downgrade"`_ ``'scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy'`` the W3C-recommended default, used by major web browsers +`"same-origin"`_ ``'scrapy.spidermiddlewares.referer.SameOriginPolicy'`` +`"origin"`_ ``'scrapy.spidermiddlewares.referer.OriginPolicy'`` +`"strict-origin"`_ ``'scrapy.spidermiddlewares.referer.StrictOriginPolicy'`` +`"origin-when-cross-origin"`_ ``'scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy'`` +`"strict-origin-when-cross-origin"`_ ``'scrapy.spidermiddlewares.referer.StrictOriginWhenCrossOriginPolicy'`` +`"unsafe-url"`_ ``'scrapy.spidermiddlewares.referer.UnsafeUrlPolicy'`` NOT recommended +``"scrapy-default"`` ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'`` Scrapy's default policy (see below) +======================================= ======================================================================== ======================================================= Scrapy's default referrer policy is a variant of `"no-referrer-when-downgrade"`_, with the addition that "Referer" is not sent if the parent request was

I like such table.

Also, adding autodocs for these policy classes looks helpful - they already have docstrings, it is just a matter of putting several autoclass labels to scrapy rst docs.

as for "see scrapy.spidermiddlewares.referer", the question was how would user find this scrapy.spidermiddlewares.referer - there is no link or anything.

Documenting policy interface sounds like too much for me; I'm not sure this is something users may want to extend.

kmike · 2017-01-18T16:59:35Z

docs/topics/spider-middleware.rst

+
+Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'``
+
+.. reqmeta:: referrer_policy


It could make sense use the same spelling everywhere in Scrapy. Currently option name (REFERER_POLICY) has one 'r' while meta key name has 'rr'.

aha, I see, we already have REFERER_ENABLED

I'm ok with changing to REFERRER_POLICY. (It's a shame about that spelling mistake in the RFCs.)

We're loosing either way - now option names are inconsistent :)

I guess the way to go is to rename the other option and a middleware itself as well; of course, it can be done later.

kmike · 2017-01-18T17:00:45Z

docs/topics/spider-middleware.rst

+Scrapy's default referrer policy.
+
+Scrapy's default referrer policy is a variant of `"no-referrer-when-downgrade"`_,
+with the addition that "Referrer" is not sent if the parent request was


header name is Referer

kmike · 2017-01-18T17:01:59Z

docs/topics/spider-middleware.rst

+using ``file://`` or ``s3://`` scheme.
+
+.. warning::
+    Scrapy's default referrer policy—just like `"no-referrer-when-downgrade"`_,


a nitpick - there should be spaces before and after a dash :)

redapple · 2017-02-21T16:21:59Z

@kmike , I rebased, updated the docs with autoclass directives and renamed the setting to REFERRER_POLICY.

redapple · 2017-02-21T16:25:26Z

Another thing could be to move the policies to scrapy/extensions, similar to HTTP cache policies (with scrapy/scrapy/extensions/httpcache.py). What do you think?

kmike · 2017-02-22T17:30:25Z

scrapy/spidermiddlewares/referer.py

+                    try:
+                        self.default_policy = _policy_classes[policy.lower()]
+                    except:
+                        raise NotConfigured("Unknown referrer policy name %r" % policy)


I think it is better to raise an error and stop the crawl instead of silently disabling the middleware if a policy is unknown.

Good point!

kmike · 2017-02-22T17:31:47Z

scrapy/spidermiddlewares/referer.py

+            policy_name = to_native_str(
+                response.headers.get('Referrer-Policy', '').decode('latin1'))
+
+        cls = _policy_classes.get(policy_name.lower(), self.default_policy)


What do you think about warning for unknown policies? They can be typos.

Makes sense.

kmike · 2017-02-22T17:40:30Z

scrapy/spidermiddlewares/referer.py

+
+    NOREFERRER_SCHEMES = LOCAL_SCHEMES
+
+    def referrer(self, response, request):


It seems the API requires response and request (not response_url and request_url) only because of urlparse_cached - you wanted to avoid duplicate computations. It leads to some ugly code when a fake Response is created. Is this correct, or are there cases response and requestis indeed needed?

I wonder how much is the effect, given that urlparse has its own internal cache (using urls as keys, not response/requests).

Right @kmike , that was the reason.
I cannot think right now of a real need for handling request/responses directly. Meta information and HTTP headers ("Referrer-Policy") are to be interpreted at middleware level, and not at policy level I believe.

Indeed the fake response is quite ugly.
Let me clean it up with URLs all along.

redapple · 2017-03-01T16:57:43Z

@kmike , I think I have addressed all your comments. (and rebased)

kmike · 2017-03-01T17:31:35Z

@redapple tests are failing - could you please check it?

redapple · 2017-03-02T15:34:54Z

@kmike , done. Sorry about that.

kmike · 2017-03-02T21:10:06Z

scrapy/spidermiddlewares/referer.py

+                         strip_default_port=True,
+                         origin_only=origin_only)
+
+    def origin(self, r):


Is there a reason some of the arguments are named r while other are named url?

Not really. I kept request/response when the 2 arguments are needed. But I can use "url" in others.

Hm, what's the reason arguments are still named request/response, and not something like request_url? Maybe I got a bit too used to type hints :)

Most probably laziness and readability. I can change that too if you prefer.

Could you please change it? When looking at code I expect scrapy Request and Response objects to be passed to .referrer method, to figure out that they are urls you need to find calling code. I agree that request_url looks worse, but imho it can help readability here.

kmike · 2017-03-02T23:05:23Z

Looks great, thanks @redapple!

redapple added enhancement security labels Oct 5, 2016

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ Referrer policies in RefererMiddleware Oct 12, 2016

kmike reviewed Oct 18, 2016

View reviewed changes

kmike reviewed Oct 21, 2016

View reviewed changes

redapple changed the title ~~Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Nov 16, 2016

redapple added this to the v1.3 milestone Nov 16, 2016

redapple changed the title ~~[MRG] Referrer policies in RefererMiddleware~~ [WIP] Referrer policies in RefererMiddleware Jan 17, 2017

redapple force-pushed the referrer-policy branch from 41eb742 to 31dd441 Compare January 17, 2017 16:58

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Jan 17, 2017

redapple changed the title ~~[MRG] Referrer policies in RefererMiddleware~~ [WIP] Referrer policies in RefererMiddleware Jan 17, 2017

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Jan 17, 2017

kmike reviewed Jan 18, 2017

View reviewed changes

redapple force-pushed the referrer-policy branch from c4e8297 to f43be28 Compare January 18, 2017 15:50

kmike reviewed Jan 18, 2017

View reviewed changes

redapple changed the title ~~[MRG] Referrer policies in RefererMiddleware~~ [WIP] Referrer policies in RefererMiddleware Feb 21, 2017

redapple force-pushed the referrer-policy branch from ebc2f8f to 4e78d19 Compare February 21, 2017 15:49

kmike reviewed Feb 22, 2017

View reviewed changes

redapple added 16 commits March 1, 2017 17:50

Update NoReferrerWhenDowngradePolicy

deb8567

Update StrictOriginPolicy

ebcacd3

Fix tests

b6c761d

Update docs with "strict-..." policies

c86f568

Update docs

e249abc

Update docs for new "referrer_policy" Request.meta key

03ff19d

Reword warning on no-referrer-when-downgrade policy

eb07285

Edit text

605935f

Use table for referrer policy options

3dc09ee

Add autoclass directives to document built-in policies

537683f

Rename setting to REFERRER_POLICY (with 2 Rs)

bc200d1

Pass URLs around instead of Request/Responses

04e4d08

Update tests

d2aa51c

Add test for Referer header on HTTP redirections

8226e77

Warn or fail with exception on unknown policies

6916dd6

Add tests for policy fallback on unknown policies from meta and headers

efa5003

redapple force-pushed the referrer-policy branch from c3c69d9 to efa5003 Compare March 1, 2017 16:51

Fix strip_url() tests

2d55d83

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Mar 2, 2017

kmike reviewed Mar 2, 2017

View reviewed changes

redapple changed the title ~~[MRG] Referrer policies in RefererMiddleware~~ [WIP] Referrer policies in RefererMiddleware Mar 2, 2017

redapple added 3 commits March 2, 2017 22:53

Rename arguments

c2c5031

Remove Legacy Policy which is equivalent to UnsafeUrl Policy

db176f8

Rename arguments (bis)

fad499a

redapple changed the title ~~[WIP] Referrer policies in RefererMiddleware~~ [MRG] Referrer policies in RefererMiddleware Mar 2, 2017

kmike merged commit 7e8453c into scrapy:master Mar 2, 2017


		Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'``

		.. reqmeta:: referrer_policy


		NOREFERRER_SCHEMES = LOCAL_SCHEMES

		def referrer(self, response, request):

[MRG] Referrer policies in RefererMiddleware #2306

[MRG] Referrer policies in RefererMiddleware #2306

Conversation

redapple commented Oct 5, 2016 • edited

codecov-io commented Oct 5, 2016 • edited

Codecov Report

kmike Oct 18, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redapple commented Jan 17, 2017

redapple commented Jan 18, 2017

redapple commented Jan 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redapple commented Feb 21, 2017

redapple commented Feb 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmike Feb 22, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redapple commented Mar 1, 2017

kmike commented Mar 1, 2017

redapple commented Mar 2, 2017

kmike Mar 2, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmike commented Mar 2, 2017

redapple commented Oct 5, 2016 •

edited

codecov-io commented Oct 5, 2016 •

edited

kmike Oct 18, 2016 •

edited

kmike Feb 22, 2017 •

edited

kmike Mar 2, 2017 •

edited