Mark cookie-sharing pixels as tracking #2147

bcyphers · 2018-08-24T19:12:27Z

Add new heuristic to detect cookie sharing via tracking pixels.

Logic works like this:
Ever time there is a request for an "image" resource to a third party, PB inspects all the query args in the request, and compares them against all the cookies on the page belonging to other domains. If any query argument has a significantly long substring in common with any cookie values, PB marks that request as tracking.

In practice, several third-party analytics services work like this. For example, https://nytimes.com uses Google Analytics. When a user visits for the first time, it sets a first-party cookie like _gid=GA1.3.1327319847.1535103025, and then loads a tracking pixel from www.google-analytics.com with a request including the argument ?_gid=1327319847.1535103025. bluekai.com is another service that appears to do the same thing.

This will probably have some false positives, so we should try to figure out how to exclude common substrings from the entropy estimation. Right now I'm ignoring all substrings that have a part of the page URL in them -- otherwise we'd mark requests like example-cdn.com/resource?site=nytimes.com as tracking.

Also: do we need to limit this to "pixels," or can we apply this to every kind of request?

I think this addresses most of #367. Closes #340, closes #2088. Part of #2114.

Add new heuristic to detect cookie syncing via tracking pixels. This includes one of the tracking methods used by google analytics. Call pixelCookieSyncAccounting from heuristicBlockingAccounting.

Change it from a "longest common substring" finder to an "all common substrings longer than X" finder. Rename some variables and update comments.

Since details.initiator isn't available in firefox (and isn't reliable in chrome), create a new data structure for tabURLs that saves the full URLs of first-party pages indexed by tab IDs. Check common substrings to make sure they are not substrings or superstrings of the tab URL.

bcyphers · 2018-08-29T05:03:54Z

Just tested this on the top 50 Majestic sites. This heuristic triggers a lot -- the scan found 66 potential pixel-tracking actions, compared to 121 third-party cookies, 16 supercookies, and 1 fingerprint.

There are a lot of false positives, but most of the "trackers" actually look like trackers. Here are some examples that look legit:

Tracker domain	First-party domain	Tracking string
google-analytics.com	linkedin.com	418299633.1535508492
facebook.com	adobe.com	W4YARwAAAK13z0nI
hexagon-analytics.com	flickr.com	82d07aeb-ce29-4e6d-a35f-76ce3545fa38
doubleclick.net	linkedin.com	418299633.1535508492
rlcdn.com	sourceforge.net	6472222808240991374
media.net	nytimes.com	6537396857404675838
crsspxl.com	sourceforge.net	78cf5b85-f0e3-4500-807f-471419b49974
crsspxl.com	sourceforge.net	6537396857404675838
chartbeat.net	nytimes.com	ZQtbgC42A0aDnCOORiLzMjV-iNK

And some likely false positives:

Tracker domain	First-party domain	Tracking string
keywee.co	nytimes.com	1152x574
pinterest.com	nytimes.com	:false,
myvisualiq.net	soundcloud.com	https%3A%2F%2F
everesttech.net	adobe.com	www.facebook.com
wp.com	wordpress.com	logged-out-homepage
wp.com	wordpress.com	x86_64
yahoo.com	flickr.com	&b=3&s=
githubapp.com	github.com	1152x574

Edit: After looking at all 66, I'd estimate 13 of the tracking strings were false alarms (most of them are pasted above), and the other 53 were legit. This might be a good time to improve the cookie entropy heuristic (https://github.com/EFForg/privacybadger/blob/master/src/js/heuristicblocking.js#L222).

bcyphers · 2018-08-29T22:04:36Z

Finished a scan of 1,200 sites (it errored out before it finished 2k).

Since we started scanning, badger-sett has learned to block 1389 domains at least once. With the new heuristic, the scan learns to block 64 domains that have never been blocked before.

Here's a snapshot of the most prominent new tracking domains:

Tracker	Times caught tracking
google-analytics.com	72
bluekai.com	54
chartbeat.net	53
crwdcntrl.net	46
visualwebsiteoptimizer.com	20
alexametrics.com	18
pippio.com	15
mfadsrvr.com	15
nexac.com	14
ssix.io	13
bouncex.net	12
trustx.org	10

At a glance, most of these do look like trackers. Once a clean scan of 2k completes, maybe we can comb through what the badger learns and try to work around false positives.

Try to better estimate entropy of a string by guessing which group of characters it was created from.

Add list of common query substrings derived from actual requests, and filter those out before estimating the entropy of a string.

ghostwords

Apologies, this is definitely going to be a multi-stage review.

My first concern is adequate performance. We've got a quadruple loop now in the common case (no cookie syncing). Can we remove any nesting? Can we precompute/cache anything?

When it comes to the heuristic, I'd rather catch fewer things now (to avoid false positives, improve performance, simplify the code maybe), and widen the net later.

Looking forward to the fixes and super excited about releasing this!

src/js/heuristicblocking.js

ghostwords · 2019-06-06T20:37:46Z

src/js/utils.js

@@ -108,8 +108,7 @@ function explodeSubdomains(fqdn, all) {
 /*
 * Estimate the max possible entropy of str using min and max
 * char codes observed in the string.
- * Tends to overestimate in many cases, e.g. hexadecimals.
- * Also, sensitive to case, e.g. bad1dea is different than BAD1DEA
+ * Sensitive to case, e.g. bad1dea is different than BAD1DEA
 */
 function estimateMaxEntropy(str) {


Are these changes beneficial for the local storage ("supercookie") case as well?

Still a question whether these changes are beneficial for/applicable to the local storage ("supercookie") case as well.

Yes, they should be. These changes are all meant to tighten up the way the heuristic worked before -- entropy = (size of character set) * (number of characters).

Should add unit tests to exercise estimateMaxEntropy code paths.

src/js/heuristicblocking.js

Only get cookies from the containing frame's origin; get cookies after checking whether a request's type is "image"; pass more arguments to accounting function to reduce util calls; better comments; consistent use of "share" instead of "sync".

Move cookie sharing detection logic out of heuristicBlockingAccounting, and call it directly from the listener.

ghostwords

Thanks! It may be best to keep using tabOrigins/tabUrls for now, unfortunately.

src/js/heuristicblocking.js

ghostwords · 2019-06-10T16:10:55Z

src/js/utils.js

@@ -108,8 +108,7 @@ function explodeSubdomains(fqdn, all) {
 /*
 * Estimate the max possible entropy of str using min and max
 * char codes observed in the string.
- * Tends to overestimate in many cases, e.g. hexadecimals.
- * Also, sensitive to case, e.g. bad1dea is different than BAD1DEA
+ * Sensitive to case, e.g. bad1dea is different than BAD1DEA
 */
 function estimateMaxEntropy(str) {


Still a question whether these changes are beneficial for/applicable to the local storage ("supercookie") case as well.

Merge pixel-sharing listener back into heuristicBlockingAccounting. Add back tabOrigins and tabURLs data structures. Clean up comments.

ghostwords

Thank you!

src/js/heuristicblocking.js

src/js/webrequest.js

src/js/heuristicblocking.js

Short circuit when there aren't any cookies or cookie values are too short. Don't check for cookie sharing inside onBeforeSendHeaders synchfonous listener.

ghostwords

Looking good! Some more feedback.

src/js/heuristicblocking.js

ghostwords · 2019-06-17T16:10:02Z

src/js/heuristicblocking.js

+   * @param cookies are the result of chrome.cookies.getAll()
+   * @returns {*}
+   */
+  pixelCookieShareAccounting: function (details, cookies) {


We should have a functional test for pixel cookie sharing detection. Like our localStorage tracking tests but for cookie sharing.

src/js/heuristicblocking.js

ghostwords · 2019-06-17T16:10:57Z

src/js/utils.js

+// Adapted from https://gist.github.com/jaewook77/cd1e3aa9449d7ea4fb4f
+// Find all common substrings more than 8 characters long, using DYNAMIC
+// PROGRAMMING
+function findCommonSubstrings(str1, str2) {


Should add unit tests to exercise findCommonSubstrings code paths.

ghostwords · 2019-06-17T16:11:38Z

src/js/utils.js

@@ -108,8 +108,7 @@ function explodeSubdomains(fqdn, all) {
 /*
 * Estimate the max possible entropy of str using min and max
 * char codes observed in the string.
- * Tends to overestimate in many cases, e.g. hexadecimals.
- * Also, sensitive to case, e.g. bad1dea is different than BAD1DEA
+ * Sensitive to case, e.g. bad1dea is different than BAD1DEA
 */
 function estimateMaxEntropy(str) {


Should add unit tests to exercise estimateMaxEntropy code paths.

rename variables to snake_case; remove _extractArgs and inline URL arg parsing because we don't capture POST requests anyway; conserve variables and remove redundant definitions.

src/js/heuristicblocking.js

If we've seen a particular origin tracking on a particular first party before, stop checking for tracking actions.

ghostwords

Alright! Let's merge to master so we can get a good week's worth of badger-sett runs. We'll follow up with tests and any other performance/bug fixes with separate PRs.

Detect cookie sharing via image (pixel) querystrings.

And top-level document/main frame images. Following up on #2147

bcyphers requested review from ghostwords and andresbase August 24, 2018 19:12

bcyphers changed the title ~~Mark cookie-syncing tracking pixels as tracking~~ Mark cookie-syncing pixels as tracking Aug 24, 2018

Add pixelCookieSyncAccounting heuristic

e99466f

Add new heuristic to detect cookie syncing via tracking pixels. This includes one of the tracking methods used by google analytics. Call pixelCookieSyncAccounting from heuristicBlockingAccounting.

bcyphers force-pushed the pixel-tracking-heuristic branch from ba28512 to e99466f Compare August 24, 2018 19:20

bcyphers added enhancement heuristic Badger's core learning-what-to-block functionality labels Aug 24, 2018

bcyphers changed the title ~~Mark cookie-syncing pixels as tracking~~ Mark cookie-sharing pixels as tracking Aug 24, 2018

bcyphers added 2 commits August 28, 2018 18:34

Update common substring function

cbb0816

Change it from a "longest common substring" finder to an "all common substrings longer than X" finder. Rename some variables and update comments.

ghostwords mentioned this pull request Sep 4, 2018

Detect and block tracking pixels #794

Open

bcyphers added 2 commits September 14, 2018 15:33

Update cookie entropy function

d560c1b

Try to better estimate entropy of a string by guessing which group of characters it was created from.

Update pixel heuristic to exclude stopwords

395592f

Add list of common query substrings derived from actual requests, and filter those out before estimating the entropy of a string.

ghostwords force-pushed the master branch from 5afce52 to 257be9c Compare September 18, 2018 15:19

bcyphers added 3 commits June 4, 2019 16:57

Merge branch 'master' into pixel-tracking-heuristic

f830c70

tracking strings must be at least 8 characters long

5b3dbfd

Turn draft code into working code; add extra check for 8chars

66871d1

bcyphers force-pushed the pixel-tracking-heuristic branch from ecf0db0 to 66871d1 Compare June 5, 2019 02:52

ghostwords requested changes Jun 6, 2019

View reviewed changes

ghostwords reviewed Jun 7, 2019

View reviewed changes

src/js/heuristicblocking.js Outdated Show resolved Hide resolved

bcyphers added 2 commits June 7, 2019 12:53

Better comments and small optimizations

f6859f4

Only get cookies from the containing frame's origin; get cookies after checking whether a request's type is "image"; pass more arguments to accounting function to reduce util calls; better comments; consistent use of "share" instead of "sync".

Remove tabOrigins and use tabData instead

0e7604c

bcyphers force-pushed the pixel-tracking-heuristic branch 2 times, most recently from 09bc72f to 7eb885b Compare June 8, 2019 00:08

Separate cookie sharing out of hBA

022e0f8

Move cookie sharing detection logic out of heuristicBlockingAccounting, and call it directly from the listener.

ghostwords requested changes Jun 10, 2019

View reviewed changes

Address more PR comments

d5f34c3

Merge pixel-sharing listener back into heuristicBlockingAccounting. Add back tabOrigins and tabURLs data structures. Clean up comments.

ghostwords requested changes Jun 11, 2019

View reviewed changes

Update comments and other small changes

948fd30

Short circuit when there aren't any cookies or cookie values are too short. Don't check for cookie sharing inside onBeforeSendHeaders synchfonous listener.

ghostwords reviewed Jun 17, 2019

View reviewed changes

Remove _extractArgs and other changes

47e3f80

rename variables to snake_case; remove _extractArgs and inline URL arg parsing because we don't capture POST requests anyway; conserve variables and remove redundant definitions.

ghostwords reviewed Jun 18, 2019

View reviewed changes

src/js/heuristicblocking.js Show resolved Hide resolved

bcyphers force-pushed the pixel-tracking-heuristic branch from b69319c to 9147d2d Compare June 19, 2019 19:39

Abort based on snitch map before action map check

e71e272

If we've seen a particular origin tracking on a particular first party before, stop checking for tracking actions.

bcyphers force-pushed the pixel-tracking-heuristic branch from 9147d2d to e71e272 Compare June 19, 2019 20:22

ghostwords approved these changes Jun 19, 2019

View reviewed changes

ghostwords merged commit e71e272 into master Jun 19, 2019

ghostwords added a commit that referenced this pull request Jun 19, 2019

Merge pull request #2147.

58deace

Detect cookie sharing via image (pixel) querystrings.

ghostwords deleted the pixel-tracking-heuristic branch June 19, 2019 22:18

This was referenced Jun 19, 2019

Detect and handle 3rd → 1st party JS (Google Analytics) #367

Closed

Add tests for pixel cookie sharing heuristic #2402

Closed

Update FAQ/UI/docs for pixel cookie sharing heuristic #2403

Closed

ghostwords mentioned this pull request Jul 3, 2019

Audit pre-trained list for missing top known trackers #2114

Open

woctezuma mentioned this pull request Jul 13, 2019

💬 Discussion | Obsolete warning about Privacy Badger and Google Analytics? privacytools/privacytools.io#1022

Closed

jawz101 mentioned this pull request Nov 5, 2019

Crosspixel (crsspxl.com) tracking pixels not being blocked #2498

Closed

ghostwords added a commit that referenced this pull request Mar 4, 2020

Limit cookie sharing to non-HttpOnly cookies

e88205d

And top-level document/main frame images. Following up on #2147

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mark cookie-sharing pixels as tracking #2147

Mark cookie-sharing pixels as tracking #2147

bcyphers commented Aug 24, 2018 •

edited

bcyphers commented Aug 29, 2018 •

edited

bcyphers commented Aug 29, 2018 •

edited

ghostwords left a comment •

edited

ghostwords Jun 6, 2019

ghostwords Jun 10, 2019 •

edited

bcyphers Jun 10, 2019

ghostwords Jun 17, 2019

ghostwords left a comment

ghostwords Jun 10, 2019 •

edited

ghostwords left a comment

ghostwords left a comment

This comment was marked as resolved.

ghostwords Jun 17, 2019

ghostwords Jun 17, 2019

ghostwords Jun 17, 2019

ghostwords left a comment

Mark cookie-sharing pixels as tracking #2147

Mark cookie-sharing pixels as tracking #2147

Conversation

bcyphers commented Aug 24, 2018 • edited

bcyphers commented Aug 29, 2018 • edited

bcyphers commented Aug 29, 2018 • edited

ghostwords left a comment • edited

Choose a reason for hiding this comment

ghostwords Jun 6, 2019

Choose a reason for hiding this comment

ghostwords Jun 10, 2019 • edited

Choose a reason for hiding this comment

bcyphers Jun 10, 2019

Choose a reason for hiding this comment

ghostwords Jun 17, 2019

Choose a reason for hiding this comment

ghostwords left a comment

Choose a reason for hiding this comment

ghostwords Jun 10, 2019 • edited

Choose a reason for hiding this comment

ghostwords left a comment

Choose a reason for hiding this comment

ghostwords left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

ghostwords Jun 17, 2019

Choose a reason for hiding this comment

ghostwords Jun 17, 2019

Choose a reason for hiding this comment

ghostwords Jun 17, 2019

Choose a reason for hiding this comment

ghostwords left a comment

Choose a reason for hiding this comment

bcyphers commented Aug 24, 2018 •

edited

bcyphers commented Aug 29, 2018 •

edited

bcyphers commented Aug 29, 2018 •

edited

ghostwords left a comment •

edited

ghostwords Jun 10, 2019 •

edited

ghostwords Jun 10, 2019 •

edited