Beacon sending lesson data on close blocked by EasyPrivacy to uBO Trusted Site

[PinkDuck]

Prerequisites

• This is not a support issue or a question. For support, questions, or help, visit /r/uBlockOrigin.
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue.
• I am running the latest version of uBlock Origin (uBO).

I tried to reproduce the issue when...

• uBO is the only extension.
• uBO uses default lists and settings.
• using a new, unmodified browser profile.

Description

Student lesson data in a commercial system is being blocked by uBO when navigating away from an active lesson. The system first attempts a fetch() request, which fails in NS_BINDING_FAILURE then falls back to sendBeacon mechanism using text/plain JSON content. However, the async delay in sending beacon means the host can be a non-Trusted Site by the time it gets intercepted by uBO. So either the EasyPrivacy list needs an exception for this particular site, or uBO could be adjusted to allow ping requests sent to Trusted Sites when processing privacy lists.

Example blocked item extract from uBO Logger:

Filter	*$ping,3p
Filter list	EasyPrivacy
 
Context	www.edp24.co.uk
Partyness	(3) edp24.co.uk ⇒ ljcreatelms.com
Type	beacon
URL	https://www.ljcreatelms.com/ScormData.aspx/Terminate

Static workaround: @@||www.ljcreatelms.com/ScormData.aspx/Terminate$ping

Where www.ljcreatelms.com host is a uBO Trusted Site

URL(s) where the issue occurs.

https://www.ljcreatelms.com (uBlock disabled) sendBeacon after navigate to https://www.edp24.co.uk/news (uBlock enabled)

Also https://www.ljcreatelms.co.uk/ and https://lp1.ljcreatelms.com/

Screenshot(s)

uBO version

1.46.0

Browser name and version

Mozilla Firefox x64 108.0.2

Settings

Added "www.ljcreatelms.com" (without quotes) to Trusted Sites, confirmed uBO inactive at that site

Notes

Tried disabling uBO in Firefox Extensions, issue did not occur
Tried navigating to another trusted site, issue did not occur
Only navigating from trusted to untrusted blocks the Beacon being sent back to Trusted
Clearing Firefox Cache (Everything) seems to increase the likelihood of occurrence
Not an issue using either AdBlock or ABP with respective EasyPrivacy/tracker options
Noticed Firefox Fetch API keepalive support bug and direct use of Beacon during beforeunload succeeds

[mapx-]

@gorhill

This works also in the other way around.

1. Whitelist https://example.com/
2. search ddg for it https://duckduckgo.com/?q=https%3A%2F%2Fexample.com%2F&ia=web
3. there will be a direct link to example.com in wiki widget on the right
4. click on it
5. uBO does not block ddg beacon neither by ||duckduckgo.com/t/ or *$ping,3p because it's called from whitelisted example.com context.

See also: #2363

Still not blocked when navigating to whitelisted page.

Still blocked when navigating from whitelisted page.

Trying with:

||duckduckgo.com/t/$badfilter
*$ping,3p,badfilter
*$ping,strict3p

Because looks like *$ping,3p is correctly not applied now. On example.com beacon has duckduckgo.com context, so is 1p.

Similarly, with *$ping,1p.

[PinkDuck]

A key issue can be seen at CanIUse/keepalive in that some user agents (Firefox included) claim Fetch API full support, but actually a Fetch during beforeunload/unload does not have guaranteed network delivery and can be seen failing with NS_BINDING_ABORT. So I added user agent detection to compensate by using Beacon API directly, the first of these which does not get blocked, though the second ping does oddly.

[gorhill]

Still not blocked when navigating to whitelisted page.

I am aware, this is another issue in a different code path, not as easily fixed.

---

The difficulty with the trusted-site scenario is that disabling uBO is a test against the top-level context, and by the time the beacon is processed by uBO, the top-level context has changed.

---

This is the details I get re. ping/beacon requests (in chronological order) in Chromium/Firefox when navigating from duckduckgo.com to example.com, so this is the information uBO has to deal with:

Chromium

{
  "documentId": "E015EE64722A8657E9F4A36AEC96DD17",
  "documentLifecycle": "active",
  "frameId": 0,
  "frameType": "outermost_frame",
  "initiator": "https://duckduckgo.com",
  "method": "POST",
  "parentFrameId": -1,
  "requestId": "1363",
  "tabId": 207876891,
  "timeStamp": 1675004947732.364,
  "type": "ping",
  "url": "https://improving.duckduckgo.com/t/pae?2236082&q=https%3A%2F%2Fexample.com%2F&ttc=70065&ct=CA&d=d&serp_return=1&g=__&sm=server_fault:q:low&bkl=r1-8&blay=w30r1,e1&dsig=nlp_qa:m&be_eclsexp=b&be_msvrtexp=b&biaexp=b&cproxexp=b&eclsexp=b&euctaexp=b&infoboxexp=b&litexp=a&msvrtexp=b&mxmexp=a&videxp=a",
  "documentUrl": "https://duckduckgo.com"
}
{
  "documentId": "E015EE64722A8657E9F4A36AEC96DD17",
  "documentLifecycle": "active",
  "frameId": 0,
  "frameType": "outermost_frame",
  "initiator": "https://duckduckgo.com",
  "method": "POST",
  "parentFrameId": -1,
  "requestId": "1364",
  "tabId": 207876891,
  "timeStamp": 1675004947735.7732,
  "type": "ping",
  "url": "https://improving.duckduckgo.com/t/lc?5629835&t=d&ss=0&sp=0&osl=6&dm=example.com&hn=example.com&u=bingv7aa&nt=0&r=r8&da=0&rl=us-en&dl=en&oll=en:24,es:1,ko:1,pl:1,zh-CN:1&pr=http&i506=0&ivc=1&q=https%3A%2F%2Fexample.com%2F&ttc=70069&ct=CA&d=d&kl=wt-wt&kp=-1&serp_return=1&g=__&sm=server_fault:q:low&bkl=r1-8&blay=w30r1,e1&dsig=nlp_qa:m&be_eclsexp=b&be_msvrtexp=b&biaexp=b&cproxexp=b&eclsexp=b&euctaexp=b&infoboxexp=b&litexp=a&msvrtexp=b&mxmexp=a&videxp=a",
  "documentUrl": "https://duckduckgo.com"
}
{
  "documentLifecycle": "active",
  "frameId": 0,
  "frameType": "outermost_frame",
  "initiator": "https://duckduckgo.com",
  "method": "GET",
  "parentFrameId": -1,
  "requestId": "1366",
  "tabId": 207876891,
  "timeStamp": 1675004947749.509,
  "type": "main_frame",
  "url": "http://example.com/",
  "documentUrl": "https://duckduckgo.com"
}
{
  "documentId": "E015EE64722A8657E9F4A36AEC96DD17",
  "documentLifecycle": "cached",
  "frameId": 20,
  "frameType": "outermost_frame",
  "initiator": "https://duckduckgo.com",
  "method": "POST",
  "parentFrameId": -1,
  "requestId": "1367",
  "tabId": 207876891,
  "timeStamp": 1675004947771.949,
  "type": "ping",
  "url": "https://improving.duckduckgo.com/t/webvitals?8113009&FCP=175&TTFB=24&LCP=329&FID=29&CLS=0.1847&effectiveType=4g&has_performance=1&is_cached=0&navigation_type=back_forward&has_back_data=1&is_loaded_from_bfcache=0&is_bounce_back=1&g=__&sm=server_fault:q:low&bkl=r1-8&blay=w30r1,e1&dsig=nlp_qa:m&be_eclsexp=b&be_msvrtexp=b&biaexp=b&cproxexp=b&eclsexp=b&euctaexp=b&infoboxexp=b&litexp=a&msvrtexp=b&mxmexp=a&videxp=a",
  "documentUrl": "https://duckduckgo.com"
}

Firefox

 {
  "requestId": "282",
  "url": "https://improving.duckduckgo.com/t/pae?2455971&q=https%3A%2F…nt_single_place=a&factexp=a&infoboxexp=e&litexp=c&msvrtexp=b",
  "originUrl": "https://duckduckgo.com/?q=https%3A%2F%2Fexample.com%2F&ia=web",
  "documentUrl": "https://duckduckgo.com/?q=https%3A%2F%2Fexample.com%2F&ia=web",
  "method": "POST",
  "type": "beacon",
  "timeStamp": 1675005871720,
  "tabId": 6,
  "frameId": 0,
  "parentFrameId": -1,
  "incognito": false,
  "thirdParty": false,
  "cookieStoreId": "firefox-default",
  "proxyInfo": null,
  "ip": null,
  "frameAncestors": [],
  "urlClassification": {
    "firstParty": [],
    "thirdParty": []
  },
  "requestSize": 0,
  "responseSize": 0
}
{
  "requestId": "284",
  "url": "https://improving.duckduckgo.com/t/lc?6232704&t=d&ss=0&sp=0&…nt_single_place=a&factexp=a&infoboxexp=e&litexp=c&msvrtexp=b",
  "originUrl": "https://duckduckgo.com/?q=https%3A%2F%2Fexample.com%2F&ia=web",
  "documentUrl": "https://duckduckgo.com/?q=https%3A%2F%2Fexample.com%2F&ia=web",
  "method": "POST",
  "type": "beacon",
  "timeStamp": 1675005871733,
  "tabId": 6,
  "frameId": 0,
  "parentFrameId": -1,
  "incognito": false,
  "thirdParty": false,
  "cookieStoreId": "firefox-default",
  "proxyInfo": null,
  "ip": null,
  "frameAncestors": [],
  "urlClassification": {
    "firstParty": [],
    "thirdParty": []
  },
  "requestSize": 0,
  "responseSize": 0
}
{
  "requestId": "285",
  "url": "http://example.com/",
  "originUrl": "https://duckduckgo.com/?q=https%3A%2F%2Fexample.com%2F&ia=web",
  "method": "GET",
  "type": "main_frame",
  "timeStamp": 1675005871734,
  "tabId": 6,
  "frameId": 0,
  "parentFrameId": -1,
  "incognito": false,
  "thirdParty": false,
  "cookieStoreId": "firefox-default",
  "proxyInfo": null,
  "ip": null,
  "frameAncestors": [],
  "urlClassification": {
    "firstParty": [],
    "thirdParty": []
  },
  "requestSize": 0,
  "responseSize": 0
}
{
  "requestId": "286",
  "url": "https://improving.duckduckgo.com/t/webvitals?9334703&FCP=365…nt_single_place=a&factexp=a&infoboxexp=e&litexp=c&msvrtexp=b",
  "originUrl": "https://duckduckgo.com/?q=https%3A%2F%2Fexample.com%2F&ia=web",
  "documentUrl": "https://duckduckgo.com/?q=https%3A%2F%2Fexample.com%2F&ia=web",
  "method": "POST",
  "type": "beacon",
  "timeStamp": 1675005871755,
  "tabId": 6,
  "frameId": 0,
  "parentFrameId": -1,
  "incognito": false,
  "thirdParty": false,
  "cookieStoreId": "firefox-default",
  "proxyInfo": null,
  "ip": null,
  "frameAncestors": [],
  "urlClassification": {
    "firstParty": [],
    "thirdParty": []
  },
  "requestSize": 0,
  "responseSize": 0
}