tracking: HTTPS-Only mode

[Thorin-Oakenpants]

edit: no longer tracking/updating since May 2021

if you see HOM in this issue, it this stands for HTTPS-Only mode

I've gone through the bugzillas and picked the eyes out. Note that we block mixed active and passive by default, and there is also the upcoming insecure downloads pref: these probably all have a bearing on how this works

update: FF83+ the UI is exposed

• article: https://blog.mozilla.org/security/2020/11/17/firefox-83-introduces-https-only-mode/

🔻 meta

• 1613063 meta ticket

🔻 landed

• FF77: 1631384 loopback / local IP exceptions
• FF78: 1625156 error page
• FF79: 1640853 exception (session only via error page)
• FF80: 1647719 pb mode only pref
• FF80: 1644152 about:preferences UI
• FF80: 1644456 exceptions site permissions (basic PageInfo, Site Identity(?))
• FF81: 1651197 learn more link on error page
• FF81: 1642387 timing issues -> see 1660945 below
• FF82: 1660945 privacy stuff
• FF82: 1662359 subresources when exempted
• FF83: 1647829 FPI compat
• FF83: 1653026 give users a way to unbreak a page: site identity panel
• FF83: 1658921 shiny new dedicated error page
• FF83: 1665062 upgraded auth
• FF85: 1678614 trr.mode = 3
• FF86: 1662138 https-only and blocking mixed content downloads
• FF88: 1665057 tweak error page
• FF88: 1691888 break endless upgrade downgrade loops
• FF89: 1658924 try secure first
• FF90: 1662710 add preferences UI to manage exceptions
• INVALID 1669424 propagate exceptions to redirects (yikes?)
• WONTFIX 1661892 infinite redirects
• WONTFIX 1655329 IP addresses

🔻 of interest

• 1647033 redirects to subdomains
• 1650779 redirects
• 1683015 3XX and redirects
• 1686756 Insecure downloads taking > 3sec are zero-byte and cancelled - see comment 7
• 1709320 HSTS Response leads to error page

🔻 misc / open

• 1653898 www issues
• 1676227 consider some kind of clue that an upgrade has happened in the UI

🔻 visuals

• https://mozilla.invisionapp.com/share/R6YJ5KRP7ZM#/screens/429547333

[earthlng]

this can leak things

correct me if I'm wrong but AFAIK this leak is only really a leak if you use DoH+ESNI.
And even then, if a site is only available over HTTP and you end up adding an HOM exception or click the "go there anyway" button then this can not be considered a leak either.

As I understand it, the background request is also only triggered if a webserver doesn't respond to the HTTPS request within 3 seconds. I only mention this because in another issue @rusty-snake said that this "always leaks" and that's not true AFAIK.
Most servers will probably respond in under 3 seconds.

we should add this as false to the user.js

we can add it inactive, for people using DoH+ESNI.

[rusty-snake]

As I understand it, the background request is also only triggered if a webserver doesn't respond to the HTTPS request within 3 seconds. I only mention this because in another issue @rusty-snake said that this "always leaks" and that's not true AFAIK.
Most servers will probably respond in under 3 seconds.

It was here.

My understanding up to now was that to each server a http and a https connection attempt is made when opening a new site. But thanks to your push I checked this again, and yes it looks like it's only done after 3secs of no response.

https://phabricator.services.mozilla.com/D85300
https://searchfox.org/mozilla-central/source/dom/security/nsHTTPSOnlyUtils.cpp#25

correct me if I'm wrong but AFAIK this leak is only really a leak if you use DoH+ESNI.

The IP of the site is always leaked w/o a VPN. And the domain is leaked if you don't use DoH/DoT/… and w/o eSNI (which needs to be supported by your firefox and the server) it's leaked too.

As long as background-requets are enabled, a attacker (e.g. your ISP) could delay all your requests and then catch these http requsts. However, you would note if every site takes much longer.

As HOM now is on about:preferences I wouldn't consider it experimental anymore (as you said). And it works with FPI (but see below). We should add it as true to the user.js. It will upgrade a lot of site which does not have https by default and if you visit one of the ten-site w/o any https, it's just one click.

FWIW: If you set privacy.firstparty.isolate.use_site=true (and FPI), exceptions don't work. (I guess this is expected, since use_site makes FPI (and therefore site-permissions/exceptions) reflecting the scheme (i.e. http://example.com != https://example.com).

[earthlng]

Thanks for mentioning the problem with use_site, I wasn't aware of that and I doubt anyone at mozilla is either.
At least I haven't seen a ticket for that yet. AFAICT use_site is still in a very early experimental state and breakage is to be expected. But yeah, that's definitely something worth mentioning in our notes/warnings for the HOM pref.

[rusty-snake]

/* 1244: enable HTTPS-Only mode [FF76+]
 * When "https_only_mode" (all windows) is true, "https_only_mode_pbm" (private windows only) is ignored
 * [NOTE] If the last pref is true, firefox sends a http requests after 3 seconds of no response to this _domain_
 * to see if the site supports https. This avoids waiting for timeouts.
 * [SETTING] to add site exceptions: Page Info>Permissions>Use insecure HTTP (FF80+)
 * [SETTING] Privacy & Security>HTTPS-Only Mode
 * [1] https://bugzilla.mozilla.org/1613063 [META]
 * [2] https://bugzilla.mozilla.org/1647829 ***/
user_pref("dom.security.https_only_mode", true); // [FF76+]
   // user_pref("dom.security.https_only_mode_pbm", true); // [FF80+]
   // user_pref("dom.security.https_only_mode.upgrade_local", true); // [FF77+]
user_pref("dom.security.https_only_mode_send_http_background_request", false); // [FF82+]

Explain how it's not a leak. Do you mean because of DNS queries. i.e the background request only leaks the eTLD?

Since FF82 it only leaks the domain , but not the site. If http://example.com/ has no https, then http://example.com/spam/egg.html too. And the domain is leaked via DNS if no DoH/DoT is used. And w/o ESNI/ECH it's leaked too. So it's not a real leak if you leak anyway.

[annevk]

Could you file a bug on the potential bad interaction between dFPI and HTTPS-only? It's a bit hard to make out what the problem is without context.

[earthlng]

@Thorin-Oakenpants

FYI, these are 2 different prefs!

• privacy.firstparty.isolate.use_site controls use_site for FPI
• privacy.dynamic_firstparty.use_site controls use_site for dFPI

Did they change the timings?

yes they increased it to 3 seconds

[rusty-snake]

STR (used Nightly 85.0a1 (2020-11-18)):

Test sites:

1. http://speedofanimals.com/ - no https at all (only on www.speedofanimals.com) = timeout
2. http://neverssl.com/ - bad cert
3. http://http.badssl.com/ - working https + downgrade using 301 Moved Permanently = infinity loop
4. http://randomwalker.info/ no https at all = timeout
5. http://neverssl.org/ - bad cert

NOTE: You need to open the http url. If you open a https url, HOM is not activated and therefore
you see no warning regarding HOM and you can not add a exception.

Nothing changed (in Nightly this means network.cookie.cookieBehavior=5 and privacy.dynamic_firstparty.use_site=true):

• sites open as expected

Enable HOM (and sanitize cookies, cache, ...):

• see warning "Using HOM - add exception if you want"
• add exception
• site opens using http

Enable FPI (and sanitize):

• same

Enable FPI.use_site (and sanitize):

• see warning "Using HOM - add exception if you want"
• add exception
• see warning "Using HOM - add exception if you want"

EDIT: Summary: dFPI is not affected. Only HOM + FPI + FPI.use_site.

[rusty-snake]

/* 1244: enable HTTPS-Only mode [FF76+]
 * When "https_only_mode" (all windows) is true, "https_only_mode_pbm" (private windows only) is ignored
 * [SETTING] to add site exceptions: Page Info>Permissions>Use insecure HTTP (FF80+)
 * [SETTING] Privacy & Security>HTTPS-Only Mode
 * [1] https://bugzilla.mozilla.org/1613063 [META]
 * [2] https://bugzilla.mozilla.org/1647829 ***/
user_pref("dom.security.https_only_mode", true); // [FF76+]
   // user_pref("dom.security.https_only_mode_pbm", true); // [FF80+]
user_pref("dom.security.https_only_mode_send_http_background_request", false); // [FF82+]
/* 1245: Upgrade requests to local resource too [FF77+] ***/
   // user_pref("dom.security.https_only_mode.upgrade_local", true);
/* 1246: Disable http background requests [FF82+]
 * If a site is upgraded by HOM and the server didn't response with in 3 seconds, firefox sends a
 * http requests to the domain in order to see whether the server supports https or no.
 * This is done to avoid waiting for a timeout which takes 90 seconds.
 * [1] https://bugzilla.mozilla.org/1642387
 * [2] https://bugzilla.mozilla.org/1660945 ***/
user_pref("dom.security.https_only_mode_send_http_background_request", false);

[crssi]

I like http://neverssl.org/, because its random, avoiding cache "problems".

[rusty-snake]

@Thorin-Oakenpants #959

Edit: thoughts on testsites:

• http://neverssl.org/ is the only (knowed) site which avoid cache problems, see @crssi above.
• http://httpforever.com/ and http://http.badssl.com/ avoid layer-8 problems becaue they show the HOM-Warning no matter if you open them using http or https. neverssl.com and neverssl.org for example show a bad-cert error if you open them using https.

[xtsm]

So, is there no proper way to add exceptions/whitelists yet?
Because there is no "Use insecure HTTP" line in Permissions tab
(FF 86.0.1)

[xtsm]

It doesn't work with some sites that act like they have a HTTPS version but this version is actually a stub asking you to use HTTP instead (e.g. this one). I'm not sure what's the reasoning behind such setups.
Guess I'll wait for the UI, thanks.

[xtsm]

Yep, manual SQL insert fixes the issue for websdr.space and several other sites, thanks.

[Thorin-Oakenpants]

I'm done tracking this [in OP]: you're going to get weird edge cases until the interwebs get their shit together and HTTP is outlawed

[Thorin-Oakenpants]

@xtsm ... https://bugzilla.mozilla.org/show_bug.cgi?id=1662710 landed

[eleius]

I was wondering if HOM also upgrades insecure sub-requests, like HTTPS Everywhere does? (I'm looking for an alternative that is not list-basted... btw I thought I'd comment here to avoid spamming the issues section)

[rusty-snake]

I'm not sure if it upgrades sub-requests. Anyway in the worst case you end up with not loading sub-request due to mixed-content blocking.

user.js/user.js

Lines 692 to 697 in 00fa8f1

	/** MIXED CONTENT ***/
	/* 1240: enforce no insecure active content on https pages
	* [1] https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/21323 ***/
	user_pref("security.mixed_content.block_active_content", true); // [DEFAULT: true]
	/* 1241: disable insecure passive content (such as images) on https pages [SETUP-WEB] ***/
	user_pref("security.mixed_content.block_display_content", true);

[crssi]

When I have tested last time looong time ago, when HOM is enabled also mixed passive content is upgraded to HTTPS.
I haven't tested for mixed active content, since its blocked anyway (also as default).

But you can easily test your scenario at https://mixed.badssl.com/.

Cheers

[crssi]

Even if you set:

/* 1241  */ user_pref("security.mixed_content.block_display_content", false);

the 1241 setting is ignored when HOM enabled.

As you can see in page source the IMG is requested over HTTP, but network console clearly shows that IMG was downloaded over HTTPS

[eleius]

As you can see in page source the IMG is requested over HTTP, but network console clearly shows that IMG was downloaded over HTTPS

So it does upgrade mixed content sub-requests, and blocks them if downloading via https is not available?

[Thorin-Oakenpants]

HTTPS-First is the one that silently fails and allows insecure
HTTPS-Only mode (HoM) attempts to upgrade everything, and never allows insecure without intervention: it complains on top-level load. If top level is HTTPS it will silently fail sub-resources after upgrading - it literally does what it says HTTPS ONLY

So it does upgrade mixed content sub-requests, and blocks them if downloading via https is not available?

HoM upgrades everything: this comment says (emphasis mine), and chris makes a point of difference with HTTPS-First not upgrading subresources, being top-level loads only

If HTTPS-Only is enabled, all connections will be upgraded to https

There was also another comment Chris made some time after that, but I can't find it, in which I'm almost certain he did state it does upgrade all subresources (and silently fail)

So that's the intent, AFAIK. If you follow the meta ticket, there are still a few holes (not web content itself that I know of: it's edge cases likeTCP, HSTS) and teething issues here and there, and tickets for finishing building tests - I'm not really following it TBH.

HoM should be pretty robust. IDK if HTTPS-E in EASE Mode is it's equal - given that it's limited by WebExt

[eleius]

HTTPS-Only mode (HoM) attempts to upgrade everything, and never allows insecure without intervention: it complains on top-level load. If top level is HTTPS it will silently fail sub-resources after upgrading - it literally does what it says HTTPS ONLY

Excellent, thanks!

[Thorin-Oakenpants]

re-opening to remind me: we should add that info to the item in the user.js

[eleius]

Btw I'm unable to set exceptions, either from the padlock icon (on/off) or from the "manage exceptions" section in about:preferences#privacy, probably because of FPI as reported by @rusty-snake above. You might want to add this info too.

[Thorin-Oakenpants]

worksforme, also where I don't see where rusty snake-eyes g.i. joe said something about FPI, but yes, you need to make sure you exception has the right FPI syntax

can you give me the site that doesn't work for you?

---

edit: you add the exception via the padlock while you are on the interstitial BEFORE you click continue to HTTP Site ignore that

example: here's my only exception

• go to http://www.impawards.com/search.php
• turn off permanently
• enjoy
• 

[Thorin-Oakenpants]

hmmm, I just tried in Nightly and got this garbage (which doesn't work) - it's using the interstitial as the 1stparty by the looks of it

What version of FF are you on? 91? I wonder when it broke for FPI

[eleius]

also where I don't see where rusty snake-eyes g.i. joe said something about FPI

Last 3 lines in this comment

can you give me the site that doesn't work for you?

Any http website, e.g. libgen.lc - I add the exception (usually setting to "off" via padlock, I also tried via manage exceptions), shutdown Firefox, and the next time there is no exception listed in "manage exceptions".

Since your example (impawards) also met the same fate, I think one of my addons is deleting them for some reason. I'll have to do some tests...

[rusty-snake]

@eleius Either don't set privacy.firstparty.isolate.use_site=true or use dFPI.

Did you set privacy.clearOnShutdown.siteSettings=true?

[rusty-snake]

http://http.badssl.com/ -> "continue to HTTP Site" -> Padlock "Not Secure" -> HTTPS-Only mode -> Off

WFM with FPI=true FPI.use_site=false

[rusty-snake]

If you set a HOM exception from the Padlock, it uses the current site. If you are on about:httpsonlyerror it will use http://example.com^firstPartyDomain=about:httpsonlyerror (fake syntax) and if you have clicked on "continue to HTTP Site", it will use http://example.com^firstPartyDomain=example.com.

[eleius]

Did you set privacy.clearOnShutdown.siteSettings=true?

Ah yes. So https exceptions are counted as "site settings".

[rusty-snake]

Permissions are site settings.

[rusty-snake]

Here's the full list:

Ci.nsIClearDataService.CLEAR_PERMISSIONS |
  Ci.nsIClearDataService.CLEAR_CONTENT_PREFERENCES |
  Ci.nsIClearDataService.CLEAR_DOM_PUSH_NOTIFICATIONS |
  Ci.nsIClearDataService.CLEAR_SECURITY_SETTINGS |
  Ci.nsIClearDataService.CLEAR_CERT_EXCEPTIONS

https://searchfox.org/mozilla-central/source/browser/modules/Sanitizer.jsm#513-517

[Thorin-Oakenpants]

IDK what's going on. In FF85 or 86 or whatever or 89 or 90 it worked, for the impawards site, single upgrades worked with FPI, adding exceptions worked with FPI (e.g. edit in sqlite if needed). When the UI for exceptions came along, it still all worked. It still works in my FF91 main Firefox.

But now it all seems broken with FPI in nightly. Seriously, I am so tired of all this shit (not you guys, but the constantness of all this stuff), that honestly, I think I might just fuck off for a year .. I've almost had it

---

@rusty-snake

and if you have clicked on "continue to HTTP Site", it will use http://example.com^firstPartyDomain=example.com

FPI is enabled. Clicking continue to HTTP on the interstitial is also broken and just loops

If FPI is disabled, everything seems to work.

[eleius]

Ci.nsIClearDataService.CLEAR_CONTENT_PREFERENCES

Ok I'm setting it to false, I just hope "content preferences" (?) cannot be abused for tracking.

[Thorin-Oakenpants]

OK, so it was you clearing site preferences on close, then? right? thank fuck for that.

As for nightly, fuck it. I just tested in my main FF and everything works fine with FPI, so until breakage happens in stable, I do not care. Taking a year off, see you in 2022

[eleius]

Oh man, sorry about that...

[Thorin-Oakenpants]

Oh man, sorry about that...

not your fault. I assumed you wouldn't be deleting site exceptions (we don;t in the user.js) because that's rather extreme. And I just jumped to the most logical conclusion that it was one of those weird edge case sites: hence why I asked what it was

I just hope "content preferences" (?) cannot be abused for tracking.

"content preferences" ? If you mean site exceptions, then no. Exceptions are not global (they are literally exceptions) so can't be used to help track you, and exceptions are only things like allow/prompt/block things - do a control-I on a page and view permissions

[eleius]

I assumed you wouldn't be deleting site exceptions (we don;t in the user.js) because that's rather extreme.

But I shouldn't have posted here using a customized js, I totally forgot I had switched that one to true.

[Thorin-Oakenpants]

pants

edit: you add the exception via the padlock while you are on the interstitial BEFORE you click continue to HTTP Site ignore that

rusty

f you set a HOM exception from the Padlock, it uses the current site. If you are on about:httpsonlyerror it will use http://example.com^firstPartyDomain=about:httpsonlyerror (fake syntax) and if you have clicked on "continue to HTTP Site", it will use http://example.com^firstPartyDomain=example.com.

sorry rusty, I simply don't use the interface or add exceptions (except the one I have had for ages) - so my method was a little RUSTY (sorry). We better add that: edit bit pointless listing off temporarily then isn't it, in order to use it you already set that (talk about ass about face)

 * [SETTING] to add site exceptions: Padlock>HTTPS-Only mode>On <-- when on the site u MOFOs
 * [SETTING] Privacy & Security>HTTPS-Only Mode

[Thorin-Oakenpants]

OK, how is this

@eleius @rusty-snake

/* 1244: enable HTTPS-Only mode in all windows [FF76+]
 * If the top-level is HTTPS, then insecure subresources are also upgraded, failing silently
 * [SETTING] to add site exceptions: Padlock>HTTPS-Only mode>On <- when on the site
 * [SETTING] Privacy & Security>HTTPS-Only Mode
 * [TEST] http://example.com [upgrade]  <--- RUSTY FIND ME SOMETHING ELSE
 * [TEST] http://neverssl.org/ [no upgrade]
 * [1] https://bugzilla.mozilla.org/1613063 [META] ***/
user_pref("dom.security.https_only_mode", true); // [FF76+]
   // user_pref("dom.security.https_only_mode_pbm", true); // [FF80+]

[rusty-snake]

/* 1244: enable HTTPS-Only mode in all windows [FF76+]
 * If the top-level is HTTPS, then insecure subresources are also upgraded, failing silently
 * [SETTING] to add site exceptions: Padlock>HTTPS-Only mode>On (After "Continue to HTTP Site")
 * [SETTING] to add site exceptions: click "Continue to HTTP Site", Padlock>HTTPS-Only mode>On
 * [SETTING] Privacy & Security>HTTPS-Only Mode
 * [TEST] http://example.com [upgrade]  <--- RUSTY FIND ME SOMETHING ELSE; Actually every site with ~correct configured~ _working_ https, _idealy without 301 upgrade_. However if it is in https://hg.mozilla.org/mozilla-central/raw-file/tip/security/manager/ssl/nsSTSPreloadList.inc, it will be upgrade regardless of HOM. Will look for a good one.
 * [TEST] http://neverssl.com/ [no upgrade]  <--- .org is dead
 * [1] https://bugzilla.mozilla.org/1613063 [META] ***/
user_pref("dom.security.https_only_mode", true); // [FF76+]
   // user_pref("dom.security.https_only_mode_pbm", true); // [FF80+]

[Thorin-Oakenpants]

shit that was meant to be the neverssl one

• @rusty-snake ^^ - what about something in tracking: HTTPS-Only mode #1047 (comment)

[Thorin-Oakenpants]

wait a second ... http://neverssl.com/ is working - #1235 (comment)

so I guess we don't need to worry about that now