Firefox 1548973 Hotfix (addons disabled due to expired intermediate certificate) #60916

deliciouslytyped · 2019-05-04T14:42:03Z

Installing the hotfix:

I didn't want to turn on user studies and it didn't seem to be pulling the hotfix immediately anyway, so I did some digging and found the hotfix installer at https://normandy.cdn.mozilla.net/api/v1/recipe/.

Search for "name": "Hotfix: Update XPI signing intermediate [Bug 1548973]", and then install the xpi from the addonUrl field. I don't know how supported this method is but it worked for me. I cannot guarantee user support, so if you can't figure things out yourself/don't consider yourself a power user it's probably a bad idea to rely on this That said, strictly on topic issues (this isn't really a mozilla issue tracker) will be handled on a best effort basis.

Example url: https:// storage.googleapis.com /moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate@mozilla.com-1.0.2-signed.xpi

Edit: see later in thread for some vague clarification from Mozilla about community workarounds, which FWIW is more or less a reasonable under the circumstances. Likely to attempt to reduce support load. TL;DR: caveat emptor. (nothing has changed)

Edit2: Fixed typo: addonIUrl -> addonUrl

Edit3: For those coming to this issue from other sources: Nixpkgs currently runs firefox 66. For those on an older version where the normandy xpi does not work (unsure if user error, or version incompatibility - have not tested it myself) a more involved workaround that is mentioned in the issue tracker is to extract the certificate from the XPI and manually import it into the certificate store. E.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1549078#c71

The text was updated successfully, but these errors were encountered:

deliciouslytyped · 2019-05-04T14:43:19Z

The issue can be followed at https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 (see also various other links hanging off this as "Depends On" and "Blocks", linked in the comments, etc )
->

alexxbb · 2019-05-04T17:12:07Z

Thank you so much! I've been looking for this!

The-SamminAter · 2019-05-04T21:35:42Z

Thanks, it worked!

LYY · 2019-05-04T23:29:04Z

It worked for me, Thanks!

deliciouslytyped · 2019-05-04T23:38:57Z

~~Slight~~ clarification of support status https://bugzilla.mozilla.org/show_bug.cgi?id=1549078#c36

Edit: correction: https://bugzilla.mozilla.org/show_bug.cgi?id=1549061 is linked somewhere, and so is https://blog.mozilla.org/addons/2019/05/04/update-regarding-add-ons-in-firefox/

The latter says: "There are a number of work-arounds being discussed in the community. These are not recommended as they may conflict with fixes we are deploying. We’ll let you know when further updates are available that we recommend, and appreciate your patience. (May 4, 15:01 EST)"

Therefore my suggestion is to periodically check the issues for additional information, caveat emptor, etc etc.

TwilightFlower · 2019-05-04T23:44:37Z

THANK YOU. the way Mozilla has been handling this is stupid. this exists, why not give people the link instead of asking them to browse unprotected for SIX FUCKING HOURS

gak1234 · 2019-05-05T01:21:05Z

Almost 18 hours unpatched, here. I couldn't enable studies - toggling the checkbox made NO change to prefs.js so studies are vaporware on my install for some reason. Just give me the stinkin' patch, Mozilla.

ghost · 2019-05-05T06:16:19Z

Installing the hotfix:

I didn't want to turn on user studies and it didn't seem to be pulling the hotfix immediately anyway, so I did some digging and found the hotfix installer at https://normandy.cdn.mozilla.net/api/v1/recipe/.

Search for "name": "Hotfix: Update XPI signing intermediate [Bug 1548973]", and then install the xpi from the addonIUrl field.

* addonUrl field

deliciouslytyped · 2019-05-05T09:39:43Z

Fixed typo.

deliciouslytyped · 2019-05-05T09:54:19Z

Added edit mentioning a second workaround method that may or may not work if the XPI doesn't work.

captainepoch · 2019-05-05T09:54:41Z

How can I uninstall this once Mozilla fixed the issue? I don't see anything at about:addons or about:studies...

deliciouslytyped · 2019-05-05T09:56:45Z

Oh, that's a good question. I just assumed it would show up in about:addons - but it makes some sense that it doesn't. I'll see if I can find anything since I will need that myself.

captainepoch · 2019-05-05T09:57:39Z

Oh, that's a good question. I just assumed it would show up in about:addons. I'll see if I can find anything since I will need that myself.

For clarification, I have Studies and Telemetry disabled in my profile

deliciouslytyped · 2019-05-05T10:00:26Z

The addon seems to be listed in about:debugging but that does not seem to provide a removal interface.

captainepoch · 2019-05-05T10:02:30Z

The addon seems to be listed in about:debugging but that does not seem to provide a removal interface.

I deleted the .xpi from the profile directory and it seems to remove the add-on, but I'm not sure if it's removed completly or not... And it seems to not store data, as far as I can see.

Edit 1:
As far as I can tell, once you run the .xpi file you can delete from your profile, everything seems to keep working.

Also, I downloaded the .xpi and unzip it. I see it injects a cert and then forces a re-verification of the add-ons' signature.

The file api.js has the data, so... Copy and paste the file content here, maybe someone with more experience can tell more about this:

/* eslint no-unused-vars: ["error", { "varsIgnorePattern": "skeleton" }]*/
ChromeUtils.defineModuleGetter(this, "XPIDatabase", "resource://gre/modules/addons/XPIDatabase.jsm");

var skeleton = class extends ExtensionAPI {
  getAPI(/* context */) {
    return {
      experiments: {
        skeleton: {
          async doTheThing() {
            // first inject the new cert
            try {
              let intermediate = "MIIHLTCCBRWgAwIBAgIDEAAIMA0GCSqGSIb3DQEBDAUAMH0xCzAJBgNVBAYTAlVTMRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMS8wLQYDVQQLEyZNb3ppbGxhIEFNTyBQcm9kdWN0aW9uIFNpZ25pbmcgU2VydmljZTEfMB0GA1UEAxMWcm9vdC1jYS1wcm9kdWN0aW9uLWFtbzAeFw0xNTA0MDQwMDAwMDBaFw0yNTA0MDQwMDAwMDBaMIGnMQswCQYDVQQGEwJVUzEcMBoGA1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjEvMC0GA1UECxMmTW96aWxsYSBBTU8gUHJvZHVjdGlvbiBTaWduaW5nIFNlcnZpY2UxJjAkBgNVBAMTHXNpZ25pbmdjYTEuYWRkb25zLm1vemlsbGEub3JnMSEwHwYJKoZIhvcNAQkBFhJmb3hzZWNAbW96aWxsYS5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC/qluiiI+wO6qGA4vH7cHvWvXpdju9JnvbwnrbYmxhtUpfS68LbdjGGtv7RP6F1XhHT4MU3v4GuMulH0E4Wfalm8evsb3tBJRMJPICJX5UCLi6VJ6J2vipXSWBf8xbcOB+PY5Kk6L+EZiWaepiM23CdaZjNOJCAB6wFHlGe+zUk87whpLa7GrtrHjTb8u9TSS+mwjhvgfP8ILZrWhzb5H/ybgmD7jYaJGIDY/WDmq1gVe03fShxD09Ml1P7H38o5kbFLnbbqpqC6n8SfUI31MiJAXAN2e6rAOM8EmocAY0EC5KUooXKRsYvHzhwwHkwIbbe6QpTUlIqvw1MPlQPs7Zu/MBnVmyGTSqJxtYoklr0MaEXnJNY3g3FDf1R0Opp2/BEY9Vh3Fc9Pq6qWIhGoMyWdueoSYa+GURqDbsuYnk7ZkysxK+yRoFJu4x3TUBmMKM14jQKLgxvuIzWVn6qg6cw7ye/DYNufc+DSPSTSakSsWJ9IPxiAU7xJ+GCMzaZ10Y3VGOybGLuPxDlSd6KALAoMcl9ghB2mvfB0N3wv6uWnbKuxihq/qDps+FjliNvr7C66mIVH+9rkyHIy6GgIUlwr7E88Qqw+SQeNeph6NIY85PL4p0Y8KivKP4J928tpp18wLuHNbIG+YaUk5WUDZ6/2621pi19UZQ8iiHxN/XKQIDAQABo4IBiTCCAYUwDAYDVR0TBAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwFgYDVR0lAQH/BAwwCgYIKwYBBQUHAwMwHQYDVR0OBBYEFBY++xz/DCuT+JsV1y2jwuZ4YdztMIGoBgNVHSMEgaAwgZ2AFLO86lh0q+FueCqyq5wjHqhjLJe3oYGBpH8wfTELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xLzAtBgNVBAsTJk1vemlsbGEgQU1PIFByb2R1Y3Rpb24gU2lnbmluZyBTZXJ2aWNlMR8wHQYDVQQDExZyb290LWNhLXByb2R1Y3Rpb24tYW1vggEBMDMGCWCGSAGG+EIBBAQmFiRodHRwOi8vYWRkb25zLm1vemlsbGEub3JnL2NhL2NybC5wZW0wTgYDVR0eBEcwRaFDMCCCHi5jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzAfgh1jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzANBgkqhkiG9w0BAQwFAAOCAgEAX1PNli/zErw3tK3S9Bv803RV4tHkrMa5xztxzlWja0VAUJKEQx7f1yM8vmcQJ9g5RE8WFc43IePwzbAoum5F4BTM7tqM//+e476F1YUgB7SnkDTVpBOnV5vRLz1Si4iJ/U0HUvMUvNJEweXvKg/DNbXuCreSvTEAawmRIxqNYoaigQD8x4hCzGcVtIi5Xk2aMCJW2K/6JqkN50pnLBNkPx6FeiYMJCP8z0FIz3fv53FHgu3oeDhi2u3VdONjK3aaFWTlKNiGeDU0/lr0suWfQLsNyphTMbYKyTqQYHxXYJno9PuNi7e1903PvM47fKB5bFmSLyzB1hB1YIVLj0/YqD4nz3lADDB91gMBB7vR2h5bRjFqLOxuOutNNcNRnv7UPqtVCtLF2jVb4/AmdJU78jpfDs+BgY/t2bnGBVFBuwqS2Kult/2kth4YMrL5DrURIM8oXWVQRBKxzr843yDmHo8+2rqxLnZcmWoe8yQ41srZ4IB+V3w2TIAd4gxZAB0Xa6KfnR4D8RgE5sgmgQoK7Y/hdvd9Ahu0WEZI8Eg+mDeCeojWcyjF+dt6c2oERiTmFTIFUoojEjJwLyIqHKt+eApEYpF7imaWcumFN1jR+iUjE4ZSUoVxGtZ/Jdnkf8VVQMhiBA+i7r5PsfrHq+lqTTGOg+GzYx7OmoeJAT0zo4c=";
              let certDB = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB);
              certDB.addCertFromBase64(intermediate, ",,");
              console.log("new intermediate certificate added");
            } catch (e) {
              console.error("failed to add new intermediate certificate:", e);
            }

            // Second, force a re-verify of signatures
            try {
              XPIDatabase.verifySignatures();
              console.log("signatures re-verified");
            } catch (e) {
              console.error("failed to re-verify signatures:", e);
            }
          }
        }
      }
    };
  }
};

wideocean · 2019-05-05T10:04:46Z

+1
This "fix" described here works for me too.
However, there is no obvious way to uninstall it?

I guess that when the official patch rolls out, we can uninstall Firefox completely and just reinstall it, right?

deliciouslytyped · 2019-05-05T10:37:41Z

Ok so I couldn't find the original source code for the XPI, would be nice if someone could post where to find it, but it's basically readable if you extract the xpi. I don't know what the content.js does, otherwise this is the code:

/* eslint no-unused-vars: ["error", { "varsIgnorePattern": "skeleton" }]*/
ChromeUtils.defineModuleGetter(this, "XPIDatabase", "resource://gre/modules/addons/XPIDatabase.jsm");

var skeleton = class extends ExtensionAPI {
  getAPI(/* context */) {
    return {
      experiments: {
        skeleton: {
          async doTheThing() {
            // first inject the new cert
            try {
              let intermediate = *omitted*
              let certDB = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB);
              certDB.addCertFromBase64(intermediate, ",,");
              console.log("new intermediate certificate added");
            } catch (e) {
              console.error("failed to add new intermediate certificate:", e);
            }

            // Second, force a re-verify of signatures
            try {
              XPIDatabase.verifySignatures();
              console.log("signatures re-verified");
            } catch (e) {
              console.error("failed to re-verify signatures:", e);
            }
          }
        }
      }
    };
  }
};

Some similar patches can be found linked from https://bugzilla.mozilla.org/show_bug.cgi?id=1549061

Basically, I guess it looks safe to just delete the addon as stated by @captainepoch but I'm not a firefox dev so YMMV.

Edit: Oops, some duplicated work. :)

deliciouslytyped · 2019-05-05T10:41:00Z

Specifically, relevant discussion can be read at https://phabricator.services.mozilla.com/D29940 , I have not read through it.

deliciouslytyped · 2019-05-05T10:44:43Z

https://news.ycombinator.com/item?id=19828631 links https://news.ycombinator.com/item?id=19827428 links http://kb.mozillazine.org/Uninstalling_extensions#Uninstalling_manually suggests the manual deletion method is fine. Who knows how up to date that article is though.

brunoais · 2019-05-05T12:13:48Z

@captainepoch
let intermediate = that's a certificate encoded in base64.
let certDB = This is internal API. In this case, it is loading the module that handles certificates.
certDB.addCertFromBase64( Adding the base64 encoded certificate to the browser's certificates.
XPIDatabase.verifySignatures(); Regenerating the certificates cache.

You can see both comments. They explain what is being done in less detail.

Jake5319 · 2019-05-05T14:28:05Z

The googleapi link to intermediate hotfix as in OP, pops window "normandy.cdn.mozilla.net Firefox prevented this site from asking you to install software on your computer"

Then if I resend url I get another popup "The add-on could not be downloaded because of a connection failure"

Any ideas?

Edit: here is complete url with ++ h++ps://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate@mozilla.com-1.0.2-signed.xpi

browser console shows 1557070136986 addons.xpi WARN Download of https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi failed: [Exception... "Certificate issuer is not built-in." nsresult: "0x80004004 (NS_ERROR_ABORT)" location: "JS frame :: resource://gre/modules/CertUtils.jsm :: checkCert :: line 163" data: no] Stack trace: checkCert()@resource://gre/modules/CertUtils.jsm:163
onStopRequest()@resource://gre/modules/addons/XPIInstall.jsm:2078

deliciouslytyped · 2019-05-05T15:02:00Z

You must have something different with your configuration and/or have something causing the site to be untrusted. I don't know, sorry. There seem to be ways to get past the prevention of installation from untrusted sites but I don't feel comfortable recommending that. You need to figure out why your Firefox doesn't trust the path.

Edit: there seem to be other people on the internet with your issue.

I guess you could download the XPI manually with curl or wget or something (right click -> save link as?), I'm just looking at your browser very suspiciously. This is basically bypassing a check it's doing and is unhappy about for some reason.

Edit 2: As long as you haven't disabled signature verification I guess it's fine.

deliciouslytyped · 2019-05-05T18:14:24Z

It looks like firefox doesn't like the certificate the addon is signed with for some reason?

Jake5319 · 2019-05-05T23:08:15Z

Thank you stefano-m!!!

pauljt · 2019-05-05T23:23:47Z

Just in case it helps future visitors, the fix (same solution as the hotfix) has already shipped now (about 3 hours ago in 66.0.4). See release notes for details and known outstanding issues: https://www.mozilla.org/en-US/firefox/66.0.4/releasenotes/

vcunat · 2019-05-06T07:31:53Z

The master and 19.03 branches have Firefox 66.0.4 now, so I assume hotfixes aren't needed anymore.

camy011 · 2019-05-06T21:39:41Z

Just wanted to throw this fix out there since I am on an older version of Firefox 56.0.2 which does not currently have a fix. You can manually import the certificate found in the xpi. To do this you will want to download the xpi from the below link possibly with another browser like IE if you ware having trouble.
https:// storage.googleapis.com /moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate@mozilla.com-1.0.2-signed.xpi
Rename as .zip or open with a zip program.
Navigate to experiments\skeleton and open api.js in a text editor.
Copy everything inside quotes after let intermediate = " until the next "
Create a new text file in notepad and add the certificate header/trailer like so and paste the value in between.

Rename the file extension to .crt
In Firefox navigate to Options -> Privacy & Security -> Certificates -> View Certificates -> Authorities -> Import...
It will show up under Mozilla Corporation as signingca1.addons.mozilla.org

vcunat · 2019-05-07T06:44:01Z

56 surprises me. I'd expect 52 ESR to be more likely (we even still have it in latest nixpkgs), but both are unmaintained upstream and thus unlikely to receive official fixes.

brunoais · 2019-05-07T08:44:46Z

@camy011 Quite curious here.
What things prevent you from using the latest firefox? Is it some extension that is still not ported to webextension and no good alternative?

vcunat · 2019-05-07T09:35:27Z

@brunoais: I remember threads about this: here and perhaps here.

brunoais · 2019-05-07T12:08:07Z

@vcunat That's what I needed. Thanks. Makes sense.

camy011 · 2019-05-07T16:06:12Z

@camy011 Quite curious here.
What things prevent you from using the latest firefox? Is it some extension that is still not ported to webextension and no good alternative?

Originally I didn't upgrade because I didn't want my addons disabled. The goal being to allow enough time for addons to update and alternatives to become available. I actually went through and installed the latest FF side-by-side yesterday and got everything working to the point that I don't need my old copy.

Some addons that don't have an alternative that I was using though:
Classic theme restorer
Menu Wizard (hide unused/reorder menu options)
Roomy bookmarks toolbar

Most of what I wanted from these addons can be done using the userChrome.css so it doesn't matter too much. Tabs on bottom, moving "Select All" and "Copy" to the top of the right click menu, hiding context menu options (if I care enough to get around to it), hiding text for bookmarks to fit more on the bar.

I also used classic theme restorer to move my file menu to my bookmarks bar to help compact the browser. While there isn't an easy way to do that it looks like I can move my bookmarks toolbar to my file menu bar which gives almost the same effect (top vs middle row).

hargobind · 2019-05-08T08:01:39Z

In pre-Quantum FF (versions 56, 55, etc.), I tried following some of the instructions in this thread, but didn't have success with extensions working again.

Finally I found this thread on Reddit which basically goes through the same process mentioned by @camy011 about extracting and installing the certificate, but it also includes the last important step about running a command needed to re-verify the signatures of all add-ons.

vcunat · 2019-05-09T11:28:15Z

It appears even fix for 52 ESR is planned, though I still can't see a new 52.x release on the download site.

For users who cannot update to the latest version of Firefox or Firefox ESR, we plan to distribute an update that automatically applies the fix to versions 52 through 60.

stefano-m mentioned this issue May 5, 2019

Please update to Firefox 66.0.4 (master and release branches) to fix broken addons #61005

Closed

vcunat closed this as completed May 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Firefox 1548973 Hotfix (addons disabled due to expired intermediate certificate) #60916

Firefox 1548973 Hotfix (addons disabled due to expired intermediate certificate) #60916

deliciouslytyped commented May 4, 2019 •

edited

Loading

deliciouslytyped commented May 4, 2019 •

edited

Loading

alexxbb commented May 4, 2019

The-SamminAter commented May 4, 2019

LYY commented May 4, 2019

deliciouslytyped commented May 4, 2019 •

edited

Loading

TwilightFlower commented May 4, 2019

gak1234 commented May 5, 2019

ghost commented May 5, 2019 •

edited by ghost

Loading

deliciouslytyped commented May 5, 2019

deliciouslytyped commented May 5, 2019

captainepoch commented May 5, 2019 •

edited

Loading

deliciouslytyped commented May 5, 2019 •

edited

Loading

captainepoch commented May 5, 2019

deliciouslytyped commented May 5, 2019

captainepoch commented May 5, 2019 •

edited

Loading

wideocean commented May 5, 2019

deliciouslytyped commented May 5, 2019 •

edited

Loading

deliciouslytyped commented May 5, 2019

deliciouslytyped commented May 5, 2019

brunoais commented May 5, 2019

Jake5319 commented May 5, 2019 •

edited

Loading

deliciouslytyped commented May 5, 2019 •

edited

Loading

deliciouslytyped commented May 5, 2019

Jake5319 commented May 5, 2019

pauljt commented May 5, 2019

vcunat commented May 6, 2019

camy011 commented May 6, 2019

vcunat commented May 7, 2019

brunoais commented May 7, 2019

vcunat commented May 7, 2019

brunoais commented May 7, 2019

camy011 commented May 7, 2019

hargobind commented May 8, 2019

vcunat commented May 9, 2019

Firefox 1548973 Hotfix (addons disabled due to expired intermediate certificate) #60916

Firefox 1548973 Hotfix (addons disabled due to expired intermediate certificate) #60916

Comments

deliciouslytyped commented May 4, 2019 • edited Loading

deliciouslytyped commented May 4, 2019 • edited Loading

alexxbb commented May 4, 2019

The-SamminAter commented May 4, 2019

LYY commented May 4, 2019

deliciouslytyped commented May 4, 2019 • edited Loading

TwilightFlower commented May 4, 2019

gak1234 commented May 5, 2019

ghost commented May 5, 2019 • edited by ghost Loading

deliciouslytyped commented May 5, 2019

deliciouslytyped commented May 5, 2019

captainepoch commented May 5, 2019 • edited Loading

deliciouslytyped commented May 5, 2019 • edited Loading

captainepoch commented May 5, 2019

deliciouslytyped commented May 5, 2019

captainepoch commented May 5, 2019 • edited Loading

wideocean commented May 5, 2019

deliciouslytyped commented May 5, 2019 • edited Loading

deliciouslytyped commented May 5, 2019

deliciouslytyped commented May 5, 2019

brunoais commented May 5, 2019

Jake5319 commented May 5, 2019 • edited Loading

deliciouslytyped commented May 5, 2019 • edited Loading

deliciouslytyped commented May 5, 2019

Jake5319 commented May 5, 2019

pauljt commented May 5, 2019

vcunat commented May 6, 2019

camy011 commented May 6, 2019

vcunat commented May 7, 2019

brunoais commented May 7, 2019

vcunat commented May 7, 2019

brunoais commented May 7, 2019

camy011 commented May 7, 2019

hargobind commented May 8, 2019

vcunat commented May 9, 2019

deliciouslytyped commented May 4, 2019 •

edited

Loading

deliciouslytyped commented May 4, 2019 •

edited

Loading

deliciouslytyped commented May 4, 2019 •

edited

Loading

ghost commented May 5, 2019 •

edited by ghost

Loading

captainepoch commented May 5, 2019 •

edited

Loading

deliciouslytyped commented May 5, 2019 •

edited

Loading

captainepoch commented May 5, 2019 •

edited

Loading

deliciouslytyped commented May 5, 2019 •

edited

Loading

Jake5319 commented May 5, 2019 •

edited

Loading

deliciouslytyped commented May 5, 2019 •

edited

Loading