Audit stable rulesets that cause Mixed Content Blocking

[pde]

I just ran the updated MCB test scripts on the stable ruleset library. Below are the domains that triggered MCB. We should check which ones actually break things. Coming down the pipeline I'll also have detection for cert breakage, and a run of these tests on the master branch. @2d1 making sure you see this.

MCB triggered: http://immunityinc.com (defined)
MCB triggered: http://www.immunityinc.com (defined)
MCB triggered: http://sitemasonmail.com.moses.com (defined)
MCB triggered: http://sitemasonmail.com (defined)
MCB triggered: http://karwansaraypublishers.com (defined)
MCB triggered: http://waffles.fm (defined)
MCB triggered: http://specialforces.com (defined)
MCB triggered: http://www.specialforces.com (defined)
MCB triggered: http://emsisoft.com (defined)
MCB triggered: http://eiseverywhere.com (defined)
MCB triggered: http://www.eiseverywhere.com (defined)
MCB triggered: http://jabber.ru (defined)
MCB triggered: http://www.amway.com (defined)
MCB triggered: http://bluehost.com (defined)
MCB triggered: http://123rf.com (defined)
MCB triggered: http://hostmonster.com (defined)
MCB triggered: http://nationalarchives.gov.uk (defined)
MCB triggered: http://www.nationalarchives.gov.uk (defined)
MCB triggered: http://www.globaltestsupply.com (defined)
MCB triggered: http://globaltestsupply.com (defined)
MCB triggered: http://chronicle.com (defined)
MCB triggered: http://www.esrb.org (defined)
MCB triggered: http://clkads.com (defined)
MCB triggered: http://www.qca.qualcomm.com (defined)
MCB triggered: http://www.e-rewards.com (defined)
MCB triggered: http://opticsinfobase.org (defined)
MCB triggered: http://digitalforensicsmagazine.com (defined)
MCB triggered: http://www.digitalforensicsmagazine.com (defined)
MCB triggered: http://www.ning.com (defined)
MCB triggered: http://www.hubspot.com (defined)
MCB triggered: http://www.hubspot.net (defined)
MCB triggered: http://guardianeatright.co.uk (defined)
MCB triggered: http://www.guardianeatright.co.uk (defined)
MCB triggered: http://dynamitedata.com (defined)
MCB triggered: http://www.dynamitedata.com (defined)
MCB triggered: http://citizen.org (defined)
MCB triggered: http://action.citizen.org (defined)
MCB triggered: http://www.dnsexit.com (defined)
MCB triggered: http://cloudaccess.net (defined)
MCB triggered: http://cdn.exm.nr (defined)
MCB triggered: http://www.sitemeter.com (defined)
MCB triggered: http://lambda-tek.com (defined)
MCB triggered: http://www.egnyte.com (defined)
MCB triggered: http://benaughty.com (defined)
MCB triggered: http://cupid.com (defined)
MCB triggered: http://www.lef.org (defined)
MCB triggered: http://lef.org (defined)
MCB triggered: http://apwu.org (defined)
MCB triggered: http://www.apwu.org (defined)
MCB triggered: http://mediamarkt.se (defined)
MCB triggered: http://www.mediamarkt.se (defined)
MCB triggered: http://www.aspectsecurity.com (defined)
MCB triggered: http://www.nextag.ca (defined)
MCB triggered: http://www.nextag.com (defined)
MCB triggered: http://www.nextag.de (defined)
MCB triggered: http://www.nextag.fr (defined)
MCB triggered: http://www.nextag.it (defined)
MCB triggered: http://www.static-nextag.com (defined)
MCB triggered: http://appworld.blackberry.com (defined)
MCB triggered: http://www.abuse.ch (defined)
MCB triggered: http://i1.mbsvr.net (defined)
MCB triggered: http://dmu.ac.uk (defined)
MCB triggered: http://www.dmu.ac.uk (defined)
MCB triggered: http://creativecommons.org (defined)
MCB triggered: http://marketwatch.com (defined)
MCB triggered: http://www.marketwatch.com (defined)
MCB triggered: http://startlogic.com (defined)
MCB triggered: http://cs.joensuu.fi (defined)
MCB triggered: http://ehow.com (defined)
MCB triggered: http://ic3.gov (defined)
MCB triggered: http://www.ic3.gov (defined)
MCB triggered: http://datapipe.com (defined)
MCB triggered: http://www.datapipe.com (defined)
MCB triggered: http://datapipe.net (defined)
MCB triggered: http://www.brainyquote.com (defined)
MCB triggered: http://jusek.se (defined)
MCB triggered: http://cbb.dk (defined)
MCB triggered: http://sapo.pt (defined)
MCB triggered: http://nicotine-anonymous.org (defined)
MCB triggered: http://www.nicotine-anonymous.org (defined)
MCB triggered: http://fastwebhost.com (defined)
MCB triggered: http://support.fastwebhost.com (defined)
MCB triggered: http://www.fastwebhost.com (defined)
MCB triggered: http://9seeds.com (defined)
MCB triggered: http://purecars.com (defined)
MCB triggered: http://unitedsafcu.org (defined)
MCB triggered: http://justhost.com (defined)
MCB triggered: http://mychatagent.com (defined)
MCB triggered: http://www.cpj.org (defined)
MCB triggered: http://nuigalway.ie (defined)
MCB triggered: http://webhostingtalk.com (defined)
MCB triggered: http://www.samsung.cn (defined)
MCB triggered: http://samsung.com (defined)
MCB triggered: http://www.samsung.com.cn (defined)
MCB triggered: http://console.ubertags.com (defined)
MCB triggered: http://miun.se (defined)
MCB triggered: http://milkandmore.co.uk (defined)
MCB triggered: http://apoteket.se (defined)
MCB triggered: http://verizonwireless.com (defined)
MCB triggered: http://www.verizonwireless.com (defined)
MCB triggered: http://vzw.com (defined)
MCB triggered: http://www.hu.liu.se (defined)
MCB triggered: http://www.imh.liu.se (defined)
MCB triggered: http://www.sigmabeauty.com (defined)
MCB triggered: http://mycanvas.com (defined)
MCB triggered: http://inetinteractive.com (defined)
MCB triggered: http://www.inetinteractive.com (defined)
MCB triggered: http://www.leahy.senate.gov (defined)
MCB triggered: http://uni-muenchen.de (defined)
MCB triggered: http://owncube.com (defined)
MCB triggered: http://www.btplc.com (defined)
MCB triggered: http://www.anpost.ie (defined)
MCB triggered: http://anpost.ie (defined)
MCB triggered: http://pass-web.ridemetro.org (defined)
MCB triggered: http://yemeksepeti.com (defined)
MCB triggered: http://www.yemeksepeti.com (defined)
MCB triggered: http://gsfacket.se (defined)
MCB triggered: http://staticstuff.net (defined)
MCB triggered: http://sbb.ch (defined)
MCB triggered: http://sverigesradio.se (defined)
MCB triggered: http://sr.se (defined)
MCB triggered: http://fusionio.com (defined)
MCB triggered: http://www.fusionio.com (defined)
MCB triggered: http://clickbank.com (defined)
MCB triggered: http://www.clickbank.com (defined)
MCB triggered: http://www.clickbank.net (defined)
MCB triggered: http://static.which.net (defined)
MCB triggered: http://www2.youm7.com (defined)
MCB triggered: http://marketfoolery.com (defined)
MCB triggered: http://www.imf.org (defined)
MCB triggered: http://imf.org (defined)
MCB triggered: http://nrc-cnrc.gc.ca (defined)
MCB triggered: http://lunarmods.com (defined)
MCB triggered: http://www.lunarpages.com (defined)
MCB triggered: http://www.lunarpages.com.mx (defined)
MCB triggered: http://tmz.com (defined)
MCB triggered: http://www.tmz.com (defined)
MCB triggered: http://xtube.com (defined)
MCB triggered: http://osha.gov (defined)
MCB triggered: http://www.osha.gov (defined)
MCB triggered: http://gravity.com (defined)
MCB triggered: http://www.yu.edu (defined)
MCB triggered: http://www.unfpa.org (defined)
MCB triggered: http://mgid.com (defined)
MCB triggered: http://www.mgid.com (defined)
MCB triggered: http://lh.co.th (defined)
MCB triggered: http://web.com (defined)
MCB triggered: http://csulb.edu (defined)
MCB triggered: http://jobamatic.com (defined)
MCB triggered: http://uniblue.com (defined)
MCB triggered: http://www.zscaler.com (defined)
MCB triggered: http://win-rar.com (defined)
MCB triggered: http://washington.edu (defined)
MCB triggered: http://www.washington.edu (defined)
MCB triggered: http://srware.net (defined)
MCB triggered: http://vpn4all.com (defined)
MCB triggered: http://discounttheatre.com (defined)
MCB triggered: http://www.bundesnetzagentur.de (defined)
MCB triggered: http://makewebeasy.com (defined)
MCB triggered: http://cbo.gov (defined)
MCB triggered: http://marketingoops.com (defined)
MCB triggered: http://movelia.es (defined)
MCB triggered: http://frictionalgames.com (defined)
MCB triggered: http://www.internap.co.jp (defined)
MCB triggered: http://qip.ru (defined)
MCB triggered: http://coochey.net (defined)
MCB triggered: http://www.coochey.net (defined)
MCB triggered: http://oag.com (defined)
MCB triggered: http://linbit.com (defined)
MCB triggered: http://www.linbit.com (defined)
MCB triggered: http://fcns.eu (defined)
MCB triggered: http://demandprogress.org (defined)
MCB triggered: http://slickdeals.net (defined)
MCB triggered: http://ideastorm.com (defined)
MCB triggered: http://www.norman.com (defined)
MCB triggered: http://domainmarket.com (defined)
MCB triggered: http://www.domainmarket.com (defined)
MCB triggered: http://www.conduit.com (defined)
MCB triggered: http://outbrain.com (defined)

[pde]

It might be good to have some way of noting in these rulesets that the mixed content situation has been audited and that we believe MCB isn't breaking layout or functionality on the site. If we did that, the MCB tests could ignore that ruleset.

[fuzzyroddis]

I for the life of me cannot find the mixed content blocking test so I'll ask, where can I find it?
Does it distinguish between active (eg. js/css/fonts etc) and passive (eg. images) content?

[jsha]

They are pretty obscure / hard to find. I'm hoping to make them a command-line option to test.sh instead, but currently the description of how to run is at the bottom of the README: https://github.com/EFForg/https-everywhere/blob/master/README.md#tests.

They look specifically for active content, because that's the only kind that gets blocked.

[fuzzyroddis]

For reference: https://github.com/EFForg/https-everywhere/blob/master/src/chrome/content/ruleset-tests.js

[fuzzyroddis]

May I suggest something like

<rule from="^http://(?:www\.)?example\.com/"
    to="https://www.example.com/"
    mixedcontent="ignore" />

or

Perhaps ignoring common blocked mixed content which doesn't affect the site, eg. some fonts, some social widgets etc.

[fuzzyroddis]

A thought: what happens if a site's MCB is ignored but later MCB becomes a problem?

[fuzzyroddis]

The idea of using perceptual hashing (eg. pHash) came to me while swimming today.

Take screenshot of "from" and "to" and compare them. I'd say the vast majority of mixed content blocking causes stylesheets to be blocked turning a nice design into black on white.

[fuzzyroddis]

@pde

Coming down the pipeline I'll also have detection for cert breakage,

Could you also write one for 4xx/5xx and timeouts? I noticed a lot of the sites tested didn't work. Might be good to clear dead sites from the rule database.

[jsha]

@pde Do you have more pending work on MCB auditing?

[fuzzyroddis]

Note #909

Where due to HTTPSE rewriting the new url conflicts with the site's Content Security Policy and is blocked.

[nemobis]

currently the description of how to run is at the bottom of the README: https://github.com/EFForg/https-everywhere/blob/master/README.md#tests.

Which says:

Now when you open the HTTPS Everywhere context menu there will be a "Run HTTPS Everywhere Ruleset Tests" menu item.

It's embarrassing, but I don't manage to do this: I see the context menu in Tools, but when I hover or click it nothing happens (Firefox 35.0). I guess I should file/look for a separate report/support item.

[nemobis]

I looked briefly into ruleset-tests.js and I'd appreciate pointers on where to look for the equivalent of PopupNotifications.getNotification("mixed-content-blocked", gBrowser.getBrowserForTab(tab)) in the case of connection refused etc. errors, which are not PopupNotifications. I've probably been looking in the wrong place of http://developer.mozilla.org/ .

[jsha]

It's often hard to find good documentation on how to do things in Firefox extensions. Keep in mind that there is not much of an extensions API per se. Instead, Firefox extensions are capable of interacting directly with the underlying Firefox implementation in many ways. So you are often looking for references on how to do things in Firefox itself.

I can help search out how to monitor for connection refused errors later in the day. I would also recommend joining #extdev on irc.mozilla.org and asking the question there. Thanks!

[pipboy96]

List is most likely very outdated. Closing, currently I work on a way to detect rulesets that trigger MCB automatically.

[marthep]

http://www.thaielectricalhub.com

[thanakornp]

https://www.xn--m3chf0ahc2cze1ecq8d.com