Causing browser crashes?

[diracdeltas]

Update: Several people have said that turning off SSL Observatory fixes this.

We are seeing more frequent browser crashes for some users. It seems to happen on OS X, Windows, and Linux.

This seems to be happening only on 3.4.5+. It occurs on at least Firefox 29 and 30, haven't seen it in Tor Browser yet.

Also being tracked here:

• https://trac.torproject.org/projects/tor/ticket/11700
• https://bugzilla.mozilla.org/show_bug.cgi?id=999434

Typical crash signatures:

• Linux: https://crash-stats.mozilla.com/report/index/7f9b693b-ee47-42fb-8db6-616622140617
• OS X: https://crash-stats.mozilla.com/report/index/ca4ee3ff-6549-4227-9706-0e6382140616
• Windows: https://crash-stats.mozilla.com/report/index/db43d540-bf8b-4315-aada-a96412140418

My vague impression is that it's occuring more often for Windows users.

[jackgu1988]

Same thing happens on Linux. Downgrading to version 3.4.5 and disabling the automatic updates solves the problem.

Someone should inspect the code changes on versions 3.5 and 3.5.1 and spot the issue.

[jcberthon]

The update to version 3.5.1 occured for me almost at the same time as the update of Firefox to version 29. And this occured on Ubuntu, OS X and Windows 7 at the same time.
Now my wife and I are experiencing regular crashes (several times a day) of Firefox, which I attribute first to the browser. But since 3 days I have disabled HTTPS Everywhere on my Windows 7 machine and I did not experienced a single crash. I am going to continue the experience on OS X and Ubuntu both for my wife and I.
Would be good that this is fixed soon.

[elias6]

I would like to say that I have been experiencing crashes in Firefox on both my 5-year-old MacBook Pro and my brand new MacBook Pro, both of which are running OS X 10.9.3 (Mavericks). I am using Firefox 30, but I remember seeing the crashes in Firefox 29 also. I can't remember if I saw it in older Firefox versions.

Firefox crashes, seemingly at random, every few hours. I think it is being caused by HTTPS Everywhere, because I disabled it and Firefox went for several days without crashing a single time.

I was using version 3.5.1 during most of the crashes. I tried upgrading to the 4.0 development version, hoping that would fix the problem, but it didn't seem to do anything.

I don't know what exactly is causing the crashes, but I have submitted some crash reports to Mozilla. Here are some of them, in case they may be helpful to anyone:

https://crash-stats.mozilla.com/report/index/ca4ee3ff-6549-4227-9706-0e6382140616
https://crash-stats.mozilla.com/report/index/69c15c3e-4807-46ae-bc07-27e502140616
https://crash-stats.mozilla.com/report/index/a7355377-60f1-4c4b-8cca-0be562140611
https://crash-stats.mozilla.com/report/index/2fd5457b-4feb-42af-be4f-0bf962140610
https://crash-stats.mozilla.com/report/index/2eea7d1b-3d63-4768-8483-4a3f92140609
https://crash-stats.mozilla.com/report/index/4c49c3c8-db81-4d3e-9efc-a7dca2140605

[diracdeltas]

Thanks, I'll look into this today. Has anyone tried version <3.5.1 on FF 29 or 30?

[jackgu1988]

Using version 3.4.5 on Linux (Firefox 30) and everything is fine here. I have disabled the automatic updates. The add-on was crashing the browser since version 28. I downgraded a while ago (version 29) and still no crashes at all.

[diracdeltas]

Current guess is that crashes were caused somewhere in the upgrade from 3.4.5 to 3.5

[diracdeltas]

But it would be good to confirm/deny whether this is happening on 3.5, since the difference between 3.5 and 3.5.1 is very minimal.

[jackgu1988]

I think it is happening in 3.5 as well. Let me use it for a day or so and I will report back if it crashes.

[jackgu1988]

OK, it only took a minute to test. Upgraded to 3.5, restarted the browser, the restored tabs started loading and Firefox crashed. I think it has never crashed again since I downgrated to 3.4.5.

[diracdeltas]

Do you have an example of a page that causes crashes?

[diracdeltas]

Actually, I think I got a crash right now while visiting https://crash-stats.mozilla.com/report/index/28fe3a51-e3ae-46e7-bcec-9e1272140616, but it didn't happen on 4.0development.17.

[diracdeltas]

For reference, this is also being discussed at https://bugzilla.mozilla.org/show_bug.cgi?id=999434 and https://trac.torproject.org/projects/tor/ticket/11700. UGH, so many bug trackers!

[andoruB]

Ah, my bad, I didn't think of looking for other similar threads. Could somebody merge the threads?
I'm pretty much sure this has been occurring since FF28.0.1, but I might be wrong, but I'm very sure 29 was indeed affected. Maybe it's OS dependant? I'm currently on a Debian-based distro called CrunchBang.
Unfortunately I don't know what exact pages cause the crash because I cannot reproduce the crash on the same pages, it just seems to happen randomly.

[diracdeltas]

@andoruB I got a very similar crash signature to yours just now on 64-bit debian. https://crash-stats.mozilla.com/report/index/7f9b693b-ee47-42fb-8db6-616622140617

[jackgu1988]

I do not think that it is OS related. It happens on Linux all the time. I do not think that it is website related either. It has occurred to me on all types of websites. Can you provide us with a diff of version 3.4.5 and 3.5? Maybe we can help.

[diracdeltas]

@jackgu1988 https://bug999434.bugzilla.mozilla.org/attachment.cgi?id=8441689

[jackgu1988]

@diracdeltas thanks! I compared the exact changes on github and I noticed that there have been quite a lot changes on the SSL observatory. I disabled it and no crash so far. I might be wrong, but usually it crashes after restarting the browser and having multiple tabs loading at the same time, using version 3.5.1.

I will continue testing tomorrow.

[trishankkarthik]

@diracdeltas Thanks for looking into this. If you look at my previous comment, it seems that 3.5.1, 4.0development.16 and 4.0development17 are all involved. I have found Firefox crash signatures in which HTTPS Everywhere (in either one of these versions) is involved in all reports.

[jackgu1988]

@Dachshund are you using the SSL observatory?

[trishankkarthik]

@jackgu1988 Yes, but I'm not sure whether that's the reason.

[jackgu1988]

@Dachshund can you try disabling it and testing whether it crashes? It has not crashed on my computer so far.

[diracdeltas]

@Dachshund Yes, I saw your comments. Thank you for commenting on every possible bug tracker/mailing list. ;)

@jackgu1988 Thanks, I'll ask on the other threads whether people have SSL Obs. enabled. (I do but am not getting crashes on 3.5.1 the majority of the time)

[trishankkarthik]

@diracdeltas You're welcome :P

[Veeshush]

Guy that started https://trac.torproject.org/projects/tor/ticket/11700 here.

Yep, I too have had SSL Observatory enabled on all the machines (3 separate Win7 64bit boxes) I've had the crashes on. I'm using https-everywhere-4.0development.17.xpi at the moment and still get crashes. I haven't tried downgrading to 3.4.5 yet.

edit
I haven't tried disabling SSL Observatory yet either. But I think that could be very related to the crashes though, cause again, all 3 of the systems I did have it enabled.

[andoruB]

@jackgu1988: When I said OS-specific I meant that maybe the problems with HTTPS Everywhere popped out during FF29.0 while on other OSes might have started much earlier, like in my case (FF 28.0.1)

I'll try and see if I get any crashes from now on with the SSL observatory disabled.

[jackgu1988]

@Veeshush since it crashes even when there are no websites loading, it seems logical to be the the observatory. Anyway, no crashes yet on 3.5.1 since I disabled it.

@andoruB sorry, I was referring to the first post of the page, where it says "My vague impression is that it's occuring more often for Windows users.".

[Argon-]

Count me in as an affected OSX user then. It happened over multiple versions of Firefox (I deactivated it finally in 29 but it started at least in 28).

For me it always happened when I clicked on a tab, especially when this triggered loading the website for this tab. I can't say for sure whether the actual loading-of-a-site process is needed or if it could happen for about:blank too.
At first I suspected a bug in the current FF version but since it didn't stop with 29 I started to deactivate extensions one by one. The crash was frequent enough for this to test and soon HTTPS Everywhere was identified as culprit.
Unfortunately my crash reports differed every time drastically, so I was never able to deduce some useful information from them.

[jackgu1988]

Well, no crashes so far with the observatory feature disabled. Under different circumstances it would have crashed more than 2-3 times by now.

[jackgu1988]

@jgillula maybe you are right. I checked the code 2 days ago but it seemed to only be a global instead of a local variable, so I paid little attention.

Calling .getService(Components.interfaces.nsISupports).wrappedJSObject; at this point, before entering the method in which the HTTPSEverywhere var was being created in 3.4.5, might return null or garbage that has been left over.

[diracdeltas]

quick suggestion i made on irc for folks trying to repro this bug:

07:18:15 PM - zyan: another test to try is to turn off "use_whitelist" in the https everywhere extension prefs in about:config
07:18:23 PM - zyan: and see if crashes happen more often
07:18:42 PM - zyan: if so, that weakly indicates that the bug is in the cert submission code
07:18:57 PM - zyan: (which submitChainArray is part of)

[jackgu1988]

@diracdeltas check my few previous messages. I have written my conclusion and it might be helpful.

[pde]

@jgillula I guess it's conceivable that this commit introduced a race condition when the SSL Observatory code tries to get a reference to the HTTPS Everywhere module. However that would cause this.HTTPSEverywhere.getWindowForChannel (channel) to raise a JS error, not a crash...

(I don't know if the chrome.manifest entries for these two modules ensure that each of them can look up a reference to the other as soon as either of their constructors start executing)

[pde]

@jsha to try to test your hypothesis about flaky TCP connections, I have disabled the Observatory server; it will now just reply with a 200 OK and not perform any cert logging. Are people continuing to see the crash?

[jsha]

@pde I think there may still be some issues with the Observatory server. Connecting takes a very long time, even if it eventually succeeds and returns 200. I've seen the below (connection timeout after 2m) as well as shorter waits like 30-60 seconds that return 200. I was also still able to reproduce the crash using my same set of tabs.

$ time curl -vvv -i https://observatory.eff.org/submit_cert -d 'domain=yahoo.com&server_ip=98.139.183.24&certlist=[""]&client_asn=-2&private_opt_in=0&browser_warning=0&padding=0'

• Adding handle: conn: 0x918b300
• Adding handle: send: 0
• Adding handle: recv: 0
• Curl_addHandleToPipeline: length: 1
• - Conn 0 (0x918b300) send_pipe: 1, recv_pipe: 0
• About to connect() to observatory.eff.org port 443 (#0)
• Trying 173.239.79.210...
• Connection timed out
• Failed connect to observatory.eff.org:443; Connection timed out
• Closing connection 0
  curl: (7) Failed connect to observatory.eff.org:443; Connection timed out

real 2m7.342s
user 0m0.016s
sys 0m0.012s

[jsha]

For aid in reproducing, I've uploaded a set of killer bookmarks at https://github.com/jsha/https-everywhere/blob/crash-repro/killer-bookmarks.json. I produced these by going through my Twitter feed and clicking every link.

To reproduce:

1. Install HTTPS Everywhere 3.5.1.
2. Bookmarks -> Show All Bookmarks -> Import and Backup -> Restore -> Choose File, and select killer-bookmarks.json.
3. Bookmarks -> killer bookmarks -> Show All In Tabs
4. Rapidly Ctrl-Tab through all tabs to force them to load.

This should trigger a crash. If it doesn't work try manually opening an additional new tab.

I'm a fan of running this in a separate dev profile (firefox -Profilemanager), and setting Firefox to open the tabs that were previously open when it starts up. It's easier to iterate that way.

[jsha]

I created an Apache config (#328) that simulates 100 servers with different SSL certificates, to make it easy to simulate submitting a large number of certificates to the Observatory simultaneously.

My branch also simulates the Observatory's /submit_cert endpoint, which means you can sleep() arbitrarily long before replying on that endpoint, to simulate server-side slowness.

Unfortunately, I was not able to reproduce a crash using this setup. One flaw: my mocked-out Observatory server sleeps during the HTTP request, after the SSL connection has been established. The situation with the live Observatory is that sometimes connections time out during the handshake. I'll have to do more work to simulate this case.

Of course, it's entirely possible that the crashes don't have to do with performance of the Observatory.

[trishankkarthik]

I've never been so happy to reproduce a crash!

Here is observatory.log:

SSL Observatory: Planning to retry submission...
SSL Observatory: Too many outstanding requests (21), not submitting
SSL Observatory: Too many outstanding requests (21), not submitting
SSL Observatory: Planning to retry submission...
SSL Observatory: Too many outstanding requests (21), not submitting
SSL Observatory: Too many outstanding requests (21), not submitting
SSL Observatory: Planning to retry submission...
SSL Observatory: Too many outstanding requests (21), not submitting
SSL Observatory: Too many outstanding requests (21), not submitting
SSL Observatory: Planning to retry submission...

And here is the crash report. It's a crash signature we've seen before: @ js::GCMarker::GrayCallback.

Unfounded speculation: too many outstanding SSL-observatory requests, Firefox may be GC-ing them, hilarity ensues on retry?

[trishankkarthik]

Another crash report on Fx 30.0. (No bad luck yet on crashing Fx 32.0a2.) observatory.log:

HTTPS Everywhere:   Outbrain (partial)
HTTPS Everywhere: Yes: https://www.outbrain.com/0.27801961540597986/0.27801961540597986
SSL Observatory: Private browsing mode: false
SSL Observatory: Private browsing mode: false
SSL Observatory: Private browsing mode: false
SSL Observatory: SHA-256 hash of cert chain for fbstatic-a.akamaihd.net is 1715E347E92C85ADCEFCA2ED0C4B56DF7D7A3FA9C834B1F2DFA7CE115DC8DBC4
SSL Observatory: Not using whitelist to filter cert chains.
SSL Observatory: Got root cert at position: 2
SSL Observatory: Too many outstanding requests (21), not submitting
HTTPS Everywhere: Got http-on-examine-response @ https://fbstatic-a.akamaihd.net/rsrc.php/v2/yb/r/GsNJNwuI-UM.gif

Crash report.

[diracdeltas]

I made a package with the SSL Obs. changes reverted to 3.4.5. Please install and let me know if there's any crashes! If not, this will be the 3.5.2 release: https://www.eff.org/files/https-everywhere-3.5.1~pre.xpi

[jackgu1988]

That's great. I had few crashes with SSL Obs. disabled a few days ago. Fewer than with it enabled, but still they went away as soon as I downgraded to 3.4.5 again.

Let's give it a try though.

[Veeshush]

Installed the https://www.eff.org/files/https-everywhere-3.5.1%7Epre.xpi just now. My crashes before were almost daily, so within a week I should be able to tell.

Also thanks to everyone working on sorting this out!

[diracdeltas]

@jackgu1988 @Veeshush I was hoping to do a release today. Any data points yet?

[jackgu1988]

@diracdeltas it seems fine so far. Spent the whole day with it and SSL Obs. enabled and no crashes so far. Even if it crashes, it should not be this bad.

Any functionality we are losing tho?

[diracdeltas]

@jackgu1988 Mostly private browsing mode detection, I think. b44e5da#diff-25d902c24283ab8cfbac54dfa101ad31

[jsha]

@diracdeltas I also confirm no crashes. I was able to reproduce 2/2 times using my technique listed above, against the 3.5.1 tag. Installing the test version you just linked, I produced no crashes in 3/3 tries.

[diracdeltas]

@jsha thanks!

[jsha]

I reduced the KeepAliveTimeout on the Observatory server, which mostly resolved the slow connections issue. In testing with curl, all my connections to observatory.eff.org completed in less than 1 second.

l tried to repro the crash with the version 3.5.1 plugin, using the steps I described above. I was able to reproduce a crash in 2 out of 4 tries, and it seemed to take a little longer to reproduce the crash. That's kind of an inconclusive result. Perhaps the crash does not require a slow Observatory server, but is exacerbated by it?

[Veeshush]

No issues with 3.5.3 yet. Like I said though, I was getting crashes before almost everyday (never making it past 3 days). So I'll have to give it a solid week to be sure.

[Veeshush]

Not a single crash since. Going on 20 days now without any issues!

[pde]

Is anyone running 4.0dev17, which we might expect to still be crashing? If it is, and we aren't going to the problematic change out of master, @diracdeltas we should probably tell mike not to ship 4.0dev18 in TBB alpha...

[diracdeltas]

@pde AFAIK 4.0dev.15 and 16 at least also had this bug.

Also, I'm planning to call the next release 5.0dev.0 if that's okay. The next stable release will be called 4.0.

[pde]

Sounds ok to me, especially if the next stable release will include all those crazy rulesets. FWIW I think your 3.5 merge had the effect of shipping the ones that will cause the most widespread breakage, since large popular sites are also the most complex and widely used.

OK we're way OT for this bug now :)

[jsha]

Fixed over a year ago. :-)