Impact of ZRTP library critical security vulnerabilities #5

Open
hellais opened this Issue Jun 27, 2013 · 34 comments

Projects

None yet

10 participants

@hellais
hellais commented Jun 27, 2013

(The main contents of this issue were removed for about a day because they contained the technical details of serious vulnerabilities in the open source ZRTP library that both Silent Phone and a number of other VoIP products used. These details were taken from Mark Dowd's blog post that had been inadvertently released before all of the venders involved had released their fixes. By the time this issue was created, Mark had taken down his post specifically to give us and other venders time to get the fixes distributed.)

(The manner in which we edited this issue were somewhat clumsy and caused a bit of not undeserved flap in the comments below. We apologize again. We take you now to the original issue, already in progress.... --jlb)


I was wondering what the impact on your product these recently disclosed multiple critical vulnerabilities inside of the ZRTP library used in silent circle.

https://webcache.googleusercontent.com/search?q=cache:http://blog.azimuthsecurity.com/2013/06/attacking-crypto-phones-weaknesses-in.html

Are silent circle clients remotely exploitable due to these ZRTP library bugs?

I also noticed that some links have now disappeared from the sites where they were published. Just in case you missed them here is an attached screenshot of the origin pages and uploaded a copy of the pdf's here:

Original Post (PDF print of web page from google page) http://temp-share.com/show/dPf3U597W
Chilling effect of original blog post (PDF print of the web page as now, without details) http://temp-share.com/show/Pf3YC4p92
Article on the ThreatPost.com vulnerability website (this has been removed too) http://temp-share.com/show/f3Yg6hqkn

azimuth security_ attacking crypto phones_ weaknesses in zrtpcpp-1
azimuth security_ attacking crypto phones_ weaknesses in zrtpcpp-3
azimuth security_ attacking crypto phones_ weaknesses in zrtpcpp-2

several flaws discovered in zrtpcpp library used in secure phone apps _ threatpost-1
several flaws discovered in zrtpcpp library used in secure phone apps _ threatpost-2

@Jim-Burrows Jim-Burrows was assigned Jun 27, 2013
@Jim-Burrows

Thank you. We've been working on this for a couple of days, and are in the process of rolling out the fixes to Silent Phone on both platforms. Mark was kind enough to with draw his blog post until the vulnerable apps are in the stores. We are grateful for his finding and reporting this problem, working with us to get it fixed and his discretion until our products reflected Werner's fixes.

@Jim-Burrows

The Android app has been updated in the Play store. More to come.

@kaepora
kaepora commented Jun 27, 2013

@Jim-Burrows Correct me if I'm mistaken, but @hellais's original post in this issue seems to have been edited without any notice in order remove the vulnerabilities that he pointed out. Could you explain why this has been done?

I understand that there might be concerns for user's safety, but I would like to link to the Wikipedia explanation of the principle of full disclosure, which is central to security practice. I'm very surprised by Silent Circle's decision.

@kaepora
kaepora commented Jun 27, 2013

@lazzarello Thanks! Any explanation as to why the original post by @hellais has been censored without notice?

@kaepora
kaepora commented Jun 27, 2013

I'm posting a copy of @hellais's original post and I sincerely hope it's not censored again.

Full post here (Link removed. Mark Dowd will be reposting his original article as soon as venders have responded to these remote execution vulnerabilities. Until then we decline to participate in publishing the technical details. --jlb)

"I was wondering what the impact on your product these recently disclosed multiple critical vulnerabilities inside of the ZRTP library used in silent circle.

(Link removed, as above.)

Are silent circle clients remotely exploitable due to these ZRTP library bugs?
I also noticed that some links have now disappeared from the sites where they were published. Just in case you missedthem here is an attached screenshot of the origin pages and uploaded a copy of the pdf's here:
Original Post (PDF print of web page from google page): http://temp-share.com/show/dPf3U597W
Chilling effect of original blog post: (PDF print of the web page as now, without details)
http://temp-share.com/show/Pf3YC4p92: Article on the ThreatPost.com vulnerability website (this has been removed )"

@hellais
hellais commented Jun 27, 2013

@Jim-Burrows it is usually good manners to state that you edited the comment that somebody else made as the phrase that appears to be said by me is actually not what I said.

From the looks of this the reason why you are doing this is not to protect users, but it's to protect the image of your company.
I find that a bit sad.

You could have at least left the part where the type of security vulnerability is said so as to give people the opportunity to understand that this issue is critical (remote unauthenticated code execution) and that they should update ASAP.

@Jim-Burrows

Sorry about that @helias. I didn't actually edit the original post. I asked one of my guys to do it and was probably a bit too kurt in what I said. My bad.

Yes, we deleted the details of the bug because while Werner got his fix out to his repository several hours ago, our products have not been fixed in the field. Mark had volunteered to embargo his blog post until we'd rolled out our fix, as is common practice. A bit of miscommunication and his very legitimate concern that users of Werner's library and libraries built upon it understand the severity of the bug cause his blog post to go out prematurely.

As mentioned above, we already have our fix in the Google Play store, so as soon as our Android users update they will be safe from these vulnerabilities, but our iOS customers will be vulnerable for a while longer as the fix makes its way through the App Store review process, so we do not particularly want to participate in publishing the details of the vulnerability.

Things got a little hectic today with the inadvertent premature release of the details before we'd completed testing Werners fixes in the context of our production apps. We had thought we had a few days to complete test and shepherd the app through Apple's review process. I'm sorry that that extra pressure caused us to be a bit hasty here. You have my apology. It was disrespectful.

On the whole the process has been working very well. @mdowd has been very helpful in finding the bugs, giving Werner and Silent Circle the details such that we could fix it rapidly, documenting the vulnerabilities in complete detail and agreeing to embargo the post for a couple of days while Werner fixed the bugs and integrated the fix into our applications. There was a small snafu that caught us a bit off balance, but aside from that things have been working really well.

Please note that our own sources here have not yet been updated. We were in the process of testing a new release and preparing a now check-in here when Mark's report arrived, and we have been pushing the testing and distribution of the fix itself ahead of the source code release. It will follow, as soon as we catch our breath.

@RiptideTempora

"I didn't actually edit the original post. I asked one of my guys to do it and was probably a bit too kurt in what I said." You sounded like a politician right there, not letting the hammer of accountability fall where it should. At least you apologized though. :P

Aside, you might want to audit every library you use for well-known vulnerabilities.

@Jim-Burrows

@RiptideTempora, I'm sorry that I came off sounding like a politician. That was not my intent. Let me try again. The deletion of the original contents without noting that it was done was MY FAULT even though I didn't do the actual editing. I described the editor as "one of my guys" rather than naming him because I, not he, deserve the blame. I gave poor directions. My intent in the passage you quote was not to shift or escape blame but to take it. That's why I ended that paragraph with "My bad".

So, the hammer of accountability should rightly fall on me, Jim Burrows, VP of Engineering and obvious klutz.

Since it is my fault, I am sorry.

As to auditing, I agree with you completely. That's why we audit and test our own work and pay others to audit and test it for us. Obviously, in this case, the auditing failed. It was my understanding that all of the libraries that we used were audited, as well as all of our own code. The fact that these problems were missed suggests that there is a problem with the auditing process, that either not all of the third party libraries were audited or that somehow the auditing was not rigorous. Besides developing, testing and deploying these fixes, we will also be looking into the process.

But, in-house and third party audits do fail. That's why we publish our sources here, so that people like Mark Dowd can find the bugs that we and those we hired missed. We are extremely grateful to Mark, and have thanked him both by phone and email and asked him to keep up the good work. He has made a valuable contribution and deserves full credit. We are also grateful that when he learned that we had not yet fully deployed the fix he chose to take down his post and went further than that and spoke to a couple of reporters who had spoken to him asking that they, too, embargo the story until we and the other venders who build products based on Werner's library can deploy a fix. Mark deserves nothing but credit.

So, in summary: Mark--good. Riptide--right. Hellais--treated disrespectfully. Jim--responsible and sorry.

@Jim-Burrows

Status update: As noted earlier, the Android version is in the Google Play Store. The iOS version has been submitted to Apple for review and posting in the App Store, and expedited handling has been requested. As soon as it comes out we expect that Mark and those who have embargoed their stories will republish or publish them, as appropriate.

@hellais
hellais commented Jun 28, 2013

@Jim-Burrows cool. Thanks for keeping us updated on this issue.

@Jim-Burrows

You're welcome, @hellais. Things have been a litte hectic here working with Werner to get the bugs fixed, tested, and rolled out to his repository, our apps and to the other third party libraries and apps that layer on top of ZRTP. We are grateful not only for Mark and Werner's hard work, but for everyone's understanding and cooperation--Mark's, yours, the reporters who embargoed their stories, and so forth while we closed the vulnerability in all the affected libraries and apps.

We've all been moving quickly, and so there have been a few false steps all around. Still, on the whole the process is working. The flaws are identified, solutions created and deployed and the word gotten out. We could all wish for a smoother process, but what is most important is closing the vulnerabilities and protecting everyone's users and customers. No doubt, as Mark and others find additional flaws in the future, the process will smooth out.

No word yet on when the iOS app will hit the app store, but Apple is expediting the process. Stay tuned.

@Jim-Burrows

Yay! The iOS version of Silent Phone with the fix is now in the app store. I want to check with Mark to make sure that other venders known to be built on the ZRTPCPP library are clear, but our smartphone apps now have updates for these problems! Many many thanks to everyone.

@kaepora
kaepora commented Jun 28, 2013

@Jim-Burrows I sincerely hope this means you'll stop censoring other developers, including myself. You have really gone against the spirit of full disclosure and open source development in this episode. It's shameful.

@kaepora
kaepora commented Jun 28, 2013

I just reviewed the update log for the iOS version of Silent Phone:

screen shot 2013-06-28 at 5 36 49 pm

I am extremely surprised to see that it includes absolutely no mention of fixes for critical vulnerabilities. Why aren't users informed?

@joncallas

Good catch. Thanks. We were in a hurry.

@kaepora
kaepora commented Jun 28, 2013

@joncallas Thanks for clarifying. I was rather under the impression that Silent Circle was testing its new Silent Changelog feature.

@Jim-Burrows

@hellais, I plan on restoring the technical details that were removed from your issue posting as soon as I hear from Mark and Werner that there are no other venders known to be still scrambling to protect their customers. Just because we at Silent Circle have our updates our doesn't mean everyone has. I have sent them both emails, but either or both of them could be sleeping given the times in Germany and Australia.

Thank you for your patience.

@Jim-Burrows

OK. We believe that the various libraries and products, both ours and other third parties, have been fixed, and Mark has reposted his blog post. We have therefore restored the original contents of @hellais's orignal issue text above.

Additionally, our CTO, Jon Callas, has blogged a bit about this on our new blog.

Once again we are sorry for any confusion or offense that our haste has caused.

@kaepora
kaepora commented Jun 29, 2013

@Jim-Burrows Do you consider this issue to be dealt with?

@Jim-Burrows

@kaepora, no, not completely. In terms of simply getting the apps updated to remove the vulnerabilities, we have created, tested and deployed to the Play and App stores the fixes, but that's only part of the story.

We have not yet posted the sources involved here. We will do that only after we and our third party security partners have completed our audit.

Additionally, as pointed out above, the haste caused by the vulnerabilities being prematurely posted caused us to make a couple of mistakes, one of which was that the descriptive text for the store submissions was too brief and generic. We are contemplating how to fix that, whether we should bundle that change with some existing bug fixes or new features, do a "null update" or just wait. (Please note that a release without changes isn't easy in the case of the App store.)

Beyond that, this first high priority, expedited, security update showed a number of flaws in our rapid deployment procedures, some of which have been apparent here, and some of which are behind the scenes. We are reviewing our procedures with an eye towards insuring that we make totally different mistakes next time. The same old mistakes are boring.

So, no, I do not consider this set of issues fully dealt with. Only our highest priority issue, protecting our customers by removing the vulnerability from the distributed applications is taken care of. "No rest for the wicked weary", as they say.

@kaepora
kaepora commented Jun 29, 2013

@Jim-Burrows During the full week when your entire user base was vulnerable, you did not issue a single public advisory, on Twitter, Facebook, or on your website, advising your users that they were currently vulnerable to these critical flaws, that their security was compromised, even ever since the launch of Silent Circle.

Even now, after pushing the updates, you still have nowhere mentioned that users must update as a matter of critical security neither on your Facebook, Twitter, or website. The change log for the iPhone version only mentioned "bugfixes" while the tweet regarding the update similarly mentioned no security implications.

Furthermore, when I ask you if you consider the issue dealt with, you list a bunch of things you still need to do, but still do not mention the fact that you have left your entire user base in the dark for the entire week for which you knew they were all compromised, and still leave them in the dark even today. Instead, you chose to protect yourselves against any possible attacks to your pretentious reputation.

I think Silent Circle is a company that has, since the beginning, exploited the reputation and clout of its founders as a reason to contravene proper security practices, proper open source development, and now even full disclosure. When I make mistakes as big as these at work, I face immense criticism and scrutiny because I am fully transparent and allow people to fully realize the extent of my failures. I contact users, write blog posts, and allow the media to cover the discovered vulnerabilities because that's what open security practice is about. You don't do this. Do I have to be Phil Zimmermann in order to be able to afford myself a shield against public critique and a license to censor, as you seem to think you are allowed to?

Your entire bet is on your reputation, and you rely on it in order to shield yourself from having to behave just like any other open source project. You rely on it in order to pretend that Silent Circle deserves special status, and that it doesn't develop flaws that it needs to address as transparently as RedPhone, OSTel or any of the open source competition. I accuse Silent Circle of being absolutely hypocritical.

In addition, @Jim-Burrows, you write the following:

Additionally, as pointed out above, the haste caused by the vulnerabilities being prematurely posted caused us to make a couple of mistakes...

What do you mean, "prematurely posted?" These vulnerabilities had been in the codebase for six years before they were discovered. This codebase is also used by other projects, who were also rendered vulnerable. You chose to monopolize the situation when you took them down, as if they only concern your product and your reputation. But here's the thing: you do not get to decide when the open source security community publishes advisories, or when their publication is premature. We're not all here trying to tend to Silent Circle's corporate comforts — we publish research and discuss bugs.
When you write things like this, you completely erode my confidence that you have any idea what proper security practice is.

I have more comments as to why I believe you are disregarding any sort of proper open source security practice:
In the past few months, you've released a number of updates to Silent Circle. However, your source code on GitHub hasn't been updated once ever since you posted it five months ago. It doesn't even build (#2)! Why is it not updated?

I have some questions about your update changelog:
screen shot 2013-06-29 at 12 37 48 pm

We know that you passed off three critical security bugs as "bugfixes", outlined in the first red rectangle. Were @hellais not to open his bug report, you would have likely reacted the same way you are reacting now: endangering your users to protect your personal reputation, and on top of it, we would have never known that "bugfixes" can stand for something much more serious to your users' safety.

Here is another vulnerability that was reported five months ago: #3 — has it been fixed? If so, why was an advisory never issued? You had all the time in the world to fix it — how was the fix described in the change log? Were users notified? If so, how? Was it also a "bugfix"?

Finally, the changelog for your Android update of Silent Phone, which, I repeat, fixed vulnerabilities that had existed for six years, is absolutely incredible:
screen shot 2013-06-29 at 12 46 24 pm

I look forward to Silent Circle censoring this post as well, just as they censored my previous post.

@samthetechie

Dear Silent circle developers, it appears that Silent Circle believes that it's founding member's reputation for "security" and "secure" (evidently reduced to marketing buzzwords) are more important than letting users know they are at risk and should upgrade as @kaepora points out :

@Jim-Burrows During the full week when your entire user base was vulnerable, you did not issue a single public advisory, on Twitter, Facebook, or on your website, advising your users that they were currently vulnerable to these critical flaws, that their security was compromised, even ever since the launch of Silent Circle.

Question 1: @Jim-Burrows Silent Circle has failed to disclose and communicate to users about a serious security bug. I have read your responses but I am still unclear, what was/is your rationale for not informing users?

Question 2: @Jim-Burrows Silent Circle has a twitter account, facebook account and a website. These seem like great communications platforms for letting your users know that you are protecting their data by doing your job diligently. But you seem to only use these platforms for positive news stories and other PR/marketing functions. What are you going to do differently in the future to protect and inform your users?

Question 3: @Jim-Burrows how are a community of developers supposed to help if your code does not compile and has not been updated in 5 months?

"In the past few months, you've released a number of updates to Silent Circle. However, your source code on GitHub hasn't been updated once ever since you posted it five months ago. It doesn't even build (#2)! Why is it not updated?"

@Jim-Burrows

Thank you @samthetechie. Your comment is timely indeed. As you may notice if you are an Android user, our latest update, 1.6.5 of Silent Phone just hit the Play Store and in it we have attempted to correct some of the problems I was noting and apologizing for a few replies back. I think this update went a little smoother. It was nice not having to rush this time.

First off, in this release we didn't leave off the "What's New?" section and in it we included not only the text for this release, but for the previous one as well. The section reads:

What's in this version:
version 1.6.4, build 182:

  • library updates: polarssl 1.2.7+, zrtpcpp – security update
  • includes fixes for ZRTP issues reported by Mark Dowd

version 1.6.5, build 189:

  • library updated: polarssl 1.2.8, zrtpcpp – security update
  • includes additional fixes for SIP/SDES issues found by the authors

Silent Circle recommends all users install this, and all security updates immediately.

Hopefully, this will inform any users who do not have automatic update turned on that this release (and the last one) include security updates and should be updated to immediately. Those who had update turned on, off course, got the update the same day that Mark's vulnerabilities were accidentally published prematurely.

Please note, that despite what has been said by the more excitable folk here and elsewhere on the net, the problems that Mark found and which we were working on with his help at the time of the premature release of the technical details was not a zero-day bug. Mark had found serious flaws in an open source library that we use, one that had been there for years.He did not have proof of concept code and there was no known exploit. Just the sort of vulnerability which, once the technical details are out, can often be turned into an exploit. That's a serious issue, no question about it. But it is not a zero day.

As to our Twitter and Facebook accounts, you make a good point. They are currently pretty much the purview of our marketing team, and have not been much used by my engineering team, the technical operations group or support, and we could quite probably make better use of them. Similarly, we are only just now getting our blog started. So much to do and so little time. As it turns out, most of our technical folk tend to be pretty heads down, getting the code designed, written, tested, and deployed leaving the outside communications more to the Sales & Marketing teams. You make a good point thank you.

The whole ZRTP SNAFU disrupted our processes, ca,using us to rush stuff out the door faster than we usually do. (Note the getting it to the App Store, which usually takes a couple of weeks just in Apple review, in less than a day.) As a result the normal checking and audits we do before clearing the source to be published here is lagging well behind, and for that I am sorry. Worse yet, last night I received word that there was a possible security issue that may slow down source release again. I need to check this out. My hope is that sources that are synched with the latest release will show up here Real Soon Now. I further hope that it will be much more readily buildable. If, however, the CSO has found something we need to fix before publishing, then he wins, automatically.

Thank you for your input, your questions and your concerns.

@youjustseem

every time silent circle responds, they sound more and more like crooks and liars

@Jim-Burrows

I'm sorry you feel that way. I'm trying to be as above board and as open as is possible. Perhaps I'm just too long-winded and old-fashioned to come off as authentic here.

It's obvious that you feel strongly about this from the fact that you seem to have created your account today specifically to make this comment. I'd be happy to talk to you, or anyone who has expressed concern here, off-line. Feel free to email me as Jim.Burrows@SilentCircle-LLC.com or call me at +1 (202) 499-6427 ext 4520.

@Jim-Burrows

We have now release the second round of security fixes for iOS and they are available in the App store. As with Android, we have added text in the "What's New" section documenting the security fixes of both 1.6.5 and 1.6.4 and have reminded users that they should install all security updates immediately.

The what's new section reads:

What's New in Version 1.6.5

Version 1.6.4:

  • Library updates: polarssl 1.2.7+, zrtpcpp – security update.
  • Includes fixes for ZRTP issues reported by Mark Dowd.

version 1.6.5:

  • Library updated: polarssl 1.2.8, zrtpcpp – security update.
  • Includes additional fixes for SIP/SDES issues found by the authors.

Silent Circle recommends that all users install all security updates immediately.

The source updates corresponding to 1.6.5 are still in the works, and should be posted here soon. We expect that the Android sources will be fully buildable, though not the iOS. We are awaiting the final security review.

@Jim-Burrows

@samthetechie, thank you for your Twitter suggestion. Now that Silent Phone 1.6.5 is out in both stores, I asked Marketing to twee a recommendation that users update to it. They have. And posted it to Facebook as well, it seems.

@kaepora
kaepora commented Aug 10, 2013

Allow me to come out of nowhere to deliver the following:
I sincerely apologize to the Silent Circle team for being overly critical. Sure, I had good points, but I need to be more open and mindful. In the future, I will make an effort to be less aggressive in making my points.

@joncallas

Thank you for the kind words, Nadim. I wish you all the best on your work. As I've said in the past, I admire the work you do on CryptoCat. I hope you have a successful hackathon.

@pettter
pettter commented Sep 6, 2013

@Jim-Burrows

When, exactly is 'soon'?

It's been two months since you said source code was coming, seven since the current version was posted. Please do spend the time and effort to rectify this. Given enough eyeballs, etc.

Given the current climate, properly vetted and implemented cryptography is absolutely critical, and having the source code available is central to the whole concept.

@Jim-Burrows

@pettter, "Soon" is today, well, actually last night.

We've just released the sources to Silent Phone for Android V1.6.5. And, yes, we released them one week after we released 1.6.6 to the Play Store, so they're a little bit stale, BUT... what delayed us was making sure that they were buildable from the GitHub repo outside our build environment. That means, assuming we got it right, that you can check out our repo here on GitHub, build your own APK, install it on your phone and run it instead of our Play Store version.

And to make lemonade out of the lemons of being one release behind, we plan on releasing 1.6.6 in a couple of weeks, so, if you try to build 1.6.5 and find that we blew it somehow, you can post an issue here and we've already got a release planned to fix it in.

I'm really sorry that "soon" took this long. It was absolutely NOT my plan, but this summer has been really really hectic (for obvious reasons) and we're a small company with limited resources. The slowness has really frustrated me, as has the fact that when I yell, "What idiot set those priorities?" each time something delayed posting here, the answer was always "me". I can try to blame all the Snowden, NSA, Prism brouhaha and the time and resource pressures it has put us under, but in the end, I'm the one who grits his teeth and says, "Yes, that's more important than the GitHub release. Make it so."

I'd be happy to have you sympathize with me for the decisions I've faced this summer, but I absolutely would not disagree with you if you blamed me for the delay. I own it.

Silent Phone for iOS sources, Silent Text for Android, and then Silent Phone for Android 1.6.6 source releases are all in the pipeline, and if you'll forgive me for using a word that I myself have sullied, they should all be here "soon".

@valentt
valentt commented Apr 20, 2015

Two years later... have issues with ZRTP library been fixed? From reading the comments it looks like it has been fixed. Why is this issue still open?

@Jim-Burrows Jim-Burrows was unassigned by valentt Apr 20, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment