Bad SSL certificate on https://ciscobinary.openh264.org

[vi]

ciscobinary.openh264.org uses an invalid security certificate.

The certificate is only valid for the following names:
*.akamaihd.net , *.akamaihd-staging.net , a248.e.akamai.net

(Error code: ssl_error_bad_cert_domain)

[fluffy]

We would like people to check the security of the download by matching the checksum of the downloaded file to the fingerprints that mozilla uses. That provides a much more secure solution.

[PhilLab]

Please re-evaluate this decision. People might not do this out of laziness / ignorance (count me in!). Making sure they at least have something from the owner of openh264.org might prevent a lot of harm.

Checking the checksum is definitely needed for true safety, but why not make the worst case a little less worse?

[PhilLab]

By the way, where can I find the fingerprint Mozilla uses?

[nils-ohlmeier]

@dminor do you know if we store the current fingerprints in tree somewhere?

[fluffy]

I'll reopen for discution

[dminor]

These should be the current fingerprints: https://searchfox.org/mozilla-central/source/toolkit/content/gmp-sources/openh264.json. However, that file is only used as a fallback when we can't reach the normal update server. The update server sends Firefox a url and a fingerprint to use. The intention is to make it easy to roll out updates to a portion of the population, but obviously does not make it easy to trace what is being downloaded. @Callek would know more about that side of things.

[Callek]

Ok, so the issue in #909 (comment) looks like its a cert issue on the domain itself...

I'm not sure what #909 (comment) is asking specifically, though Mozilla is downloading the binaries from http://ciscobinary.openh264.org/ and not https://... so we don't encounter the certificate issue. We do however have code that validates that our update service is the correct service (via cert) and that the urls it sends over are validated once downloaded with a sha512 algorithm, so we never invoke untrusted code (even if a MITM struck and gave the user unexpected content)

Finally we codesign the binaries on windows, and gpg sign on linux (note the gpg sig is harder to verify since we don't expose this anywhere officially, nor is it fully guaranteed to exist in the future) e.g. http://ciscobinary.openh264.org/openh264-linux64-2e1774ab6dc6c43debb0b5b628bdf122a391d521.zip.asc

So I think this issue comes down to @fluffy and Cisco in general for if they want the ciscobinary.openh264.org to have a valid https cert.

[sijchen]

for cross-reference https://bugzilla.mozilla.org/show_bug.cgi?id=1102531

[flossposse]

Can this issue be revisited? This is a potential security issue for anyone downloading these files.

[nanonyme]

This is not just potential but real security vulnerability. The GPG that was there seems to be gone. Cisco binaries are at this point not safe to use without locally reproducing same binary so you can collect sha256.

[mcatanzaro]

Hi @fluffy, could somebody from Cisco help fix this please?

There are two related issues above that can be closed as duplicates.

[fluffy]

HTTPS is not supported for this. We think a better approach is to check the fingerprint of the downloaded files.

[nanonyme]

@fluffy then can you please start providing strong hashes for the files from GitHub releases? Download server only has weak hashes which are useless by modern standards and they are untrusted since you are not supporting TLS or providing GPG scheme.

[flossposse]

@fluffy Is there a secure way to get the fingerprints?

[mcatanzaro]

We think a better approach is to check the fingerprint of the downloaded files.

@fluffy can you show us a secure page where these fingerprints or hashes are published? I think it does not exist, and that everyone is just doing trust-on-first-use today: download OpenH264 insecurely, hash it, and just trust that the original download was probably/hopefully not hijacked. This is inadequate nowadays, as apparently we are no longer allowed to ignore supply chain security. :)

Possible solutions:

• Fix the TLS certificate on ciscobinary.openh264.org
• Publish hashes on any other website that does use TLS
• Use GitHub releases instead

[fluffy]

Mozilla is the group that is securely publishing the hashes. They actually compile the code to. I'me feeling guilty I can not point you at the exactly location.

Even if we did move to https from github, the whole process by which the executables get to github is not very secure thus the realiance on the hashes.

[Callek]

Few places Mozilla takes on stuff for this plugin [context I used to be a release engineer there, and one of the primary people responsible for delivering this]

https://searchfox.org/mozilla-central/source/taskcluster/ci/openh264-plugin/kind.yml
https://searchfox.org/mozilla-central/source/taskcluster/ci/openh264-signing/kind.yml

https://searchfox.org/mozilla-central/source/testing/mozharness/configs/openh264
https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/openh264_build.py

Some more results:
https://searchfox.org/mozilla-central/search?q=openh264&path=openh264&case=false&regexp=false

But yes, Mozilla builds them for Cisco [at least used to, but I think still does] and then Cisco published the binaries on their own server. Mozilla publishes update information to point Firefox at the binaries and at the related file hashes, this update information is protected much the same way actual Firefox updates are protected.

[nanonyme]

If we can get confirmation that this is in fact still the case that Mozilla builds the files (I think this one was already confirmed by @fluffy above) and hashes originate from inside of Mozilla's internal infrastructure, then the supply chain is fine. Hashes obviously cannot be calculated based on any payloads downloaded over the Internet unencrypted to be trustable.

[mcatanzaro]

Let's also post instructions for how to verify the hashes somewhere prominent (like the toplevel README.md).

[mcatanzaro]

And probably also instructions explaining that ciscobinary.openh264.org is not intended to be accessed via HTTPS.

And close #3662

[flossposse]

Thanks for that info @Callek!

I'm not super familiar with the Mozilla codebase or infrastructure, so I could be wrong about this, but it appears like this is the script that is being used to generate the hashes for the openh264 plugn: https://searchfox.org/mozilla-central/source/dom/media/tools/generateGmpJson.py

If the script is invoked without arguments, it will (insecurely) download binaries from http://ciscobinary.openh264.org/ and use them to calculate hashes.
If the script is invoked with the --url argument, then it will download binaries from the URL in the argument.

Assuming this script is what's used to generate the hashes, we can consider this secure if Mozilla is passing in a http://localhost/, file://, or https:// URL pointing to the artifacts on their internal build server when they generate the hashes, and if Mozilla doesn't ever invoke that script without passing in a secure URL.

Unfortunately, I can't find any CI tasks in Mozilla-Central that invoke generateGmpJson or that upload the artifacts to Cisco, so I can't tell whether the script is being used securely or not (or at all). The fact that the insecure behavior is the default one reduces my confidence.

[bhearsum]

Hi, I'm a current engineer on Mozilla's Release Engineering team (hi @Callek!), and I just want to clarify one point.

I'm not super familiar with the Mozilla codebase or infrastructure, so I could be wrong about this, but it appears like this is the script that is being used to generate the hashes for the openh264 plugn: https://searchfox.org/mozilla-central/source/dom/media/tools/generateGmpJson.py

That script may be used to generate the in tree hashes (I'm not sure - I'm checking with the folks that update them), but it is definitely not used to generate the hashes served by our update server. Those are generated by hand by Release Engineers, who pull the builds securely from our CI system.

Unfortunately, I can't find any CI tasks in Mozilla-Central that invoke generateGmpJson or that upload the artifacts to Cisco, so I can't tell whether the script is being used securely or not (or at all). The fact that the insecure behavior is the default one reduces my confidence.

As @Callek alluded to in an earlier comment - we do not use that script as part of builds, signing, or update publishing. Most of the build task is described in stanzas like this, which uses mozharness/scripts/openh264_build.py to make the builds, and pulls the source via https from a pinned revision in a github repository.

My belief is that the checked in hashes may be used in the update server cannot be contacted - but I'm still working to confirm that. If they are, we obviously need to do better here.

[nanonyme]

@bhearsum okay, so what is the right way to discover the hashes? It doesn't seem like it can be https://searchfox.org/mozilla-central/source/toolkit/content/gmp-sources/openh264.json as it is about OpenH264 1.8.1.1 which is at this point ancient. https://aus5.mozilla.org/update/3/GMP/120.0/20200518093924/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.18363.1016%20(x64)/default/default/update.xml does indeed point to more modern release (2.3.2) but not latest either (2.4.0).

[bhearsum]

@bhearsum okay, so what is the right way to discover the hashes? It doesn't seem like it can be https://searchfox.org/mozilla-central/source/toolkit/content/gmp-sources/openh264.json as it is about OpenH264 1.8.1.1 which is at this point ancient. https://aus5.mozilla.org/update/3/GMP/120.0/20200518093924/WINNT_x86_64-msvc-x64/en-US/release/Windows_NT%2010.0.0.0.18363.1016%20(x64)/default/default/update.xml does indeed point to more modern release (2.3.2) but not latest either (2.4.0).

As far as I know, we're still only pointing Firefox users at 2.3.2. You can find the complete list of hashes for all platforms in https://aus-api.mozilla.org/api/v1/releases/OpenH264-2.3.2. (And you can find out the latest OpenH264 that we're shipping via the mapping field on https://aus-api.mozilla.org/api/v1/rules/17820.)

[nanonyme]

Okay, then what @fluffy said is clearly bogus and this issue should be open.

[nils-ohlmeier]

I think the issue is a little two fold: for Mozilla this isn't really a big issue as they simply know the hashes of the binaries they build and bake them either into Firefox or put them on secured/trustworthy server for Firefox to download from.

But for everybody else who hasn't build the binary them self it's a little bit more tricky.

[nanonyme]

@nils-ohlmeier no, the problem is Cisco says Mozilla builds the binaries but binaries differ from what Mozilla builds.

[mcatanzaro]

So for avoidance of doubt, fluffy seems to be mistaken with regards to the release process. The binary releases appear to most likely be built by Cisco, not by Mozilla. Certainly there doesn't appear to be any known way to download hashes for them from Mozilla.

[nils-ohlmeier]

@mcatanzaro @bhearsum provided the URL for downloading the hashes which Mozilla uses. It is https://aus-api.mozilla.org/api/v1/releases/OpenH264-2.3.2 (and that is a HTTPS URL and the hashes are using modern, stronger hash algorithms).

Now Mozilla isn't always using the latest version of OpenH264. That is a completely different problem.

I don't know who build the 2.4.0 binaries. Maybe the more secure solution would be then for Cisco to not publish any binaries, until Mozilla builds them and provides them to Cisco?

@bhearsum I'm pretty sure from when I used to work on this that Bryce told me that the build in hashes in Firefox are only used if the update services are not reachable. So who ever did OpenH264 updates recently forgot to update these internal hashes (which if I remember it correctly is a pretty common issue).

[Erick555]

Even for older releases like 2.3.1 Cisco doesn't present those mozilla builds anywhere but only shows their own. The binary difference as checked with diffoscope is massive.

Is the recommendation to use those mozilla builds instead of ones listed in openh264 github release page? Does this comply with Cisco binary license?

[mcatanzaro]

(We are now pretty confident that the binaries produced by Mozilla for Firefox simply do not correspond to the binaries released by Cisco.)

[nils-ohlmeier]

I guess who's binaries you trust more depends on your individual perspective. The Mozilla binaries do have better hashes which can be fetched from an HTTPS URL.

My personal guess is that Cisco in general pays the MPEG royalties for all it's encoders. So as long as the encoders are distributed by Cisco they should be covered. But I'm neither a lawyer nor do I speak for Cisco.

[mcatanzaro]

I don't mean I don't trust the Mozilla binaries. I just mean they are not the binaries that Cisco is releasing. I think we've established this by now. :)

[flossposse]

I guess who's binaries you trust more depends on your individual perspective. The Mozilla binaries do have better hashes which can be fetched from an HTTPS URL.

yeah, I don't think anyone is suggesting that Mozilla's or Cisco's binaries are more or less trustworthy than the other's. It's about which delivery method lets you be confident that what you received is what they built.

I'd love it if both Mozilla and Cisco could deliver hashes (though preferably not md5) over https.

[mcatanzaro]

I'd really like to see the GitHub releases fixed. E.g. https://github.com/cisco/openh264/releases/tag/v2.4.0 says "All the binaries have been digitally signed" but in fact they are just MD5 hashes, not digital signatures, and any man-in-the-middle attacker who can change the binaries can also just as easily change the hashes. And even if the hashes were delivered securely, MD5 hasn't been safe for nearly 20 years now.

We have detected that the OpenH264 2.4.1 binaries have unexpectedly changed after release for at least x86_64 and aarch64. The binaries that I downloaded on February 2 do not match the binaries distributed today, February 8. I don't know when they changed. Here are sha256sums and sizes for the files distributed on February 2. The x86_64 libopenh264-2.4.1-linux64.7.so.bz2 that I downloaded on February 2 was 633714 bytes and had a SHA-256 hash of 47107cfc244f2d4c87d1f360f682024d7d50322d8c66fb4a1ae69bd976675285. The same file downloaded today, February 8, is 633796 bytes and has a SHA-256 hash of ca413853d99d960ebcd5ae5b4c65a85bb2b5598e9042e64700a9f4b737ca3a3f.

Needless to say, this should never happen without some sort of official announcement telling people to expect it. I'm assuming Cisco has intentionally changed the binaries and that there is no attacker, but absent some sort of announcement, there's no way for us to know.

This incident further reinforces the importance of supply chain security. I was worried about theoretical issues; I certainly didn't expect it to be a real problem in practice. Properly supporting TLS would be the bare minimum to restore confidence; adding GPG signatures would be better. Please stop using MD5 hashes and certainly stop referring to them as "digital signatures."

[torokati44]

I'm assuming Cisco has intentionally changed the binaries

I'm assuming it has to do with this, maybe: #3728

Properly supporting TLS would be the bare minimum to restore confidence; adding GPG signatures would be better. Please stop using MD5 hashes and certainly stop referring to them as "digital signatures."

100% agree.

[torokati44]

About MD5:
https://twitter.com/realhashbreaker/status/1770161965006008570

Here is a 72-byte alphanum MD5 collision with 1-byte difference for fun:

md5("TEXTCOLLBYfGiJUETHQ4hAcKSMd5zYpgqf1YRDhkmxHkhPWptrkoyz28wnI9V0aHeAuaKnak")
=
md5("TEXTCOLLBYfGiJUETHQ4hEcKSMd5zYpgqf1YRDhkmxHkhPWptrkoyz28wnI9V0aHeAuaKnak")

🤡🚨

[mcatanzaro]

About MD5:

I was about to report you as a spammer, but you don't look like a spammer. Did you just paste the wrong link there?

[torokati44]

What, yes, sorry, how did that end up in my clipboard? 😂

[torokati44]

Okay, now I know. Fixed the link. 😅

[fluffy]

The more I read on this more concerned I get. If cisco had compiled these or changed them, the binaries hash would not be in FF and FF fox would fail to load the binaries. Perhaps that is happening.

I wonder if we could get a call together with the people at Mozilla and Cisco and verify all of this was going as expected. I am also sure that if mozilla wants to change to better hashes, GPG, post quntum, whatever, Cisco is in favour of better security and glad to post them.

[fluffy]

nils-ohlmeier has this right. The goal was to set this up so that you only had to trust Mozilla, not Cisco, yet Cisco could cover the MPEG-LA royalty fees for people that wanted to use this. Here is a video that explains how this was set up https://vimeo.com/79578794

It is certainly possible things have migrated from this along the way but I think the goals remain the same.

[Erick555]

Trusting Mozilla may be ok although there is a catch - there is a time lag (counted in months) between Cisco making release and Mozilla added it in their setup. For example Mozilla is still at OpenH264 2.3.1.

That means when someone wanted a bugfix then they need to fix it in source code, wait for Cisco release then wait for Mozilla adding it which is many months of delay even for high priority bugfixes (from user perspective). If Cisco used stronger verification for released binaries then nobody would face dilemma between trust and time.

[mcatanzaro]

Hi @fluffy I have watched your video, but we are very confident that Mozilla is not in any way involved in the Cisco OpenH264 builds that are hosted released here on GitHub. They have their own separate builds, which I presume are used only by Firefox. So I think they have nothing to do with this. I believe there are three separately widely-used builds:

• Official Cisco releases hosted released here on GitHub (these are what we're discussing in this thread)
• Fedora RPM builds, built by Fedora and kindly hosted by Cisco (as a Fedora developer: thank you, it's been a tremendous help)
• Mozilla builds, presumably built by Mozilla and kindly hosted by Cisco? (not relevant to this thread)

My interest in the official Cisco releases is that we're using them for flatpak runtime extensions, which are basically instructions for clients to download the binaries from Cisco.

[Erick555]

Mozilla builds, presumably built by Mozilla and kindly hosted by Cisco? (not relevant to this thread)

Yes, mozila builds are hosted on Cisco servers. With few adjustments it should be technically possible to use them in flatpak runtime but there is release delay catch I mentioned.

Official Cisco releases hosted here on GitHub (these are what we're discussing in this thread)

On GitHub there are just links posted, everything (Official, Mozilla, Fedora) is hosted on ciscobinary.openh264.org

[fluffy]

Than you @Erick555 and @mcatanzaro - I was not understanding the problem. Sorry.

Can someone summarize the problem and we are on this and what it is Cisco ( or Mozilla ) should do. I do not think I can get significantly more Cisco people working on this but I can get Cisco to help make the right thing happen. I would also add that I think we could add more/other external maintainers for this as long as that person was acceptable to both Mozilla and Cisco.

[mcatanzaro]

Thanks. TL;DR:

1. Please fix invalid TLS certificate on https://ciscobinary.openh264.org/ so web browsers can load that link without security warnings. (Should be easy.)
2. Please switch to SHA-256 hashes rather than MD5 hashes for future GitHub releases. Don't call hashes "digital signatures" and don't use "signed" in the filenames. (Should be easy.)
3. Optional: if you want to provide real digital signatures, provide GPG signatures for future releases and post the public key or keyring we should validate the signatures against. This is not essential.

[fluffy]

That does not seem to deal with the problem reported above of "We have detected that the OpenH264 2.4.1 binaries have unexpectedly changed after release for at least x86_64 and aarch64." At this point it is not clear to me who is making the builds or who uploaded the ones that changed. That should deeply concern everyone and gets to why we do not view they way this is working as a secure build chain.

[Erick555]

The changed binaries problem was discussed here. Yes, such accidents combined with lack of secure verification make current state even worse.

[mcatanzaro]

At this point it is not clear to me who is making the builds or who uploaded the ones that changed.

The 2.4.0 and 2.4.1 releases were created by @BenzhengZhang, who I presume works for Cisco. He would know who is making the builds.

The unexpected change to the binaries turned out to be an innocent mistake to fix the version number. The original 2.4.1 binaries improperly reported their version as 2.4.0, and the replacement binaries correctly report 2.4.1. Benzheng has already agreed to avoid replacing previously-released binaries in the future; a better solution would have been to just release 2.4.2.

[ErikCumps]

I don't mind on which issue this gets fixed, as long as it gets fixed. 😊

Browsers are more and more reluctant to connect with plain http sites (like it or not) and there is really, really no point at all in using a TLS certificate for a webserver that is not matching the identity of that server.

So please fix the invalid TLS certificate on https://ciscobinary.openh264.org/, so that web browsers can load that link without security warnings.