Skip to content
This repository has been archived by the owner on Feb 12, 2023. It is now read-only.

Total GPG Failure All 1.13.0 Downloads #4852

Closed
ghost opened this issue Dec 4, 2017 · 20 comments
Closed

Total GPG Failure All 1.13.0 Downloads #4852

ghost opened this issue Dec 4, 2017 · 20 comments
Labels
M-packaging Affected Module: Packaging

Comments

@ghost
Copy link

ghost commented Dec 4, 2017

1.13.0 binaries fail GPG checks.

macOS .dmg has no sig file. It can't be verified.

Windows installers (32 and 64) give BAD signature.

Windows "portable" downloads have no sig files (and crazy filenames too).

@tox-user
Copy link
Contributor

tox-user commented Dec 4, 2017

Where did you download them from?

@sudden6
Copy link
Member

sudden6 commented Dec 4, 2017

Just checked and verified the source from https://github.com/qTox/qTox/releases signatures for the source are correct.

For macOS there are no signatures, since it's built on TravisCI and we can't verify the build process there.

For the Windows installers I just pinged @tux3, because he does the Windows builds.

@sudden6 sudden6 added the M-packaging Affected Module: Packaging label Dec 4, 2017
@ghost
Copy link
Author

ghost commented Dec 4, 2017

Yeah, binaries fail. Source tarballs mostly help Linux/BSD. Average people d/l Windows or macOS binaries.

Now, offering a secure end-to-end crypto tool that cannot d/l validate is, well, pretty darn goofy. Have a dev sign Travis builds by hand. I would assume somehow he'd d/l his latest build while logged into Travis over some TLS or ssh pipe.

Where did you download

https://github.com/qTox/qTox/releases/download/v1.13.0/qTox.dmg
https://qtox-win.pkg.tox.chat/qtox/win64/download
https://qtox-win.pkg.tox.chat/qtox/win64/download-sig
https://qtox-win.pkg.tox.chat/qtox/win32/download
https://qtox-win.pkg.tox.chat/qtox/win32/download-sig
https://qtox-win.pkg.tox.chat/qtox/win64/download-portable
https://qtox-win.pkg.tox.chat/qtox/win32/download-portable

@tox-user
Copy link
Contributor

tox-user commented Dec 4, 2017

@nurupo mentioned there were some issues with signing those builds some time ago: Tox/tox.chat#171

@ghost
Copy link
Author

ghost commented Dec 5, 2017

Thanks, he's wrong in saying

Btw, if someone does hack ... the website ... The only way users would notice ... is if they have previously added our signing key to their keyring

Users can validate by public keyservers independent of any websites a la

gpg --recv-keys 0x31039166FA902CA50D05D6085AF9F2E29107C727 0xC7A2552D0B250F983827742C133203A3AC399151 0xCA9221C5389B7C50AA5F779352A50775BE13DF17 0xDA262CC93C0E1E525AD21C8596775D454B8EBF44 0x2880C860D95C909D3DA45C687E086DD661263264 0xBA7883E22F9D35945BA3376053137C3033F09008 0xAED311349C23A123E5C4AA4B139CA0453DA2D773

Indeed GPG will tell them if they happen to have the keys on their keyrings already. Unless the hackers also compromise the public keyserv network everything will work as it should. These matters have been thought through many times in many places, there is a reason why Linux distros sign their packages with GPG nowadays.

@sudden6
Copy link
Member

sudden6 commented Dec 5, 2017

@nixitixin of course one could manually sign Travis builds, but that is still no guarantee Travis doesn't insert a backdoor (unlikely, but it's the cloud™). Also none of us runs macOS, so we can't even check if the binary does something suspicious.

Windows builds should be signed though and I already notified the person who can do that.

@nurupo
Copy link
Contributor

nurupo commented Dec 5, 2017

@nurupo mentioned there were some issues with signing those builds some time ago: Tox/tox.chat#171

That issue I have mentioned in there has nothing to do with the issue being discussed here.

@nurupo
Copy link
Contributor

nurupo commented Dec 5, 2017

Thanks, he's wrong in saying
Users can validate by public keyservers independent of any websites a la
gpg --recv-keys 0x31039166FA902CA50D05D6085AF9F2E29107C727 0xC7A2552D0B250F983827742C133203A3AC399151 0xCA9221C5389B7C50AA5F779352A50775BE13DF17 0xDA262CC93C0E1E525AD21C8596775D454B8EBF44 0x2880C860D95C909D3DA45C687E086DD661263264 0xBA7883E22F9D35945BA3376053137C3033F09008 0xAED311349C23A123E5C4AA4B139CA0453DA2D773

I'm not wrong, you just misunderstood what I said. In my scenario users get those public keys from the hacked website, so the hackers modified the website replacing the valid public keys with their own, which are also available on public keyservers, so the users would run the gpg --recv-keys for hackers' keys that they find listed on the website and would verify that hackers' binaries are indeed signed by hackers' keys. If it's the first time they get the keys, they have no way of knowing whether they are legitimate or not. However, if they already had the legitimate keys imported before, they would see that the keys listed on the website don't match the ones that they had previously imported, which would ring the bells.

@nurupo
Copy link
Contributor

nurupo commented Dec 5, 2017

Reproduced the verification failure

# gpg --recv-keys 0x139CA0453DA2D773
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 139CA0453DA2D773: public key "qTox Team <barrdetwix@gmail.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1

# gpg --fingerprint 0x139CA0453DA2D773
pub   rsa4096 2016-07-03 [SC]
      AED3 1134 9C23 A123 E5C4  AA4B 139C A045 3DA2 D773
uid           [ unknown] qTox Team <barrdetwix@gmail.com>

# gpg --verify setup-qtox64-1.13.0.exe.asc setup-qtox64-1.13.0.exe
gpg: Signature made Tue Nov 28 21:02:04 2017 UTC
gpg:                using RSA key AED311349C23A123E5C4AA4B139CA0453DA2D773
gpg: BAD signature from "qTox Team <barrdetwix@gmail.com>" [unknown]

# sha512sum setup-qtox64-1.13.0.exe.asc
696e88e94ea09ccb5582f897a40963114be473e7c5c0e49fd4624e0bef95ca05aeca25261ee2e06fc3c06e4d47950616cd0cd61248c86e8d1e42063109cc9a9f  setup-qtox64-1.13.0.exe.asc

# sha512sum  setup-qtox64-1.13.0.exe
34328733ef52145319cc9c853dffe447f71ba7c9c9bfc7d7014ec70bc7c29bba690956592f65c8789b3a86e14e0ff43250c6c387be650f830620688d88023af7  setup-qtox64-1.13.0.exe

@nurupo
Copy link
Contributor

nurupo commented Dec 5, 2017

I have found what the issue is.

https://qtox-win.pkg.tox.chat/qtox/win64/download redirects to to the latest build of qTox on our build server.

wget https://qtox-win.pkg.tox.chat/qtox/win64/download
--2017-12-05 00:46:14--  https://qtox-win.pkg.tox.chat/qtox/win64/download
Resolving qtox-win.pkg.tox.chat (qtox-win.pkg.tox.chat)... 45.79.166.124
Connecting to qtox-win.pkg.tox.chat (qtox-win.pkg.tox.chat)|45.79.166.124|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://build.tox.chat/view/qtox/job/qTox_pkg_windows_x86-64_stable_release/lastSuccessfulBuild/artifact/setup-qtox64-1.13.0.exe [following]
--2017-12-05 00:46:14--  https://build.tox.chat/view/qtox/job/qTox_pkg_windows_x86-64_stable_release/lastSuccessfulBuild/artifact/setup-qtox64-1.13.0.exe
Resolving build.tox.chat (build.tox.chat)... ^C

Those stable binaries are supposed to be built only on releases and stay the same until a new release, but apparently there was a misconfiguration which caused qTox to be rebuilt every time someone pushed to qTox repository.

In the image below you can see 3 builds done in December that shouldn't have been done.

screenshot1

In the image below you can see that it was caused by a push to to the qTox repository, which rebuilt the updater, which in turn caused the setup executable to be rebuilt.

screenshot2

Using the build from November 28th the verification succeeds:

# wget https://build.tox.chat/job/qTox_pkg_windows_x86-64_stable_release/194/artifact/setup-qtox64-1.13.0.exe
--2017-12-05 00:56:54--  https://build.tox.chat/job/qTox_pkg_windows_x86-64_stable_release/194/artifact/setup-qtox64-1.13.0.exe
Resolving build.tox.chat (build.tox.chat)... 104.236.189.201, 2604:a880:1:20::31:5001
Connecting to build.tox.chat (build.tox.chat)|104.236.189.201|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22909839 (22M) [application/x-msdownload]
Saving to: 'setup-qtox64-1.13.0.exe'

setup-qtox64-1.13.0.exe                         100%[======================================================================================================>]  21.85M  3.89MB/s    in 11s     

2017-12-05 00:57:06 (1.92 MB/s) - 'setup-qtox64-1.13.0.exe' saved [22909839/22909839]

# sha512sum setup-qtox64-1.13.0.exe
526c48b9eb4c6b58f24a9f8e37d1b44a6bfca5784bc941c84ed70968cf4e0f475a537a9dc6b5f277fb6a64468c97c05d6040e79bf4903a8a146fceb99f1e754b  setup-qtox64-1.13.0.exe

# gpg --verify setup-qtox64-1.13.0.exe.asc setup-qtox64-1.13.0.exe
gpg: Signature made Tue Nov 28 21:02:04 2017 UTC
gpg:                using RSA key AED311349C23A123E5C4AA4B139CA0453DA2D773
gpg: Good signature from "qTox Team <barrdetwix@gmail.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: AED3 1134 9C23 A123 E5C4  AA4B 139C A045 3DA2 D773

I will try to fix the rebuilding issue on Jenkins now by deleting the newer builds that shouldn't have happened (partially why I have made the screenshots of them, since they will be gone now), this should make the lastSuccessfulBuild/artifact/setup-qtox64-1.13.0.exe point to the November's build. Will report once I'm done and once I verify it working.

@nurupo
Copy link
Contributor

nurupo commented Dec 5, 2017

Should be fixed. The download links now point to the right qTox build. Just re-download the .exe and it should verify with your existing signature file.

@tux3 I have removed the qTox updater from pkg jobs' upstream and also disabled pkg jobs for a good measure, so you will have to enable them when making a new release.

@ghost
Copy link
Author

ghost commented Dec 5, 2017

Thanks for fixing the problem quickly. The fix worked for me on Windows installer exe's. Starting with a fresh blank GPG keyring:

$ gpg --recv-keys 0x31039166FA902CA50D05D6085AF9F2E29107C727 0xC7A2552D0B250F983827742C133203A3AC399151 0xCA9221C5389B7C50AA5F779352A50775BE13DF17 0xDA262CC93C0E1E525AD21C8596775D454B8EBF44 0x2880C860D95C909D3DA45C687E086DD661263264 0xBA7883E22F9D35945BA3376053137C3033F09008 0xAED311349C23A123E5C4AA4B139CA0453DA2D773
gpg: requesting key 9107C727 from hkp server keys.gnupg.net
gpg: requesting key AC399151 from hkp server keys.gnupg.net
gpg: requesting key BE13DF17 from hkp server keys.gnupg.net
gpg: requesting key 4B8EBF44 from hkp server keys.gnupg.net
gpg: requesting key 61263264 from hkp server keys.gnupg.net
gpg: requesting key 33F09008 from hkp server keys.gnupg.net
gpg: requesting key 3DA2D773 from hkp server keys.gnupg.net
gpg: /XXXXXX/XXXXXXXXXX/.gnupg/trustdb.gpg: trustdb created
gpg: key 9107C727: public key "Polshakov Dmitriy Vladimirovich " imported
gpg: key AC399151: no valid user IDs
gpg: this may be caused by a missing self-signature
gpg: key BE13DF17: public key "Alexey Razinkov " imported
gpg: key 4B8EBF44: public key "sudden6 " imported
gpg: key 61263264: public key "tux3/mlkj " imported
gpg: key 33F09008: public key "Zetok Zalbavar " imported
gpg: key 3DA2D773: public key "qTox Team " imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 7
gpg:           w/o user IDs: 1
gpg:               imported: 6  (RSA: 6)


$ wget --trust-server-names --input-file=fetch.txt


$ gpg --verify setup-qtox32-1.13.0.exe.asc 
gpg: assuming signed data in `setup-qtox32-1.13.0.exe'
gpg: Signature made Tue Nov 28 14:02:09 2017 MST using RSA key ID 3DA2D773
gpg: Good signature from "qTox Team "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: AED3 1134 9C23 A123 E5C4  AA4B 139C A045 3DA2 D773


 
$ gpg --verify setup-qtox64-1.13.0.exe.asc 
gpg: assuming signed data in `setup-qtox64-1.13.0.exe'
gpg: Signature made Tue Nov 28 14:02:04 2017 MST using RSA key ID 3DA2D773
gpg: Good signature from "qTox Team "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: AED3 1134 9C23 A123 E5C4  AA4B 139C A045 3DA2 D773

The Windows portable builds still have crazy filenames and lack sigs as does the macOS dmg file. On that security discussion, we're getting into weeds. Yes I grasped all variant scenarios implied by your remarks, and there are more than you think. Better men than us have thought them through. Ask why GPG is used to sign any binaries anywhere.

It's not just a matter of uploading fake keys, a spoof website, and jiggered apps. There is a web of trust involved. And I don't just mean technical GPG trust sigs, but many eyes watching websites, never mind authors.

As "the enemy" I'd just hack a dev with a 0-day and insert spoofed binaries from his home PC to the unmodified qTox website. Right now that's easy for unsigned binaries. Please worry about this immediate threat over some hypothetically possible GPG caper as complex as Ocean's Eleven. Harden the build chains, github passphrase entropies, network firewalls, and such like.

I don't see crypto attack theory as in scope. We just need GPG sigs to work as on all other open-source projects. The perfect is the enemy of the good. Whether or not releases can be hacked, "make them work for it." It's possible to sign webpages, post sigs in side channels, etc. Many open-source project release announcements include sigs in the announcement e-mail.

Thanks

@tox-user
Copy link
Contributor

tox-user commented Dec 5, 2017

@sudden6 If Travis can insert a backdoor, can't GitHub do that too somehow?

@nixitixin nobody is questioning the importance of GPG, but if I understood correctly those remaining builds will not be signed until the problem @nurupo mentioned in the issue I linked is resolved. Don't we need to know at least one person from the web of trust for it to work? We need to know someone who signed the key. Otherwise we won't know if the key from the keyserver wasn't uploaded by the attacker.

@nurupo
Copy link
Contributor

nurupo commented Dec 5, 2017

The Windows portable builds still have crazy filenames and lack sigs

They are not really portable builds, they are unstable "nightly" builds based on the latest development progress. Whenever someone pushes code change to qTox git repository, it builds a new "nightly" qTox. (I put nightly in quotes because the build is done not once per day overnight, but every single time someone pushes code, so it can be rebuilt multiple times a day or once a week). I have complained to qTox people a few times that they shouldn't be calling those "portable builds", that's misleading, not to mention that they still use system-wide qTox settings (%AppData% directory) instead of saving them somewhere in a local directory, meaning that they are not portable in the sense that if you run such qTox off a USB drive, your tox profile and other settings will be saved on the computer's system drive and not on the USB drive. Maybe I should just open a PR with the change, bringing this up takes more effort than just fixing it myself.

The nightly builds do lack signatures as we haven't developer any signing process for them. They were setup mainly for testing purposes rather than serious use, which is that the stable releases are for, so we were fine with having those unsigned. It would be nice to have them signed and it's something that I want eventually to do, but it's not very simple. The builds are automated, so the signing would have to be automated too, and if the website would link to the nightly builds (which it currently does), we might need to update the website every time a build is made as it's a static page website. In fact, the website would need to be updated through GitHub so that the source and the actual website stay in sync, meaning we'd need to setup a GitHub bot, etc. Alternatively, we could make the website a bit more dynamic. We also can't sign the nightlies on the build server directly due to some security concerns, so the nightlies would need to be automatically uploaded on some other signing server, meaning we'd need to develop some Jenkins plugin or write an inodewatch+rsync application in C... anyway, it's quite complicated and not a 5-minute task, if we do this we want to get it done right and not rush a half-complete solution. It's also a bit low priority as we have more important things and tings that require our immediate attention on our plate and we are quite undermanned at the moment to deal with everything.

@nurupo
Copy link
Contributor

nurupo commented Dec 5, 2017

@sudden6 If Travis can insert a backdoor, can't GitHub do that too somehow?

It can't if you sign your commits and releases, since it would invalidate the signatures. You could even sign your issue messages if you wanted to, just like you pgp sight email. In the worst case it can only deny addition of new commits/releases or providing (targeted?) users outdated commits/releases.

@ghost
Copy link
Author

ghost commented Dec 5, 2017

Good clarifications. I'd remove nightlies from the main d/l page. Just footnote a link on it to some other page for nightly builds. Those aren't important to sign. Most developers compile from git anyway if they're doing nightly work like that. So the trust is basically placed on github.

When Travis emits a final build, not much more need happen. A dev signs .exe's and .dmg's by hand and uploads .asc files. Automation isn't worth it. Releases happen every 6-8 weeks and signing is a 2-minute job.

The .asc files needn't appear in the same web page as binaries, if that's hassle. URLs are URLs and can point anywhere from the main d/l page.

Thanks

@tox-user
Copy link
Contributor

tox-user commented Dec 5, 2017

How do you verify that the binary you got from Travis is what you want it to be? That it doesn't have a backdoor?

@nurupo
Copy link
Contributor

nurupo commented Dec 5, 2017

Having reproducible builds is one way, it's a build property that makes the same source code result in the same bit-by-bit identical binary. Someone trusted could build it on their own macOS system and verify that the resulting binary is bit-by-bit identical to the one Travis has produced, meaning that Travis didn't backdoor it. Builds are not reproducible by default though, you need to put quite some effort to make them be reproducible. I'm also not sure if it's possible to do with macOS, as reproducible builds are generally made in a VM with specific compiler version, library version, user name, system time, etc. so there is a legal question of running a non-free OS like macOS in a VM.

So given that qTox builds are not reproducible, and even if they were no one has a macOS system to check if Travis has modified the build, no one can verify the binaries from Travis, and without being able to verify you are left with the only choices of either blindly trusting or not trusting it.

Similar applies to the Linux packages setup on the OBS.

Our build server where the Windows qTox is built it a bit more trustworthy than Travis or OBS since @tux3 has full root access to it and he can exemine it whenever he wants. It's still not as good as having reproducible builds though.

@ghost
Copy link
Author

ghost commented Dec 5, 2017

no one has a macOS system

Previsouly addressed in closed issue #4095. Reproducible building is overkill right now. Someone just needs to sign .dmg's that Travis emits based on his trust in them, a better quality trust than ours, to stop MITM attacks if nothing else. A sig at least guarantees we're getting the binary .dmg that qTox devs expect us to get.

Over-focus on nightlies is distracting from basics. Developers know their way around these matters, average users don't. Focus on average users. The nightly jazz also confuses web pages having both "portable" (so-called) and "nightly" (so-called) listings right next to stable official releases. That's not the way to offer apps to the public. Send developers elsewhere and keep things simple and straight for everyone else.

Thanks

@anthonybilinski
Copy link
Member

The core issue of "Total GPG Failure All 1.13.0 Downloads" seems totally resolved, and the issue of signing generated binaries has been discussed in #5001. Going to close as #5001 can continue the remaining discussion from here.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
M-packaging Affected Module: Packaging
Projects
None yet
Development

No branches or pull requests

4 participants