New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: x/crypto: deprecate unused, legacy and problematic packages #30141

Open
FiloSottile opened this Issue Feb 9, 2019 · 60 comments

Comments

Projects
None yet
@FiloSottile
Copy link
Member

FiloSottile commented Feb 9, 2019

x/crypto is meant to follow the same philosophy as the rest of the Go cryptography standard libraries: safe, useful subsets of widely used and robust primitives and protocols. Some packages in x/crypto don't clear that bar anymore, and I propose we freeze and deprecate (but not remove) them.

The intended outcomes are

  • reducing the maintenance burden, redirecting resources towards more useful work
  • avoiding confusion for users who consider inclusion in x/crypto to be an endorsement
  • encouraging community maintained replacements where needed

All open and future issues and CLs that are not security relevant will be closed linking to this issue.

/cc @agl with whom we discussed this at RWC, and @golang/proposal-review for approval

Why

There are different reasons for deprecating different packages, and I give a short explanation for each of them below, but here's a summary.

  • Some protocols simply don't have a common enough subset, with each user needing something different from it. The result is that our package is not useful as-is to anyone, and forks proliferate more than if we didn't have an x/crypto package at all.
  • Some packages simply can't reach the bar of safe-by-default (anymore) because the underlying protocol or primitive is broken.
  • Some packages require a maintainer that has deep knowledge of the many relevant specs and the state of the ecosystem. While it makes sense for the Go team to allocate the resources for, say, TLS and X.509, it doesn't for others.
  • A few packages are simply not used enough to justify the maintenance effort, and their inclusion in x/crypto is confusing to users.

What packages

  • blowfish/ — '90s cipher fundamentally broken due to its block size, its main use is as a bcrypt/ internal and it has no optimized implementation
  • bn256/ — a broken elliptic curve that does not provide 128 bits of security anymore, with a community fork (github.com/cloudflare/bn256)
  • cast5/ — another '90s cipher, only used in PGP
  • md4/ — ancient broken hash function, only useful as target practice for learning how to generate hash collisions on a laptop (which is very fun!)
  • openpgp/
    • the protocol predates modern cryptography engineering, which is still causing severe vulnerabilities, and any new project should rely on alternative modern solutions
    • the ecosystem never solved its major UX and key management issues
    • most importantly, this implementation is aging and incomplete because the specification is sprawling and there is no common subset to target; apparently everyone who's relying on it made their own fork to add features or fix issues, which is a much worse outcome than not having a golang.org/x/crypto package at all
  • otr/ — a messaging security protocol from 2000's, mostly obsoleted by the Signal double ratchet and other messaging protocols like OMEMO and MLS, and a very rarely used package
  • ripemd160/ — a hash function from the '90s only used by Bitcoin, with no optimized implementation
  • tea/, xtea/ — ancient block ciphers, apparently used by one single package in the ecosystem
  • twofish/ — AES finalist, not used in any popular protocol, with no advantages over AES
  • xts/ — a block mode that should only ever be used for full disk encryption, which unsurprisingly no one is implementing in Go, and I honestly didn't even know it was in here

I was also hoping to kill pkcs12/ because it's a legacy file format that relies on primitives even worse than the ones listed above, and the package only does decoding while a better fork on GitHub also does encoding, but it has way too many users so that's only been frozen instead.

@gopherbot gopherbot added this to the Proposal milestone Feb 9, 2019

@gopherbot gopherbot added the Proposal label Feb 9, 2019

@davecgh

This comment has been minimized.

Copy link

davecgh commented Feb 9, 2019

I would prefer to see ripemd160 maintained as it is used by more cryptocurrencies than just Bitcoin. Honestly, Bitcoin is a pretty big deal on its own and would be enough reason to maintain it in my opinion.

Additionally, I performed a quick survey via godoc.org importers and it shows that the ripemd160 package is imported by at least 511 packages at the current time, including many widely known pieces of software such as Btcsuite, Discord, Decred, Ethereum, and Ripple.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

FiloSottile commented Feb 9, 2019

@davecgh I am not aware of any reasons to pick RIPEMD-160 for a new design, so the fact that cryptocurrencies inherited it from the Bitcoin paper is not a good enough reason not to deprecate it.

I don't expect it will require any maintenance anyway—it's a hash function, the package is almost certainly fine as is—but deprecating it will send the message to users not to look there if they just need a hash function, and communicates that we would for example not take an optimized assembly implementation for it.

@aksdb

This comment has been minimized.

Copy link

aksdb commented Feb 9, 2019

Just picking two examples to illustrate my opinion:
md4 is used in the NTLM protocol.
pkcs12 is used to store certificates in a lot of real world applications.

Both are dated, one is a PITA. Yet I think x/crypto should not be reduced to state of the art ciphers/protocols since we are not always developing self contained systems where we can safely choose the best available tech, we also have to connect to legacy systems or simply not-that-well-developed third-party systems.

Also I think that especially the pretty self contained and matured algorithms like MD4 don't suffer much from "being unmaintained". I mean: what would you maintain? The algorithm is stable (although "broken"). We cannot do anything about attacks on the algorithm (apart from refraining from future use).

I understand your point with OpenPGP. Though it might also be an option to try and get one of the forks to be reintegrated and actually maintain that.

If a living protocol (that still receives updates; has a moving spec) rots ... yeah, throw it out. The sophisticated implementations however, that are simply not "maintained" because there is nothing to maintain there ... I would like to keep them.

@kaepora

This comment has been minimized.

Copy link

kaepora commented Feb 9, 2019

  • otr/ — a messaging security protocol from 2000's, mostly obsoleted by the Signal double ratchet and other messaging protocols like OMEMO and MLP, and a very rarely used package

The reasons given for obsoleting the otr package are perfectly sensible, but if it's not too much of a burden I would argue for nevertheless keeping this package: this is because in the XMPP world, the OTR protocol is still used as the standard secure messaging protocol and Signal has virtually no adoption there.

Aside from this argument I admittedly have nothing else with which to claim that keeping otr is relevant.

@tredoe

This comment has been minimized.

Copy link

tredoe commented Feb 9, 2019

  • blowfish can not be compared with AES since the former has a fixed block size of 64 bits; the later, just like Serpent and Twofish, have block sizes of 128 bits.

  • The fact of that a algorithm been used into a public or known project doesn't mean that it doesn't been used at private projects. At my case, I've used mainly Ripemd160 instead of both MD5 and SHA1 due to greater security, and I use it instead of SHA2 according to the requirements about security and perfomance that my project needs.

  • Twofish and also Serpent have greater security than Rijndael, but the later was the AES winner due to greater performance. A lot of people not only chooses an algorithm due to performance else by greater security.

  • The block mode XTS is a block mode very safe, only recommended for full-disk encryption. So, do you want to remove it because you didn't know about it, and the coders of open source are not using it, don't you?

    They are coders that build commercial and/or private projects.
    At my case, I'm using it like alternative to cryptsetup.

@jdevelop

This comment has been minimized.

Copy link

jdevelop commented Feb 9, 2019

I'd rather keep openpgp (and would consider making it compatible with GnuPg2 ).

Still the problems there are quite surprising

Would be cool to have someone to take a look.

@charles-dyfis-net

This comment has been minimized.

Copy link

charles-dyfis-net commented Feb 9, 2019

I'd similarly be very disappointed to see openpgp removed. Frankly, the implementation even as it exists is much higher-quality than the upstream GnuPG one, incomplete as it may be; removing it would thus be a loss to the community as a whole.

Admittedly, I'm one of those folks with a private fork, but I have approval from my employer's legal team to submit our changes upstream; if getting off my arse and doing so is the cost of keeping it in-tree (and thus being sure that the existing functionality continues to work with future versions of Go), I'm happy to do so.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 9, 2019

@tredoe @charles-dyfis-net, this issue is about freezing+deprecating them, not removing them.

@dominikh

This comment has been minimized.

Copy link
Member

dominikh commented Feb 9, 2019

@bradfitz To be fair, deprecating them has almost the same effect as removing them; people will look elsewhere for alternatives that aren't marked as deprecated. Which I imagine is the intended effect. Unfortunately, the alternatives may not always be of higher quality. For example, I'd much rather see people use x/crypto/md4 over some other, random md4 implementation.

I don't think the "these algorithms are broken" holds as an argument. A lot of legacy still relies on them, unfortunately. As such, we should rather look at the cost of not deprecating a package. One can of course argue that another well-organized party that isn't Go should maintain legacy packages.

Have we explored why these packages made it into x/crypto in the first place?

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 9, 2019

To be fair, deprecating them has almost the same effect as removing them

I disagree. The effects are quite different: removing them breaks users, and deprecating them does not. Deprecating them is just adding a comment saying, "Yo, new applications shouldn't use this, and also we're not adding new features so don't send PRs."

people will look elsewhere for alternatives that aren't marked as deprecated. Which I imagine is the intended effect.

Sounds like the intended effect is reducing maintenance cost by not accepting changes to add features or optimize the performance.

@aksdb

This comment has been minimized.

Copy link

aksdb commented Feb 9, 2019

I think the "deprecated" marker would move the same thinking you have here down the line. Your argument for deprecation (or removal) is that the code is old and the algorithms should not be used. While the code is actually most likely still fine and just doesn't need any maintenance (if it does, someone will report it and or create a PR).
If it's marked "deprecated" it will lead to managers/decision makers/etc. thinking "oh, that code is not maintained, we cannot use it. go find something else".
I had discussions like that myself. A library for a microcontroller we were working with was over 7 years old and did not have a single change. It took a considerate amount of time to convince my superior that this did not matter since a) the code was well documented b) had good test coverage c) the microcontroller in question was that old as well - there was simply nothing that could have changed in the meantime.
The same holds for the algorithms in question. MD4, cast5, blowfish, twofish, etc. will not change. They may not be a good choice for cryptographic strength ... but they may still be the right choice to implement a specific interface. And then it's better to have a central source that does not imply "we will get rid of this soon" instead of a bunch of third party sources.

@dominikh

This comment has been minimized.

Copy link
Member

dominikh commented Feb 9, 2019

If it's marked "deprecated" it will lead to managers/decision makers/etc. thinking "oh, that code is not maintained, we cannot use it. go find something else".

Case in point: dominikh/go-tools#413

Everywhere in the standard library, deprecated means "stop using this, move to an alternative".

@FiloSottile

This comment has been minimized.

Copy link
Member Author

FiloSottile commented Feb 9, 2019

Remember that deprecation comes with a message, so we can explain the reasons and implications for each package individually, and I'll make sure to provide as much context as possible.

There will be packages that say "this algorithm is broken [or superfluous] because X and should not be used except for compatibility with legacy systems". This does not call for an alternative implementation, but it warns people off using it, and recommends moving to non-broken things.

Other packages will say "this implementation is incomplete and unmantained, consider an alternative protocol like X, or a community maintained package". This hopefully leads to a centralized target for maintenance efforts which does not involve our resources.


I am replying to a few points individually below, before commenting please check if your point was already made above, to prevent this from becoming a giant thread nobody reads or feels like participating in.

I think that especially the pretty self contained and matured algorithms like MD4 don't suffer much from "being unmaintained".

Indeed, deprecating them comes at near-zero cost because there are not going to be new features and bug fixes. But the message to developers about not using them is important.

in the XMPP world, the OTR protocol is still used as the standard secure messaging protocol and Signal has virtually no adoption there

Ack, but I'm afraid that's not enough to justify the Go project maintaining an implementation. I would probably not even accept a Signal protocol implementation in x/crypto.

blowfish can not be compared with AES since the former has a fixed block size of 64 bits;

Good point! That makes it fundamentally vulnerable to Sweet32, updated the proposal.

The fact of that a algorithm been used into a public or known project doesn't mean that it doesn't been used at private projects.

I use usage in public projects as an indicator of popularity and therefore usefulness to the ecosystem, not as proof that nobody uses the package at all, and that's one part of the tradeoff decision on whether it's worth the resources to maintain.

Twofish and also Serpent have greater security than Rijndael

AES-256 is secure enough for Top Secret data in NSA's Suite B, I don't feel the need to carry anything else to fill a security gap.

The block mode XTS is a block mode very safe, only recommended for full-disk encryption.

XTS is quite bad in fact. See also a better tweakable cipher.

Admittedly, I'm one of those folks with a private fork [of openpgp], but I have approval from my employer's legal team to submit our changes upstream

Almost half of all issues open against x/crypto packages are for x/crypto/openpgp, and a good chunk of all CLs are for it as well. I'm not blaming the community for not contributing back here, I blame the protocol for requiring too much maintenance to properly implement. It just doesn't make sense for us to put in the effort to do the review and design work to bring this package to a state in which we are happy about it. Hopefully the community will centralize on a fork.

I'd much rather see people use x/crypto/md4 over some other, random md4 implementation.

MD4 is a bad example, because it's literally as broken as a hash function can be. Go is a memory-safe language, so a random implementation off the street can't introduce an RCE without really trying, and the next worse thing a hash function from the last 20 years can have is a collision, and well, I might still have some Python to generate one on a laptop.

I don't think the "these algorithms are broken" holds as an argument. A lot of legacy still relies on them, unfortunately. As such, we should rather look at the cost of not deprecating a package.

The cost includes developers being misled into thinking that a primitive or protocol can serve their needs because it's in x/crypto, diluting the safety properties of using the standard library.

Have we explored why these packages made it into x/crypto in the first place?

Some to support OpenPGP, some have always been there from before there was a conscious effort to limit what was in x/crypto, some were reasonable at the time and aren't anymore, some were a mistake.

Everywhere in the standard library, deprecated means "stop using this, move to an alternative".

If you are using any of the above, I would like you to stop using it and move to an alternative. Sometimes an alternative implementation, sometimes an alternative algorithm. If you know you can't because of legacy compatibility reasons, at least I made you aware of it, and made sure the question got asked at review time, and the answer hopefully documented.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 10, 2019

Having seen cryptography mismanaged in the standard libraries of many other languages I use, this seems like a great move to me, and a breath of fresh air by comparison. Cryptography in the standard library should be held to a high bar, and I'm glad to see that bar enforced.

In languages with a "batteries included" standard library like Go's, the ecosystem is largely shaped by what's available in the standard library. Go has pretty much the best cryptography ever seen in a language's standard library with a few exceptions, and this proposal covers most of the exceptions.

For implementations of legacy protocols, I think users are almost certainly better off with something out-of-tree. This not only discourages accidental use in new projects, but also provides greater agility in the event of a security incident, as the response does not involve a new release of the language/standard library. Deprecating these algorithms from the standard library isn't a value judgment on their overall utility, just a signal that people should move to out-of-tree implementations. The remaining concern there is that they will lose the reluctant maintainers these algorithm implementations previously had. If the algorithms are truly useful and important, I think you can find new maintainers, and ones who aren't reluctant. Otherwise, it's entirely possible they're not as useful as they're claimed to be.

The remaining arguments seem to fall into a category I call "unusual requirements". For things like symmetric ciphers and hash functions, there are huge reasons to prefer algorithms like AES and SHA-2 over things like Serpent, Twofish, or RIPEMD-160, namely AES has hardware acceleration through a number of different platform-specific instructions like AES-NI which provide better performance and sidechannel resistance. In the case of RIPEMD-160 it has only 80-bits preimage resistance which is considered rather weak by today's standards. The specific security and performance requirements (much less threat model) weren't stated in any of these cases, but in my opinion as a cryptography professional specializing in symmetric encryption I think the claims made by people alleging they selected these algorithms due to "unusual requirements" are, to put it bluntly, factually inaccurate.

tl;dr: eliminating junk cryptography from the Go stdlib will help improve the overall security of the ecosystem, so I say go for it.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Feb 10, 2019

... as the response does not involve a new release of the language/standard library.

@tarcieri, this bug is about x/crypto, which is not the standard library.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 10, 2019

@bradfitz well don’t I look foolish. Mea culpa

@claucece

This comment has been minimized.

Copy link

claucece commented Feb 10, 2019

Hey!

Sorry for jumping into the discussion.

  • otr/ — a messaging security protocol from 2000's, mostly obsoleted by the Signal double ratchet and other messaging protocols like OMEMO and MLP, and a very rarely used package

Well, with the upcoming OTRv4 protocol, I don't how true this will still hold. There is a lot of people now interested in using OTRv4 again and implementing it on different clients; so I don't if it will be a shame to loose it here. That said, for the most people who are actually using OTR in golang, they actually use this library instead: https://github.com/coyim/otr3

I also don't know the state of OTR in golang (if it is fully implemented). I can check if needed.

I don't know what MLP means, is that MLS?

Btw, here some useful links around OTRv4, in case someone wants to check its interest/proposal:

Hope this information helps :)

@FiloSottile

This comment has been minimized.

Copy link
Member Author

FiloSottile commented Feb 10, 2019

That said, for the most people who are actually using OTR in golang, they actually use this library instead: https://github.com/coyim/otr3

I'm happy to hear there is work on the protocol, and that there is a community package for the previous version. That suggests deprecating the x/crypto one would be good for the ecosystem, letting it focus on modern and maintained implementations.

I don't know what MLP means, is that MLS?

Yes, thanks, fixed.

@tredoe

This comment has been minimized.

Copy link

tredoe commented Feb 10, 2019

blowfish can not be compared with AES since the former has a fixed block size of 64 bits;

Good point! That makes it fundamentally vulnerable to Sweet32, updated the proposal.

Simply because the attack is possible doesn't mean it is particularly easy to carry out. For Sweet32, the attacker needs to be able to monitor traffic passing between the user and a vulnerable website, as well as control JavaScript on a web page loaded by the user's browser. It would take about 38 hours to collect hundreds of gigabytes of data necessary to decrypt the authentication cookie.

https://blog.securityevaluators.com/what-is-sweet32-4a62dca90296

The fact of that a algorithm been used into a public or known project doesn't mean that it doesn't been used at private projects.

I use usage in public projects as an indicator of popularity and therefore usefulness to the ecosystem, not as proof that nobody uses the package at all, and that's one part of the tradeoff decision on whether it's worth the resources to maintain.

Fair point

Twofish and also Serpent have greater security than Rijndael

AES-256 is secure enough for Top Secret data in NSA's Suite B, I don't feel the need to carry anything else to fill a security gap.

Although NSA has categorized this in Suite B, they have also recommended using higher than 128-bit keys for encryption. So, should be also removed or deprecated AES-128?

Besides of that, and althought I wouldn't assume that NSA has cracked AES ciphers, I would assume that most crypto systems that use AES have implementation flaws that the NSA could exploit when they want throught its own data center, because the NSA are actively subverting security on the standard level or in collusion with software developers.

The block mode XTS is a block mode very safe, only recommended for full-disk encryption.

XTS is quite bad in fact. See also a better tweakable cipher.

At the first, in practice, many encrypted filesystems already use encryption that's strictly worse than XTS, and they generally use the same block-based model as full disk encryption for exactly the same performance reasons.

Besides, the title of such article is confusing because many people will only ever encounter XTS when setting up full disk encryption with dm-crypt, where they'll be presented with the choice between XTS and CBC-ESSIV.

And finally, it's also not saying that you shouldn't use XTS for full disk encryption. In fact it seems to say it's probably OK for full disk encryption: "It’s certainly better than ECB, CBC, and CTR for FDE. For the crappy job we ask it to do, XTS is probably up to the task."

@DrWhax

This comment has been minimized.

Copy link

DrWhax commented Feb 10, 2019

I would agree that getting rid of otr(which I believe is v2?) from x/crypto makes sense, there's an otrv3 implementation(https://github.com/coyim/otr3) in Golang and a partial implementation of otrv4 in Golang(https://github.com/otrv4/otr4) underway.

As we've been discussing internally in some calls with Ian Goldberg, we'd like to get rid of v2 in favor of v3 and especially v4, so, keeping the otrv2 implementation in x/crypto doesn't make much sense.

@claucece

This comment has been minimized.

Copy link

claucece commented Feb 10, 2019

Yes, I agree with @FiloSottile and @DrWhax . We will surely develop OTRv4 in golang, and it will maintained; so deprecating the old version in golang libs (which might be in between v2 and v3) makes sense. In the future, if wanted, we can implement OTRv4 in golang libs itself. Thanks!

@gtank

This comment has been minimized.

Copy link

gtank commented Feb 10, 2019

I can confirm that well-meaning but uninformed developers choose equally randomly among the options in crypto/ and x/crypto, and therefore I strongly support the proposal. In fact, I gave a GopherCon talk once about exactly this problem.

Deprecating an algorithm in x/crypto isn't the same as banishing it from the Earth, it just means Go's very few crypto reviewers will have more time to focus on things that are/should be widely used.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Feb 13, 2019

This proposal should be put on hold for now.

We shouldn't remove anything from x/ until (1) modules are enabled by default and (2) we have started tagging versions. Once we have started tagging versions, we can discuss removing things; modules will mean users don't automatically get the latest copy, and the tagged version will make it easy to record that what exactly they depend on.

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Feb 13, 2019

@rsc perhaps I'm missing something, but it appears that removal isn't part of the proposed plan here:

and I propose we freeze and deprecate (but not remove) them

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Feb 13, 2019

@bradfitz and @andybons point out that I misunderstood and that this proposal is only about marking things deprecated, not removing them. Sorry.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Feb 13, 2019

Taking md4 as an example, obviously that should be marked as completely broken and something no one should use except under extreme duress. It's funny that crypto/md5 says

r$ go doc md5
package md5 // import "crypto/md5"

Package md5 implements the MD5 hash algorithm as defined in RFC 1321.

MD5 is cryptographically broken and should not be used for secure
applications.

but md4 does not. Obviously it should. Adding that kind of documentation note doesn't require a proposal.

Similarly, x/net/websocket says:

r$ go doc golang.org/x/net/websocket
package websocket // import "golang.org/x/net/websocket"

Package websocket implements a client and server for the WebSocket protocol
as specified in RFC 6455.

This package currently lacks some features found in an alternative and more
actively maintained WebSocket package:

    https://godoc.org/github.com/gorilla/websocket

That's another very clear message that seems fine.

But it seems like a few are not quite in that camp. OpenPGP for example, isn't cryptographically broken, right? I'm less sure about marking that package as deprecated. Is there an established fork that we should point to instead? Or should we invite them to own x/crypto/openpgp?

@FiloSottile

This comment has been minimized.

Copy link
Member Author

FiloSottile commented Feb 13, 2019

@rsc I want to send a stronger message than just adding a line to some package-level docs that nobody will pay attention to (at the very least for existing dependencies). I want developers to make an active, hopefully justified, choice to ignore a tooling-provided flag.

If you need some WebSocket feature you'll know, and if it's not in golang.org/x/net/websocket, you might look at the docs and learn where to go look for it. If you are using a bad hash, it will look and feel like a perfectly good hash.

But it seems like a few are not quite in that camp. OpenPGP for example, isn't cryptographically broken, right? I'm less sure about marking that package as deprecated. Is there an established fork that we should point to instead? Or should we invite them to own x/crypto/openpgp?

OpenPGP is legacy, brittle, and very high maintenance to implement. I discourage any new system from using it, and I don't think it's worth any of our limited resources to maintain. If it were only for lack of resources, I would be seeking an owner for x/crypto/openpgp, but given that it's not something people should be encouraged to use, I don't think that would be a service to our users.

I expand on this in the "Why" section above, under the "openpgp/" bullet, and at #30141 (comment).

@claucece

This comment has been minimized.

Copy link

claucece commented Feb 14, 2019

I just wanted to briefly pointed out that while it is true that there are vulnerabilities surrounding PGP (specifically, the CBC/CFB Gadget Attack -although, the current draft of OpenPGP RFC4880 (bis05) reflect the recommendation of the authors to deprecate the SE packet type and that modified ciphertexts should not be displayed-, as Direct Exfiltration is more around abusing vulnerabilities in Apple Mail, iOS Mail and Mozilla Thunderbird), some libraries might depend on this pgp to work. Take the case of gopass, which is widely used. I agree on that if this is an unmaintained package, it should be good to deprecate it; but maybe giving a solution to projects that use it will be in place.

  • the protocol predates modern cryptography engineering, which is still causing severe vulnerabilities, and any new project should rely on alternative modern solutions

I'm not aware of any alternative modern solutions to pgp that are widely deployed.

I think that deprecating is a right approach; but maybe with giving alternatives as well :)

My two cents :)

@gtank

This comment has been minimized.

Copy link

gtank commented Feb 14, 2019

I don't think there is a direct modern alternative to PGP, because the modern alternative is to use more specific solutions to your problem. For example, here's two common uses of PGP and some modern, maintained alternatives:

  • Signatures for OS or package updates
    Modern alternative: something like OpenBSD's signify or minisign, either of which you could build in Go using x/crypto/ed25519. I wrote a tiny one as a party trick last month. These are extremely simple and robust because they only support one algorithm, lack any notion of a keyring, and are usually decoupled from complex messaging formats (which should be a separate concern from your signature cryptography). They're also broadly compatible with each other because the keys are all EdDSA keys.
  • Encrypting and authenticating data blobs
    Modern alternative: nacl/box. It supports public-key encryption using modern modes and primitives. Keybase made a pretty solid attempt at specifying this with saltpack, which has a Go implementation using the above x/crypto package.

This is the wrong place to be litigating the future of PGP, so I'm going to stop doing it. If you're the maintainer of a widely used Go package that depends on x/crypto/openpgp, feel free to message me off-thread! I will be happy to look at your use case to see if we can think of something that will get you away from it.

@mimoo

This comment has been minimized.

Copy link

mimoo commented Feb 14, 2019

It's sad but there isn't much alternatives to PGP out there. I personally use this tiny script I made to encrypt content to people and give them the key in another channel.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

FiloSottile commented Feb 14, 2019

Polite nudge not to litigate the state of PGP in this issue. Only comment if you think your remark has not been made above, and it might change the decision to deprecate even if there are multiple reasons to deprecate openpgp/.

@ddevault

This comment has been minimized.

Copy link

ddevault commented Feb 15, 2019

I intend to add OpenPGP support to my email client, written in Go, and had considered using this library for that purpose. The remarks in this thread about the dangers and obsolecene of PGP are obscenely exagerrated.

@Mikotochan

This comment has been minimized.

Copy link

Mikotochan commented Feb 15, 2019

Efail has nothing to do with the openpgp and a lot to do with programs that used the GPG api incorrectly.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 16, 2019

The OpenPGP specification predates authenticated encryption, and retroactively added bespoke authentication schemes. There have been attempts to modernize OpenPGP by adding support for authenticated encryption modes, however these proposals never made it to the RFC stage.

The lack of authenticated encryption is the root cause of Efail's CBC/CFB gadget attack as CBC and CFB are both unauthenticated encryption modes. The recommended protocol changes to avoid similar attacks in the future were as follows (emphasis mine):

How did EFAIL influence the developments in the current standards?

There is an ongoing work on two new email security standards. Both considered countermeasures presented in our paper.
The S/MIME standard draft references our EFAIL paper and recommends the usage of authenticated encryption with AES-GCM. Furthermore, it warns that different parts in multipart/mixed emails should be treated as being of different origins.
The OpenPGP standard draft deprecates Symmetrically Encrypted (SE) data packets which are not protected by MDCs. It proposes AEAD protected data packets and mentions that the implementation should not allow users to access erroneous data.

regarding @Mikotochan's comment:

...used the GPG api incorrectly.

Blaming the users of cryptographic APIs for their failure to use them correctly has shown itself to be a particularly bad strategy for developing secure software. The alternative many developers of cryptographic libraries have pursued as a result is designing hard-to-misuse (a.k.a. "misuse resistant") APIs.

That seems to be the direction @FiloSottile would like to go with x/crypto.

@ddevault

This comment has been minimized.

Copy link

ddevault commented Feb 16, 2019

x/crypto doesn't implement the GPG API. It implements PGP.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 16, 2019

@ddevault as I just mentioned, it implements OpenPGP. Do you have a specific concern about what I just said regarding the deficiencies of the OpenPGP protocol?

@ddevault

This comment has been minimized.

Copy link

ddevault commented Feb 16, 2019

Yes. You quoted:

...used the GPG api incorrectly.

The GPG API is a porcelain CLI tool. OpenPGP is a standard. By my measure x/crypto is already doing a damn good job of avoiding the problems which led to EFAIL: shitty usage of the GPG command line tool by applications.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 16, 2019

The GPG API is a porcelain CLI tool. OpenPGP is a standard.

That is why I specifically asked you:

Do you have a specific concern about what I just said regarding the deficiencies of the OpenPGP protocol?

It would appear you are the one colluding the CLI tool with the protocol. So I'll ask you again: is there something about my analysis of the OpenPGP protocol you take issue with?

@ddevault

This comment has been minimized.

Copy link

ddevault commented Feb 16, 2019

You're conflating EFAIL, GPG, and OpenPGP into a single incoherent argument. I'm trying to disentangle it.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 16, 2019

If your problem is collusion of terms, could you please respond to my questions about the OpenPGP protocol without colluding terms yourself?

@ddevault

This comment has been minimized.

Copy link

ddevault commented Feb 16, 2019

As far as I can tell, you haven't actually said anything at all about OpenPGP.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 16, 2019

I will quote https://efail.de/ again:

How did EFAIL influence the developments in the current standards?

[...]
The OpenPGP standard draft deprecates Symmetrically Encrypted (SE) data packets which are not protected by MDCs. It proposes AEAD protected data packets and mentions that the implementation should not allow users to access erroneous data.

The larger point is that Efail was recognized by the relevant authors of the OpenPGP and S/MIME standards as a protocol-level failure, which there is ongoing work to address.

@ddevault

This comment was marked as disruptive content.

Copy link

ddevault commented Feb 16, 2019

"EFAIL" is an alarmist puff piece written by morons to slander PGP and inflate their egos. The standards don't need to change to fix the problems it mentions. The proposals help... marginally. The problem is not and was never with OpenPGP, it's with poorly written email clients (e.g. all email clients).

@w8rbt

This comment has been minimized.

Copy link

w8rbt commented Feb 16, 2019

I know this is a discussion and note a vote, but if I could vote on this, I'd vote to not deprecate OpenPGP. It has RFCs (2440, 4880) and is widely deployed in corporations and security firms. I use it a lot (in and out of Go) to encrypt files to multiple keys and it works well. I do agree with you that it is complicated and I can imagine it is hard to maintain (I've opened issues that your guys have fixed)... Thanks for doing that!

I don't feel as strongly about the other algos/methods, but I like blowfish a lot, too, it's pretty decent. And if you care at all about Windows Active Directory, then you need MD4 which is the heart of NTLM hashes and still widely used in corporate environments. I know, it's a weak hash, but that's the reality we have to deal with as developers in the trenches where Go is a god send.

Thanks for the consideration.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 16, 2019

Please keep the conversation polite and respectful. Thanks.

@Mikotochan

This comment has been minimized.

Copy link

Mikotochan commented Feb 17, 2019

@tarcieri
You could argue that the reason that programs used the GPG api is that the GPG api is crap, but this is absolutely irrelevant to the issue at hand which concerns OpenPGP as a standard. I also never blamed anyone, so please do not put words in my mouth.

The OpenPGP standard right now supports something called MDS (Modification Detection Code) which prevents the attack described in Efail, even though it uses CFB.

@erahn

This comment has been minimized.

Copy link

erahn commented Feb 17, 2019

One major concern I have about deprecating OpenPGP is its use in signing git commits. It doesn't seem unreasonable to agree that is important to many here, but git only supports PGP right now. Deprecating OpenPGP will remove a very useful programmatic alternative to the gpg command line tool. If this goes away, what is another good alternative to manage and generate keys other than the gpg tool?

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 17, 2019

@Mikotochan attacks like Efail are strategically eliminated through the use of authenticated encryption, as authenticated encryption primitives don't disclose ciphertexts when decrypting unless MAC verification is successful.

It's true a supplementary MAC can be and has been added to several protocols including OpenPGP, however this approach of trying to shoehorn authentication into protocols which were originally designed without it as opposed to moving to an authenticated encryption mode is brittle and has failed repeatedly in practice. Efail is an example of such a failure, and one of the takeaways from OpenPGP and S/MIME standards authors was the need to update the relevant standards to support authenticated encryption modes.

It is very much a protocol-level problem with OpenPGP. Perhaps it will be addressed.

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Feb 17, 2019

@FiloSottile

I think it's wrong to deprecate/freeze packages that implement protocols or algorithms just because those algorithms "should not be used in new developments". Go is not only used to implement new software and new protocols; it is also used to implement software that must interoperate with the existing world, imperfect as it might be.

For instance, I've used Go to download third-party files (not authored by me) through HTTP together with their OpenPGP signature and verify their signature after download. I think this is a perfectly good use-case for the OpenPGP package; I want to provide this feature that gives an additional layer of security to users of my software, irrespective of whether there would superior crypto alternatives to solve this problem; it is not my choice to use OpenPGP to sign those third party files, I don't and shouldn't start a crypto war with authors of these third-party files so that they switch to a new signature algorithm just because I don't have a maintained OpenPGP Go package anymore.

If you feel strongly that OpenPGP is not a good choice for new software that starts from scratch and doesn't need to interoperate with the world, than I'm sure a documentation note is good (or in fact, a cycle of good articles on your blog would do even better). But if I need to interoperate with the existing world, I just need an OpenPGP packaged that is actively maintained and compatible, irrespective on the merits of the algorithm.

If you want to retarget x/crypto to a short group of "good algorithms that are good choices" instead of "comprehensive library to interoperate with existing usages of crypto protocols", it would be a serious pity for me, but it's your library and your choice as a maintainer. I really hope somebody will fork it because it provides serious benefits to Go users the way it is now; I personally don't need x/crypto directory list to school me about good choices of crypto algorithms; there's Google, conferences, and papers for that.

The argument that I buy is that you don't have enough resources. In that case, you should first evaluate asking the community for maintainers, before freezing the libraries for good.

@Mikotochan

This comment has been minimized.

Copy link

Mikotochan commented Feb 17, 2019

@tarcieri

attacks like Efail are strategically eliminated through the use of authenticated encryption

The use of MDS in OpenPGP constitutes an instance of AE.

authenticated encryption primitives don't disclose ciphertexts when decrypting unless MAC verification is successful

That only has to do with the implementation of said primitives.

as opposed to moving to an authenticated encryption mode is brittle and has failed repeatedly in practice

Can you give examples of such failure? (and no, Efail is not one of these)

Efail is an example of such a failure

Again, Efail is caused only by the incorrect usage of the (potentially ill-designed) GPG api.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Feb 17, 2019

The use of MDS in OpenPGP constitutes an instance of AE.

When I say "authenticated encryption", I'm referring to IND-CCA3 constructions, e.g. AES-GCM, as presented through an AEAD interface. These constructions, unlike the CBC and CFB modes used by OpenPGP, will never disclose ciphertext until it has been authenticated. There is no way to implement a protocol using these modes that will be vulnerable to something like the Efail attack.

Again note that there ongoing work to add these modes to the OpenPGP spec (motivated by the Efail attack), which in and of itself should be a clear indication that the existing bespoke authentication scheme is considered insufficient by OpenPGP protocol spec authors.

Can you give examples of such failure?

This is getting somewhat off topic, but BEAST, Lucky13, and POODLE are all examples such attacks on TLS. Since that time TLS has specified ciphersuites which incorporate AES-GCM and ChaCha20Poly1305 (both AEAD modes) as mandatory to implement, and Internet traffic has largely switched to using these ciphersuites.

Note that SSH was also vulnerable to similar attacks. Here's a choice quote:

In this paper we show that, contrary to this consensus, the SSH BPP specification in [26, Section 6] has serious design flaws that lead directly to plaintext-recovering attacks against SSH.

The main takeaway from this is AEAD modes are well-studied and have security properties which are provable under the random oracle model. Authentication schemes like SSH's BPP and OpenPGP's MDS are amateur constructions which do not have security proofs. In the case of SSH's BPP, it failed in practice.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

FiloSottile commented Feb 18, 2019

Final nudge not to debate the merits of OpenPGP as a protocol independently of the x/crypto/openpgp implementation in this thread anymore. Further comments about it will be hidden as off-topic. This thread is close to being too long to be useful, I'm afraid.

I am still convinced OpenPGP is not a modern protocol and ecosystem. But again, that's not alone the reason for deprecation. x/crypto/openpgp is an incomplete implementation (and not in the useful way crypto/tls is), can't be complete without being inconsistent with the rest of the Go standard and additional cryptographic libraries, and is a significant resource drain. The fact that the protocol is legacy is simply the reason not to ignore those facts.

@rasky Your argument is a very valid argument against removal, which is indeed not in scope. We disagree on the endorsement value of something being in x/crypto though: while you might vet your choice of algorithms further than "it's in x/crypto, it must be fine", I know for a fact that's not common. I think that justifies deprecation. As for freezing, note that all packages are either very fine as-is (hashes, ciphers), or a resource drain incommensurate to their usefulness (openpgp). As for finding a maintainer, I feel a level of personal responsibility over the quality of x/crypto packages, so while I do hope a community maintained package endures, I would like it to be elsewhere.

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Feb 18, 2019

@FiloSottile OK I then hope somebody will fork and continue x/crypto the way it is today as it does provide useful value. In fact, I'll think of doing it myself.

I'll just note that my argument is against deprecation (not removal), because I can't really use x/crypto/openpgp anymore in production software, if I know for a fact that is unmaintained and deprecated, for reasons that I think are obvious. Both deprecating and removing has the net effect of forcing me to remove it from downstream maintained projects (removing is just harsher, though I always vendor third-party libraries...).

@Merovius

This comment has been minimized.

Copy link

Merovius commented Feb 18, 2019

@FiloSottile

can't be complete without being inconsistent with the rest of the Go standard and additional cryptographic libraries

Can you elaborate on this? I might be overlooking something (long thread), but I can't find any argument why this would be the case. Also, FWIW, I don't think "completeness" is a useful criterion, because it can almost never be well-defined. Like, net/http is not "complete".


FWIW, I acknowledge, that x/crypto/openpgp is costly to maintain. But TBH, to me that's pretty much the reason I don't want it to get deprecated. I thought about it for a couple of days and there really is no group or person I can think of that I'd trust to maintain it in the future. It also feels a bit dissonant to me, that its derided as badly maintained and it's pointed out that its the package in x/crypto with most open CLs and there is a subtext of "community-maintained vs. x/". It seems to me this implies that there are actually people willing to contribute to maintaining it. And that, if the cost thusly is not having enough reviewers, that adding more of them (like whomever we'd expect to maintain a fork) would be an alternative solution. And that, as far as I can see, the only thing speaking against that is some implicit notion of "x/ is not by the community".

Or, to put it another way: If the idea is to deprecate x/crypto/openpgp in favor of a "community-maintained" package - what is preventing that community-maintained package from living at x/crypto/openpgp?

@FiloSottile

This comment has been minimized.

Copy link
Member Author

FiloSottile commented Feb 18, 2019

OK I then hope somebody will fork and continue x/crypto the way it is today as it does provide useful value. In fact, I'll think of doing it myself.

@rasky I'd suggest forking packages that you need and that require maintenance (AFAICT, only openpgp) individually, rather than forking x/crypto. Better yet, help centralize on one of the existing x/crypto/openpgp forks.

@Merovius If you search the issue tracker and Gerrit you'll find x/crypto/openpgp is missing a lot of features, and even more won't even show up because many people shared privately that they gave up long ago and just added things to their own fork. I believe pouring in the resources to add them all would make it a jack-of-all-trades, more aligned to the OpenSSL philosophy than to the Go one.

Or, to put it another way: If the idea is to deprecate x/crypto/openpgp in favor of a "community-maintained" package - what is preventing that community-maintained package from living at x/crypto/openpgp?

I touched on it in #30141 (comment):

As for finding a maintainer, I feel a level of personal responsibility over the quality of x/crypto packages, so while I do hope a community maintained package endures, I would like it to be elsewhere.

If I'm not going to put in the design and review effort (because I'm going to redirect it to other Go crypto work, obviously), I don't want it living on in x/crypto. I'm not saying this is or should be the policy for any other part of the standard or additional libraries, but it feels like a reasonable position for cryptographic code, partially confirmed by your statement above:

I thought about it for a couple of days and there really is no group or person I can think of that I'd trust to maintain it in the future.

@Merovius

This comment has been minimized.

Copy link

Merovius commented Feb 18, 2019

I feel a level of personal responsibility over the quality of x/crypto packages […]
If I'm not going to put in the design and review effort (because I'm going to redirect it to other Go crypto work, obviously), I don't want it living on in x/crypto.

I don't know a polite way to say this. But what makes you, as a person, so special here? Why is you feeling that way an argument for how the Go project as a whole should act? IMO it is a problem if you feel personal responsibility for x/crypto to the degree that it's dictating policy - as it's a project repository, not a personal one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment