Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto: import fiat-crypto implementations #40171

Open
mdempsky opened this issue Jul 12, 2020 · 21 comments
Open

crypto: import fiat-crypto implementations #40171

mdempsky opened this issue Jul 12, 2020 · 21 comments

Comments

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Jul 12, 2020

The fiat-crypto project (https://github.com/mit-plv/fiat-crypto) generates formally-verified, high-performance modular arithmetic implementations, useful for crypto primitives like Curve25519, Poly1305, and the NIST ECC curves that are used within the Go standard library. They're currently working on a Go backend.

BoringSSL has imported their implementations for Curve25519 and P-256: https://boringssl.googlesource.com/boringssl/+/master/third_party/fiat/

At https://go-review.googlesource.com/c/crypto/+/242177, I've uploaded a WIP CL that imports their Curve25519 implementation (w/ minor tweaks), and demonstrates a significant performance improvement over the current "generic" implementation. (The existing amd64 assembly implementation is still considerably faster though.)

This proposal is to import and make use of those implementations.

Open questions:

  1. Which algorithms should be imported? BoringSSL only imports two. Should we import more?

  2. Do we import both 32-bit and 64-bit implementations? We could import just one implementation and still get a performance speedup (e.g., 386 sees a -10% performance boost with curve25519_fiat_64.go, and amd64 sees a -30% boost with curve25519_fiat_32.go), but they do better still with the CPU-appropriate implementations (-30% for 386 w/ 32-bit, and -61% for amd64 w/ 64-bit).

  3. How should the code be imported? E.g., should it be separated into a third_party or vendor directory with its own LICENSE file like how BoringSSL does it?

/cc @agl @FiloSottile

@mdempsky mdempsky added the Proposal label Jul 12, 2020
@gopherbot gopherbot added this to the Proposal milestone Jul 12, 2020
@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Jul 13, 2020

fiat-crypto's 64-bit P-224 implementation is about 2x as fast as Go's existing portable, constant-time P-224 implementation on amd64. Their 32-bit implementation is about the same speed on 386.

I expect P-256 will be similar to Curve25519 (i.e., existing assembly implementations are faster, but still worth measuring), but using fiat-crypto for P-384 and P-521 should be both much faster and provide a constant-time implementation (unlike the current, generic math/big code).

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Aug 7, 2020
@rsc rsc moved this from Incoming to Active in Proposals Aug 12, 2020
@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Aug 18, 2020

@agl @FiloSottile Ping.

While here, I'll point out the fiat-crypto implementation also speeds up curve25519 on ppc64le:

name               old time/op   new time/op   delta
ScalarBaseMult-32    152µs ± 1%     92µs ± 4%  -39.82%  (p=0.000 n=17+20)

name               old speed     new speed     delta
ScalarBaseMult-32  210kB/s ± 0%  347kB/s ± 2%  +65.27%  (p=0.000 n=17+17)

(I don't have any ARM workstations to readily benchmark on.)

@rsc
Copy link
Contributor

@rsc rsc commented Aug 26, 2020

I do have some concerns about adding new license notice requirements in the libraries, because those transitively apply to every Go binary anyone builds (that imports net/http at least).

I would feel much more comfortable about this if we could get the code contributed under CLA so that the Go license notice would cover it.

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Aug 26, 2020

@JasonGross Do you think we can get fiat-crypto's Go code contributed under Google's CLA? The normal process is documented at https://golang.org/doc/contribute.html#cla.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 16, 2020

Ping @JasonGross. We'd be happy to use this code but don't want to impose new notice requirements on every Go binary.

@JasonGross
Copy link

@JasonGross JasonGross commented Sep 16, 2020

Ah, sorry, I meant to follow up on this earlier. As discussed on openssl/openssl#12201 (comment), MIT unfortunately doesn't permit signing CLAs on projects that it holds copyright to. :-/

@rsc
Copy link
Contributor

@rsc rsc commented Sep 18, 2020

@JasonGross, thanks for replying. I certainly understand MIT not wanting to complete CLAs.

An alternative solution to our problem of imposing new notice requirements on every Go binary would be if the generator outputs could be licensed under a non-attribution license such as MIT-0 or a source-code-attribution-only license such as BSD-1-Clause.

Do you think that is a possibility?

@JasonGross
Copy link

@JasonGross JasonGross commented Sep 18, 2020

That seems quite likely. Let me chat with me colleagues and see if it's feasible.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 18, 2020

Thanks very much.

@JasonGross
Copy link

@JasonGross JasonGross commented Sep 22, 2020

@rsc We're in the process of re-licensing under user's choice, MIT OR BSD-1-Clause. However, it seems that BSD-1-Clause is not listed under https://pkg.go.dev/license-policy, even though MIT-0 and BSD-0-Clause are. Is this an oversight? Will BSD-1-Clause in fact be sufficient?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 22, 2020

BSD-1-clause should be fine for our purposes. Thanks very much for tackling this.

I don't know why it's not listed in pkg.go.dev. Maybe it's just not very common. CC @jba .

@jba
Copy link
Contributor

@jba jba commented Sep 22, 2020

Looking into it.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 23, 2020

BSD-1-Clause will be fine, thanks.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 23, 2020

Based on the discussion above, this seems like a likely accept.

@rsc rsc moved this from Active to Likely Accept in Proposals Sep 23, 2020
@JasonGross
Copy link

@JasonGross JasonGross commented Sep 23, 2020

I've gotten approval from everyone and have prepared mit-plv/fiat-crypto#881. Hopefully we'll get it merged in the next couple of days.

@JasonGross
Copy link

@JasonGross JasonGross commented Sep 25, 2020

The code has now been relicensed under MIT OR BSD-1-Clause OR Apache-2.0

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 25, 2020

Thanks!

@rsc
Copy link
Contributor

@rsc rsc commented Sep 30, 2020

Thanks so much @JasonGross!

Accepted.

@rsc rsc moved this from Likely Accept to Accepted in Proposals Sep 30, 2020
@rsc rsc modified the milestones: Proposal, Backlog Sep 30, 2020
@rsc rsc changed the title proposal: crypto: import fiat-crypto implementations crypto: import fiat-crypto implementations Sep 30, 2020
@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Oct 5, 2020

Thank you @JasonGross and everyone! Excited for this to move forward.

We can start by importing the P-384 and P-521 implementations, which currently are bad enough that they'd get the most benefit, and then move to the other fields.

The next step is figuring out a reproducible process to generate the code, and get the output clean enough to pass code review (so we are not tied forever to the code generator, if need be). Happy to help with that in the coming weeks.

@FiloSottile FiloSottile self-assigned this Oct 5, 2020
@FiloSottile FiloSottile modified the milestones: Backlog, Go1.16 Oct 5, 2020
@mmcloughlin
Copy link
Contributor

@mmcloughlin mmcloughlin commented Oct 5, 2020

Regarding code generation from fiat-crypto, ecckiila might be useful to look at. This was used for P-384 and P-521 in NSS 3.55.

@JasonGross
Copy link

@JasonGross JasonGross commented Oct 19, 2020

Note, by the way, that we currently test our Rust Crate on our CI against crypto tests in Dalek, and we test our generated C code against BoringSSL tests. Currently our Go CI only checks that the code compiles, but we'd love to extend it with any sort of integration tests you think might make sense.

Note also that if you want to include the code generator in your process (rather than just fetching our checked-in generated code), our CI is probably a pretty decent place to start, as we regenerate all the code from scratch on every CI run to make sure that we're aware of any changes to the generated code.

And, of course, I'm happy to help with and support any changes you want made to the generated code itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.