Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto: import fiat-crypto implementations #40171

Open
mdempsky opened this issue Jul 12, 2020 · 36 comments
Open

crypto: import fiat-crypto implementations #40171

mdempsky opened this issue Jul 12, 2020 · 36 comments

Comments

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Jul 12, 2020

The fiat-crypto project (https://github.com/mit-plv/fiat-crypto) generates formally-verified, high-performance modular arithmetic implementations, useful for crypto primitives like Curve25519, Poly1305, and the NIST ECC curves that are used within the Go standard library. They're currently working on a Go backend.

BoringSSL has imported their implementations for Curve25519 and P-256: https://boringssl.googlesource.com/boringssl/+/master/third_party/fiat/

At https://go-review.googlesource.com/c/crypto/+/242177, I've uploaded a WIP CL that imports their Curve25519 implementation (w/ minor tweaks), and demonstrates a significant performance improvement over the current "generic" implementation. (The existing amd64 assembly implementation is still considerably faster though.)

This proposal is to import and make use of those implementations.

Open questions:

  1. Which algorithms should be imported? BoringSSL only imports two. Should we import more?

  2. Do we import both 32-bit and 64-bit implementations? We could import just one implementation and still get a performance speedup (e.g., 386 sees a -10% performance boost with curve25519_fiat_64.go, and amd64 sees a -30% boost with curve25519_fiat_32.go), but they do better still with the CPU-appropriate implementations (-30% for 386 w/ 32-bit, and -61% for amd64 w/ 64-bit).

  3. How should the code be imported? E.g., should it be separated into a third_party or vendor directory with its own LICENSE file like how BoringSSL does it?

/cc @agl @FiloSottile

@gopherbot gopherbot added this to the Proposal milestone Jul 12, 2020
@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Jul 13, 2020

fiat-crypto's 64-bit P-224 implementation is about 2x as fast as Go's existing portable, constant-time P-224 implementation on amd64. Their 32-bit implementation is about the same speed on 386.

I expect P-256 will be similar to Curve25519 (i.e., existing assembly implementations are faster, but still worth measuring), but using fiat-crypto for P-384 and P-521 should be both much faster and provide a constant-time implementation (unlike the current, generic math/big code).

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Aug 7, 2020
@rsc rsc moved this from Incoming to Active in Proposals Aug 12, 2020
@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Aug 18, 2020

@agl @FiloSottile Ping.

While here, I'll point out the fiat-crypto implementation also speeds up curve25519 on ppc64le:

name               old time/op   new time/op   delta
ScalarBaseMult-32    152µs ± 1%     92µs ± 4%  -39.82%  (p=0.000 n=17+20)

name               old speed     new speed     delta
ScalarBaseMult-32  210kB/s ± 0%  347kB/s ± 2%  +65.27%  (p=0.000 n=17+17)

(I don't have any ARM workstations to readily benchmark on.)

@rsc
Copy link
Contributor

@rsc rsc commented Aug 26, 2020

I do have some concerns about adding new license notice requirements in the libraries, because those transitively apply to every Go binary anyone builds (that imports net/http at least).

I would feel much more comfortable about this if we could get the code contributed under CLA so that the Go license notice would cover it.

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Aug 26, 2020

@JasonGross Do you think we can get fiat-crypto's Go code contributed under Google's CLA? The normal process is documented at https://golang.org/doc/contribute.html#cla.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 16, 2020

Ping @JasonGross. We'd be happy to use this code but don't want to impose new notice requirements on every Go binary.

@JasonGross
Copy link

@JasonGross JasonGross commented Sep 16, 2020

Ah, sorry, I meant to follow up on this earlier. As discussed on openssl/openssl#12201 (comment), MIT unfortunately doesn't permit signing CLAs on projects that it holds copyright to. :-/

@rsc
Copy link
Contributor

@rsc rsc commented Sep 18, 2020

@JasonGross, thanks for replying. I certainly understand MIT not wanting to complete CLAs.

An alternative solution to our problem of imposing new notice requirements on every Go binary would be if the generator outputs could be licensed under a non-attribution license such as MIT-0 or a source-code-attribution-only license such as BSD-1-Clause.

Do you think that is a possibility?

@JasonGross
Copy link

@JasonGross JasonGross commented Sep 18, 2020

That seems quite likely. Let me chat with me colleagues and see if it's feasible.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 18, 2020

Thanks very much.

@JasonGross
Copy link

@JasonGross JasonGross commented Sep 22, 2020

@rsc We're in the process of re-licensing under user's choice, MIT OR BSD-1-Clause. However, it seems that BSD-1-Clause is not listed under https://pkg.go.dev/license-policy, even though MIT-0 and BSD-0-Clause are. Is this an oversight? Will BSD-1-Clause in fact be sufficient?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 22, 2020

BSD-1-clause should be fine for our purposes. Thanks very much for tackling this.

I don't know why it's not listed in pkg.go.dev. Maybe it's just not very common. CC @jba .

@jba
Copy link
Contributor

@jba jba commented Sep 22, 2020

Looking into it.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 23, 2020

BSD-1-Clause will be fine, thanks.

@rsc
Copy link
Contributor

@rsc rsc commented Sep 23, 2020

Based on the discussion above, this seems like a likely accept.

@rsc rsc moved this from Active to Likely Accept in Proposals Sep 23, 2020
@JasonGross
Copy link

@JasonGross JasonGross commented Sep 23, 2020

I've gotten approval from everyone and have prepared mit-plv/fiat-crypto#881. Hopefully we'll get it merged in the next couple of days.

@JasonGross
Copy link

@JasonGross JasonGross commented Sep 25, 2020

The code has now been relicensed under MIT OR BSD-1-Clause OR Apache-2.0

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 25, 2020

Thanks!

@rsc
Copy link
Contributor

@rsc rsc commented Sep 30, 2020

Thanks so much @JasonGross!

Accepted.

@rsc rsc moved this from Likely Accept to Accepted in Proposals Sep 30, 2020
@rsc rsc modified the milestones: Proposal, Backlog Sep 30, 2020
@odeke-em
Copy link
Member

@odeke-em odeke-em commented Feb 4, 2021

I am kindly moving this to Go1.17, as there hasn’t been much action since October 2020.

@odeke-em odeke-em modified the milestones: Go1.16, Go1.17 Feb 4, 2021
@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Feb 16, 2021

Since Go 1.17 development should be opening up again in the near future, it seems like a good time to discuss how to move forward here.

Would it make sense for the fiat-crypto project to provide a Go module that can be vendored into the standard library, analogous to the rust crate they provide? Or is it preferred for the Go project to own running the generators and checking in the generated code somewhere?

@JasonGross
Copy link

@JasonGross JasonGross commented Feb 16, 2021

If the deployment of the Go module can be automated in the way the Rust crate currently is, I'm happy to merge a PR that does that (and do any additional registration steps necessary).

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Feb 16, 2021

@JasonGross I think it would just require creating a go.mod file (e.g., go mod init github.com/mit-plv/fiat-crypto/fiat-go/src in your fiat-go/src directory), and then users can just import "githtub.com/mit-plv/fiat-crypto/fiat-go/src" and the Go command knows how to fetch your source.

It would require that the code directory is buildable with go build ./... though. Currently, https://github.com/mit-plv/fiat-crypto/tree/master/fiat-go/src has source files from a bunch of different Go packages grouped together (e.g., curve25519_32.go is package fiat_25519, but p224_32.go is fiat_p224). Is it reasonable to split those into separate subdirectories for each package? Alternatively, generating them all in the same package (e.g., package fiat) is probably fine too, as the linker can easily omit any unused code.

One last detail is the APIs need to be exported from the package (i.e., start with an uppercase letter), at least the functionality expected to be used by end users. Conventionally, Go uses mixed caps identifiers (e.g., MixedCaps or mixedCaps). There's typically some leeway allowed for generated code (esp. since this package probably wouldn't be used directly by end users, but instead via higher-level standard library APIs), but to the extent that it's possible to reasonably follow those conventions, I think that would be appreciated.

Happy to chat further if you have more questions. Also, @FiloSottile may have more opinions on how this should be integrated, as owner of Go's crypto packages.

@JasonGross
Copy link

@JasonGross JasonGross commented Feb 24, 2021

I'm nearly done with a PR to adjust the casing conventions. Two questions there: What's the convention for package names? Is the convention for type identifiers the same as for function identifiers?

I'm not sure what to do about the packages, though. In particular, right now the 32-bit and 64-bit implementations have identical identifier names, but live in different file names. (This is useful, for example, in C, where you can just #include a different file depending on the machine architecture and otherwise write the same code.)
I'm happy to either put everything in the same package, or put everything in directories according to the package (does the directory have to be named identically to the package?), whichever is desired.

JasonGross added a commit to mit-plv/fiat-crypto that referenced this issue Feb 24, 2021
JasonGross added a commit to JasonGross/fiat-crypto that referenced this issue Feb 24, 2021
@JasonGross
Copy link

@JasonGross JasonGross commented Feb 24, 2021

@mdempsky Could you review mit-plv/fiat-crypto#907 and let me know if the name adjustments seem good?

@egonelbre
Copy link
Contributor

@egonelbre egonelbre commented Feb 24, 2021

@JasonGross few things that stood out to me:

  • Package names are usually lowercase. Instead of Fiat25519 it would be fiat25519. And each package is in a separate folder.
  • The func names don't need to duplicate the package name, so Fiat25519Carry can be Carry, since it's already in the package fiat25519. The fully qualified name would then be fiat25519.Carry.
@egonelbre
Copy link
Contributor

@egonelbre egonelbre commented Feb 24, 2021

PS: it's worthwhile to wait for @mdempsky-s comments, since he has probably a better vision how to integrate this than me.

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Feb 24, 2021

@JasonGross Thanks, I'll comment on that PR.

JasonGross added a commit to JasonGross/fiat-crypto that referenced this issue Feb 26, 2021
JasonGross added a commit to JasonGross/fiat-crypto that referenced this issue Feb 26, 2021
JasonGross added a commit to JasonGross/fiat-crypto that referenced this issue Feb 26, 2021
JasonGross added a commit to JasonGross/fiat-crypto that referenced this issue Feb 26, 2021
JasonGross added a commit to JasonGross/fiat-crypto that referenced this issue Feb 26, 2021
JasonGross added a commit to JasonGross/fiat-crypto that referenced this issue Mar 2, 2021
JasonGross added a commit to mit-plv/fiat-crypto that referenced this issue Mar 3, 2021
@JasonGross
Copy link

@JasonGross JasonGross commented Mar 3, 2021

mit-plv/fiat-crypto#907 has been merged, so things like import "github.com/mit-plv/fiat-crypto/fiat-go/32/curve25519" should work now. Out of curiosity, what does vendoring the module into the Go standard library look like / consist of?

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Mar 3, 2021

@JasonGross Thanks!

It looks like https://pkg.go.dev/github.com/mit-plv/fiat-crypto/fiat-go/32/curve25519 is confused about the license though. I'm guessing because the fiat-crypto repo doesn't contain any of the files mentioned in https://pkg.go.dev/license-policy. Do you think you could rename the COPYRIGHT file to COPYING? Then hopefully licensecheck will approve it.

As for vendoring, it just amounts to copying snapshots of Go module sources into the Go project's repo for redistribution. You can see the Go modules that are currently vendored at https://github.com/golang/go/tree/master/src/vendor and https://github.com/golang/go/tree/master/src/cmd/vendor. I don't think there's any action you'll need to take there.

@JasonGross
Copy link

@JasonGross JasonGross commented Mar 3, 2021

I'm guessing the actual issue is that the licensing files are at the project root rather than at the root of the fiat-go directory, given that https://pkg.go.dev/license-policy claims to discover licenses in LICENSE-APACHE and LICENSE-MIT, both of which we have. If this doesn't work (how do I get https://pkg.go.dev/github.com/mit-plv/fiat-crypto/fiat-go to bump the version to the latest commit?), I'm happy to try renaming COPYRIGHT to COPYING, though note that this may not be enough, as we release under "MIT OR Apache-2.0 OR BSD-1-Clause", and GitHub, for example, does not recognize this SPDX expression (crates.io, on the other hand, does seem to support "OR", and I cannot tell whether or not https://github.com/google/licensecheck supports multiple licenses / how it handles this case.

@JasonGross
Copy link

@JasonGross JasonGross commented Mar 3, 2021

As for vendoring, it just amounts to copying snapshots of Go module sources into the Go project's repo for redistribution. You can see the Go modules that are currently vendored at https://github.com/golang/go/tree/master/src/vendor and https://github.com/golang/go/tree/master/src/cmd/vendor. I don't think there's any action you'll need to take there.

Sounds good! Do updates to the snapshots happen automatically? On some schedule?

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Mar 3, 2021

I'm guessing the actual issue is that the licensing files are at the project root rather than at the root of the fiat-go directory, given that https://pkg.go.dev/license-policy claims to discover licenses in LICENSE-APACHE and LICENSE-MIT, both of which we have.

Hm, good point. I also wonder if it's getting confused because of multiple licenses. I've filed #44758 to look into this.

At least I don't think this should be blocking for being able to actually use fiat-go. It just prevents being able to see the documentation via the pkg.go.dev interface. I think it's a good sign that it fetched the module at all.

how do I get https://pkg.go.dev/github.com/mit-plv/fiat-crypto/fiat-go to bump the version to the latest commit?

Sorry, I'm not sure. I think it periodically refreshes. I'm not sure how often.

Do updates to the snapshots happen automatically? On some schedule?

They're done manually. Usually shortly before each release, and then as needed/appropriate for development between releases.

Do you think that should work for fiat-go too? What's the process been like for providing code to other projects?

@mdempsky
Copy link
Member Author

@mdempsky mdempsky commented Mar 3, 2021

And just as I post that last comment, it looks like pkg.go.dev updated, and it does recognize the license now: https://pkg.go.dev/github.com/mit-plv/fiat-crypto/fiat-go@v0.0.0-20210303142032-cc0899379641/32/curve25519

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants