crypto: import fiat-crypto implementations

[mdempsky]

The fiat-crypto project (https://github.com/mit-plv/fiat-crypto) generates formally-verified, high-performance modular arithmetic implementations, useful for crypto primitives like Curve25519, Poly1305, and the NIST ECC curves that are used within the Go standard library. They're currently working on a Go backend.

BoringSSL has imported their implementations for Curve25519 and P-256: https://boringssl.googlesource.com/boringssl/+/master/third_party/fiat/

At https://go-review.googlesource.com/c/crypto/+/242177, I've uploaded a WIP CL that imports their Curve25519 implementation (w/ minor tweaks), and demonstrates a significant performance improvement over the current "generic" implementation. (The existing amd64 assembly implementation is still considerably faster though.)

This proposal is to import and make use of those implementations.

Open questions:

1. Which algorithms should be imported? BoringSSL only imports two. Should we import more?

2. Do we import both 32-bit and 64-bit implementations? We could import just one implementation and still get a performance speedup (e.g., 386 sees a -10% performance boost with curve25519_fiat_64.go, and amd64 sees a -30% boost with curve25519_fiat_32.go), but they do better still with the CPU-appropriate implementations (-30% for 386 w/ 32-bit, and -61% for amd64 w/ 64-bit).

3. How should the code be imported? E.g., should it be separated into a third_party or vendor directory with its own LICENSE file like how BoringSSL does it?

/cc @agl @FiloSottile

[mdempsky]

fiat-crypto's 64-bit P-224 implementation is about 2x as fast as Go's existing portable, constant-time P-224 implementation on amd64. Their 32-bit implementation is about the same speed on 386.

I expect P-256 will be similar to Curve25519 (i.e., existing assembly implementations are faster, but still worth measuring), but using fiat-crypto for P-384 and P-521 should be both much faster and provide a constant-time implementation (unlike the current, generic math/big code).

[mdempsky]

@agl @FiloSottile Ping.

While here, I'll point out the fiat-crypto implementation also speeds up curve25519 on ppc64le:

name               old time/op   new time/op   delta
ScalarBaseMult-32    152µs ± 1%     92µs ± 4%  -39.82%  (p=0.000 n=17+20)

name               old speed     new speed     delta
ScalarBaseMult-32  210kB/s ± 0%  347kB/s ± 2%  +65.27%  (p=0.000 n=17+17)

(I don't have any ARM workstations to readily benchmark on.)

[rsc]

I do have some concerns about adding new license notice requirements in the libraries, because those transitively apply to every Go binary anyone builds (that imports net/http at least).

I would feel much more comfortable about this if we could get the code contributed under CLA so that the Go license notice would cover it.

[mdempsky]

@JasonGross Do you think we can get fiat-crypto's Go code contributed under Google's CLA? The normal process is documented at https://golang.org/doc/contribute.html#cla.

[rsc]

Ping @JasonGross. We'd be happy to use this code but don't want to impose new notice requirements on every Go binary.

[JasonGross]

Ah, sorry, I meant to follow up on this earlier. As discussed on openssl/openssl#12201 (comment), MIT unfortunately doesn't permit signing CLAs on projects that it holds copyright to. :-/

[rsc]

@JasonGross, thanks for replying. I certainly understand MIT not wanting to complete CLAs.

An alternative solution to our problem of imposing new notice requirements on every Go binary would be if the generator outputs could be licensed under a non-attribution license such as MIT-0 or a source-code-attribution-only license such as BSD-1-Clause.

Do you think that is a possibility?

[JasonGross]

That seems quite likely. Let me chat with me colleagues and see if it's feasible.

[rsc]

Thanks very much.

[JasonGross]

@rsc We're in the process of re-licensing under user's choice, MIT OR BSD-1-Clause. However, it seems that BSD-1-Clause is not listed under https://pkg.go.dev/license-policy, even though MIT-0 and BSD-0-Clause are. Is this an oversight? Will BSD-1-Clause in fact be sufficient?

[ianlancetaylor]

BSD-1-clause should be fine for our purposes. Thanks very much for tackling this.

I don't know why it's not listed in pkg.go.dev. Maybe it's just not very common. CC @jba .

[jba]

Looking into it.

[rsc]

BSD-1-Clause will be fine, thanks.

[rsc]

Based on the discussion above, this seems like a likely accept.

[JasonGross]

I've gotten approval from everyone and have prepared mit-plv/fiat-crypto#881. Hopefully we'll get it merged in the next couple of days.

[JasonGross]

The code has now been relicensed under MIT OR BSD-1-Clause OR Apache-2.0

[ianlancetaylor]

Thanks!

[rsc]

Thanks so much @JasonGross!

Accepted.

[odeke-em]

I am kindly moving this to Go1.17, as there hasn’t been much action since October 2020.

[mdempsky]

Since Go 1.17 development should be opening up again in the near future, it seems like a good time to discuss how to move forward here.

Would it make sense for the fiat-crypto project to provide a Go module that can be vendored into the standard library, analogous to the rust crate they provide? Or is it preferred for the Go project to own running the generators and checking in the generated code somewhere?

[JasonGross]

If the deployment of the Go module can be automated in the way the Rust crate currently is, I'm happy to merge a PR that does that (and do any additional registration steps necessary).

[mdempsky]

@JasonGross I think it would just require creating a go.mod file (e.g., go mod init github.com/mit-plv/fiat-crypto/fiat-go/src in your fiat-go/src directory), and then users can just import "githtub.com/mit-plv/fiat-crypto/fiat-go/src" and the Go command knows how to fetch your source.

It would require that the code directory is buildable with go build ./... though. Currently, https://github.com/mit-plv/fiat-crypto/tree/master/fiat-go/src has source files from a bunch of different Go packages grouped together (e.g., curve25519_32.go is package fiat_25519, but p224_32.go is fiat_p224). Is it reasonable to split those into separate subdirectories for each package? Alternatively, generating them all in the same package (e.g., package fiat) is probably fine too, as the linker can easily omit any unused code.

One last detail is the APIs need to be exported from the package (i.e., start with an uppercase letter), at least the functionality expected to be used by end users. Conventionally, Go uses mixed caps identifiers (e.g., MixedCaps or mixedCaps). There's typically some leeway allowed for generated code (esp. since this package probably wouldn't be used directly by end users, but instead via higher-level standard library APIs), but to the extent that it's possible to reasonably follow those conventions, I think that would be appreciated.

Happy to chat further if you have more questions. Also, @FiloSottile may have more opinions on how this should be integrated, as owner of Go's crypto packages.

[JasonGross]

I'm nearly done with a PR to adjust the casing conventions. Two questions there: What's the convention for package names? Is the convention for type identifiers the same as for function identifiers?

I'm not sure what to do about the packages, though. In particular, right now the 32-bit and 64-bit implementations have identical identifier names, but live in different file names. (This is useful, for example, in C, where you can just #include a different file depending on the machine architecture and otherwise write the same code.)
I'm happy to either put everything in the same package, or put everything in directories according to the package (does the directory have to be named identically to the package?), whichever is desired.

[JasonGross]

@mdempsky Could you review mit-plv/fiat-crypto#907 and let me know if the name adjustments seem good?

[egonelbre]

@JasonGross few things that stood out to me:

• Package names are usually lowercase. Instead of Fiat25519 it would be fiat25519. And each package is in a separate folder.
• The func names don't need to duplicate the package name, so Fiat25519Carry can be Carry, since it's already in the package fiat25519. The fully qualified name would then be fiat25519.Carry.

[egonelbre]

PS: it's worthwhile to wait for @mdempsky-s comments, since he has probably a better vision how to integrate this than me.

[mdempsky]

@JasonGross Thanks, I'll comment on that PR.

[JasonGross]

mit-plv/fiat-crypto#907 has been merged, so things like import "github.com/mit-plv/fiat-crypto/fiat-go/32/curve25519" should work now. Out of curiosity, what does vendoring the module into the Go standard library look like / consist of?

[mdempsky]

@JasonGross Thanks!

It looks like https://pkg.go.dev/github.com/mit-plv/fiat-crypto/fiat-go/32/curve25519 is confused about the license though. I'm guessing because the fiat-crypto repo doesn't contain any of the files mentioned in https://pkg.go.dev/license-policy. Do you think you could rename the COPYRIGHT file to COPYING? Then hopefully licensecheck will approve it.

As for vendoring, it just amounts to copying snapshots of Go module sources into the Go project's repo for redistribution. You can see the Go modules that are currently vendored at https://github.com/golang/go/tree/master/src/vendor and https://github.com/golang/go/tree/master/src/cmd/vendor. I don't think there's any action you'll need to take there.

[JasonGross]

I'm guessing the actual issue is that the licensing files are at the project root rather than at the root of the fiat-go directory, given that https://pkg.go.dev/license-policy claims to discover licenses in LICENSE-APACHE and LICENSE-MIT, both of which we have. If this doesn't work (how do I get https://pkg.go.dev/github.com/mit-plv/fiat-crypto/fiat-go to bump the version to the latest commit?), I'm happy to try renaming COPYRIGHT to COPYING, though note that this may not be enough, as we release under "MIT OR Apache-2.0 OR BSD-1-Clause", and GitHub, for example, does not recognize this SPDX expression (crates.io, on the other hand, does seem to support "OR", and I cannot tell whether or not https://github.com/google/licensecheck supports multiple licenses / how it handles this case.

[JasonGross]

As for vendoring, it just amounts to copying snapshots of Go module sources into the Go project's repo for redistribution. You can see the Go modules that are currently vendored at https://github.com/golang/go/tree/master/src/vendor and https://github.com/golang/go/tree/master/src/cmd/vendor. I don't think there's any action you'll need to take there.

Sounds good! Do updates to the snapshots happen automatically? On some schedule?

[mdempsky]

I'm guessing the actual issue is that the licensing files are at the project root rather than at the root of the fiat-go directory, given that https://pkg.go.dev/license-policy claims to discover licenses in LICENSE-APACHE and LICENSE-MIT, both of which we have.

Hm, good point. I also wonder if it's getting confused because of multiple licenses. I've filed #44758 to look into this.

At least I don't think this should be blocking for being able to actually use fiat-go. It just prevents being able to see the documentation via the pkg.go.dev interface. I think it's a good sign that it fetched the module at all.

how do I get https://pkg.go.dev/github.com/mit-plv/fiat-crypto/fiat-go to bump the version to the latest commit?

Sorry, I'm not sure. I think it periodically refreshes. I'm not sure how often.

Do updates to the snapshots happen automatically? On some schedule?

They're done manually. Usually shortly before each release, and then as needed/appropriate for development between releases.

Do you think that should work for fiat-go too? What's the process been like for providing code to other projects?

[mdempsky]

And just as I post that last comment, it looks like pkg.go.dev updated, and it does recognize the license now: https://pkg.go.dev/github.com/mit-plv/fiat-crypto/fiat-go@v0.0.0-20210303142032-cc0899379641/32/curve25519