BLST-based crypto package

[tarakby]

Overview

This PR is a major refactor of the Flow crypto library, in particular the the BLS 12-381 curve implementation. It updates the curve implementation from relying on Relic external library to relying on BLST external library.

This has two advantages:

• Flow crypto becomes an easily usable Go package, that is ready to import and build without extra friction steps. Previously the package needed a setup step to build a Relic static library using non-Go tools.
• BLST is written and optimized specifically for the BLS12-381 curve, while Relic is a generic library supporting multiple curves and use cases. Moving to BLST improves the package performance significantly.

Changes

Flow crypto library implementation provides an implementation of the BLS signature scheme, BLS-based threshold signature, DKG for BLS-based threshold signature, and BLS-based SPoCK. All of these schemes are set up using the BLS 12-381 curve.
The implementation of all these schemes has a common structure into 3 layers:

1. Go upper layer with public functions and structures - thin cryptography-related implementations
2. C middle-layer: the major layer implementing all the cryptographic schemes/protocols above
3. C lower layer implementing all BLS12-381 tools (field arithmetic, point arithmetic, pairing)

Layer 3 used to use Relic tools before this PR, and this PR is updating it to use BLST tools. As a consequence, Layer 2 underwent a major refactor to adapt to new BLST tools. Layer 1 also underwent a minor refactor.

The PR also integrates the new Flow crypto library into onflow/flow-go. In particular it updates all the test and build tooling (workflows, tests, makefiles)

This PR is made of the smaller PRs below, where more details can be found:

• [crypto] add blst source code #3851
• [Crypto] new BLST-based scalar and field element type  #3993
• [crypto] E2 BLST-based type  #4230
• [Crypto] use deterministic randomness in C backend #4253
• [Crypto] merge master to BLST feature branch #4264
• [Crypto] add BLS cross tests against Go-BLST  #4328
• [Crypto] BLST-based E1 and G1 types  #4355
• [Crypto] BLST pairing #4377
• [Crypto] unplug relic from the crypto package and the rest of flow-go #4380
• [crypto] multiple improvements of the code base #4397
• [Crypto] enable ADX support by default #4405
• [Crypto] C formatting and DKG optimization #4645
• [Crypto] C code sanitization #4654
• [Crypto] misc updates #4681
• [Crypto] merge master branch #4687
• [Crypto] update BLST version  #4693
• [Crypto] minor clean up #4740
• Merge master into BLST feature branch #4863
• [Crypto] Code review changes and fix access integration test #4817

Testing

In addition to CI tests on this repo, the new module has been tested on transient Flow networks with up to 150 nodes (using the Benchnet2 framework). Results can be found here.

Benchmark

(on an Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz)

For a quick read, you can focus only on the BLS_Sign (3.9x improvement) and BLS_Verify(3.6x improvement).

• Relic based library

BenchmarkBLS12381_ScalarMult/G1-8                     8408            144367 ns/op
BenchmarkBLS12381_ScalarMult/G2-8                     2839            407404 ns/op
BenchmarkBLS12381_MapToG1-8                           4389            268728 ns/op
BenchmarkBLS12381_SubgroupCheck/G1-8                  4442            249818 ns/op
BenchmarkBLS12381_SubgroupCheck/G2-8                  1035           1141323 ns/op
BenchmarkBLS_Sign-8                                   1824            591269 ns/op
BenchmarkBLS_Verify-8                                  326           3652662 ns/op
BenchmarkBLS_BatchVerify/happy_path-8                   12          93642932 ns/op
BenchmarkBLS_BatchVerify/unhappy_path-8                 12         141479731 ns/op
BenchmarkBLS_VerifySignatureManyMessages-8             276           4352977 ns/op
BenchmarkBLS_Aggregate/PrivateKeys-8                 12282             97610 ns/op
BenchmarkBLS_Aggregate/PublicKeys-8                    297           3919277 ns/op
BenchmarkBLS_Aggregate/Signatures-8                     16          65730951 ns/op
BenchmarkBLS_ThresholdSimpleKeyGen-8                    43          25577217 ns/op

• BLST based library (without ADX)

BenchmarkBLS12381_ScalarMult/G1-8                           9153            114988 ns/op
BenchmarkBLS12381_ScalarMult/G2-8                           4369            245211 ns/op
BenchmarkBLS12381_MapToG1-8                                14991             80184 ns/op
BenchmarkBLS12381_SubgroupCheck/G1-8                       19898             61015 ns/op
BenchmarkBLS12381_SubgroupCheck/G2-8                       15300             89980 ns/op
BenchmarkBLS_Sign-8                                         5122            196962 ns/op
BenchmarkBLS_Verify-8                                        920           1255320 ns/op
BenchmarkBLS_BatchVerify/happy_path-8                         26          44006822 ns/op
BenchmarkBLS_BatchVerify/unhappy_path-8                       24          43398434 ns/op
BenchmarkBLS_VerifySignatureManyMessages-8                    34          35534403 ns/op
BenchmarkBLS_Aggregate/PrivateKeys-8                       99625             11719 ns/op
BenchmarkBLS_Aggregate/PublicKeys-8                          434           2658353 ns/op
BenchmarkBLS_Aggregate/Signatures-8                           55          20230502 ns/op
BenchmarkBLS_ThresholdSimpleKeyGen-8                          73          14797332 ns/op
BenchmarkBLS_ThresholdSignatureReconstruction-8              220           5502518 ns/op

• BLST based library (with ADX)

BenchmarkBLS12381_ScalarMult/G1-8                          13239             90085 ns/op
BenchmarkBLS12381_ScalarMult/G2-8                           6196            192959 ns/op
BenchmarkBLS12381_MapToG1-8                                20506             58979 ns/op
BenchmarkBLS12381_SubgroupCheck/G1-8                       24444             50676 ns/op
BenchmarkBLS12381_SubgroupCheck/G2-8                       20234             58880 ns/op
BenchmarkBLS_Sign-8                                         7964            152160 ns/op
BenchmarkBLS_Verify-8                                       1154           1008637 ns/op
BenchmarkBLS_BatchVerify/happy_path-8                         34          34089439 ns/op
BenchmarkBLS_BatchVerify/unhappy_path-8                       31          33988129 ns/op
BenchmarkBLS_VerifySignatureManyMessages-8                    42          27193364 ns/op
BenchmarkBLS_Aggregate/PrivateKeys-8                       89925             11401 ns/op
BenchmarkBLS_Aggregate/PublicKeys-8                          586           1995054 ns/op
BenchmarkBLS_Aggregate/Signatures-8                           73          14013157 ns/op
BenchmarkBLS_ThresholdSimpleKeyGen-8                          91          11392687 ns/op
BenchmarkBLS_ThresholdSignatureReconstruction-8              278           4393246 ns/op

Risks

• substitute a generic implementations that works for multiple algorithms/curves/fields (RELIC) with a library that only supports BLS12-381 curve.
• RELIC exports arithmetic tools on their external functions while BLST exports mainly BLS tools and Flow crypto rather imports the internal non-exported arithmetic tools (risk of BLST not maintaining same internal functions in future updates)

Review assignment:

Note: Please do not review files underonflow/flow-go/crypto/internal/ and onflow/flow-go/crypto/blst_src/ (about 135 files) because these are files copied from the BLST repo without changes. The reasoning behind the copy is explained in this README file.

Review is open to everyone interested. In particular, a review from the following members is appreciated:

1. onflow/flow-go/crypto/ - .c and .h files: @durkmurder
2. onflow/flow-go/crypto/ - go files (tests included): @jordanschalm
3. Makefile and .yml and Dockerfile files: @gomisha / @peterargue
4. READMEfiles : any member from above is welcome to review
5. onflow/flow-go/ - go files outside /crypto: @peterargue

[durkmurder]

Reviewed .*c files. Great accomplishment! ⭐