Pasta / Halo2 MSM bench

[mratsim]

Benchmarks and optimization of MSM for the Halo2 Pasta curves

Current on 8 cores i9-11980HK with Clang

TODO

• Bench vs Zcash / Privacy Scaling Exploration implementation
• Make "no ASM" an environment variable to divide by 2 the number of nimble tasks
• Generate coefficient-point pairs in parallel
• Bench on large core count
• Tune if needed

[mratsim]

Might have missed something, there is no benchmark for MSM in Halo2 or pasta_curves repo?

Used this PR: https://github.com/zcash/halo2/pull/619/files
file: https://github.com/zcash/halo2/blob/b131df023c9a860244b5b0a24d03a1c249f4c82c/halo2_proofs/benches/multiexp.rs

[mratsim]

Rebench on same scale as Halo2

Something is really strange.

For 2⁸ inputs / 256 inputs it takes 13ms, this is even slower than my naive scalar mul at 8.7ms. And multithreaded I am at 313µs so 41x faster.

The code in the PR is indeed multithreaded: https://github.com/zcash/halo2/blob/b131df0/halo2_proofs/src/arithmetic.rs#L28-L30

Is the benchmark flawed and measuring the trusted setup setup as well? https://github.com/zcash/halo2/blob/b131df0/halo2_proofs/benches/multiexp.rs#L25-L28

Thing is, time is roughly doubling each time we double the input size but MSM should scale as O(n/log n) while the Rust bench seems to grow linearly.

With 2¹⁵ inputs (32748), they take 1.7s while Constantine takes 12ms so a 141.7x ratio which sounds crazy. And the naive implementation in Constantine takes 1.11 seconds

[mratsim]

Benching vs Supranational at https://github.com/supranational/pasta-msm

2x faster

[mratsim]

On a watercooled overclocked 18-core i9-9980XE

[mratsim]

From Zcash repo: https://github.com/zcash/halo2/pull/619/files#diff-a07879d4aa4c95cfbfb03f5de33deee89d548aba465ff7bbdc5965d24463b0cb

Similar perf issues on Zcash:

• 26ms for 256 inputs while on my machine Constantine is 0.412ms, a 63x ratio
• 3.3932s for 32768 inputs while Constantine is 0.011s, a 308x ratio.

[mratsim]

Supranational: https://github.com/supranational/pasta-msm

There is a ratio 2.22x in favor of Constantine