Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving Shplonk implementation #326

Merged
merged 4 commits into from
Feb 20, 2024
Merged

Improving Shplonk implementation #326

merged 4 commits into from
Feb 20, 2024

Conversation

storojs72
Copy link
Member

@storojs72 storojs72 commented Feb 15, 2024

Fixes #303

This PR introduces 2 improvements:

  • micro-optimisation related to skipping operations with constant $P_i$ polynomial;
  • adding clean parallelism to verifier.

Next PR should include careful merging Shplonk and HyperKZG implementations leaving hyperkzg.rs source file and removing shplonk.rs (fixing #302). I expect some more improving of prover performance with this merge.

Concrete results of benchmarking at this stage (Shplonk vs current HyperKZG) are following (similar to #301 (comment) - comparable verifier and improved prover):

PCS-Proving 10/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...                                                                           
                        time:   [7.1720 ms 7.2653 ms 7.3883 ms]
                        change: [-31.436% -18.925% -5.6975%] (p = 0.02 < 0.05)
                        Performance has improved.
PCS-Proving 10/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, areci...                                                                           
                        time:   [6.1288 ms 6.1909 ms 6.2744 ms]
                        change: [-9.9977% -5.3310% -0.2003%] (p = 0.06 > 0.05)
                        No change in performance detected.

PCS-Verifying 10/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, ar...                                                                            
                        time:   [2.1010 ms 2.1137 ms 2.1270 ms]
                        change: [+2.9871% +3.7964% +4.4698%] (p = 0.00 < 0.05)
                        Performance has regressed.
PCS-Verifying 10/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, are...                                                                            
                        time:   [2.0729 ms 2.0768 ms 2.0833 ms]
                        change: [-4.8892% -4.0022% -2.9876%] (p = 0.00 < 0.05)
                        Performance has improved.

PCS-Proving 12/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...                                                                           
                        time:   [16.913 ms 16.931 ms 16.957 ms]
                        change: [-0.7987% -0.1235% +0.8007%] (p = 0.82 > 0.05)
                        No change in performance detected.
PCS-Proving 12/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, areci...                                                                           
                        time:   [15.337 ms 16.789 ms 18.565 ms]
                        change: [-28.641% -18.755% -9.1390%] (p = 0.00 < 0.05)
                        Performance has improved.

PCS-Verifying 12/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, ar...                                                                            
                        time:   [2.1503 ms 2.1542 ms 2.1584 ms]
PCS-Verifying 12/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, are...                                                                            
                        time:   [2.1019 ms 2.1392 ms 2.2256 ms]
                        change: [+1.7808% +32.189% +91.488%] (p = 0.21 > 0.05)
                        No change in performance detected.

PCS-Proving 14/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...                                                                           
                        time:   [48.476 ms 48.525 ms 48.570 ms]
PCS-Proving 14/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, areci...                                                                           
                        time:   [41.439 ms 41.612 ms 41.822 ms]
                        change: [-8.0785% -5.6529% -3.2291%] (p = 0.00 < 0.05)
                        Performance has improved.

PCS-Verifying 14/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, ar...                                                                            
                        time:   [2.1573 ms 2.1594 ms 2.1622 ms]
PCS-Verifying 14/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, are...                                                                            
                        time:   [2.0895 ms 2.0916 ms 2.0946 ms]
                        change: [-2.3498% -1.5942% -0.8665%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Benchmarking PCS-Proving 16/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 8.9s or enable flat sampling.
PCS-Proving 16/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...                                                                          
                        time:   [160.76 ms 161.32 ms 161.89 ms]
Benchmarking PCS-Proving 16/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, areci...: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 8.1s or enable flat sampling.
PCS-Proving 16/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, areci...                                                                          
                        time:   [145.09 ms 147.90 ms 150.21 ms]
                        change: [-2.7417% -0.2984% +2.5516%] (p = 0.84 > 0.05)
                        No change in performance detected.

PCS-Verifying 16/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, ar...                                                                            
                        time:   [2.0656 ms 2.0703 ms 2.0750 ms]
PCS-Verifying 16/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, are...                                                                            
                        time:   [2.1035 ms 2.1090 ms 2.1131 ms]
                        change: [-37.671% -20.462% -6.4595%] (p = 0.10 > 0.05)
                        No change in performance detected.

Benchmarking PCS-Proving 18/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 6.1s.
PCS-Proving 18/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...                                                                          
                        time:   [607.39 ms 609.91 ms 612.96 ms]
PCS-Proving 18/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, areci...                                                                          
                        time:   [492.11 ms 495.68 ms 499.42 ms]
                        change: [-0.0073% +1.0534% +2.0617%] (p = 0.07 > 0.05)
                        No change in performance detected.

PCS-Verifying 18/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, ar...                                                                            
                        time:   [2.1093 ms 2.1109 ms 2.1117 ms]
PCS-Verifying 18/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, are...                                                                            
                        time:   [2.2284 ms 2.2310 ms 2.2341 ms]
                        change: [-0.2053% -0.0275% +0.1617%] (p = 0.78 > 0.05)
                        No change in performance detected.

Benchmarking PCS-Proving 20/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 23.0s.
PCS-Proving 20/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, arec...                                                                          
                        time:   [2.2841 s 2.3119 s 2.3544 s]
Benchmarking PCS-Proving 20/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, areci...: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 22.3s.
PCS-Proving 20/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, areci...                                                                          
                        time:   [2.3847 s 2.6213 s 2.9045 s]
                        change: [+12.963% +24.355% +39.072%] (p = 0.00 < 0.05)
                        Performance has regressed.

PCS-Verifying 20/arecibo::provider::hyperkzg::EvaluationEngine<halo2curves::bn256::engine::Bn256, ar...                                                                            
                        time:   [2.1522 ms 2.1714 ms 2.1860 ms]
PCS-Verifying 20/arecibo::provider::shplonk::EvaluationEngine<halo2curves::bn256::engine::Bn256, are...                                                                            
                        time:   [2.2558 ms 2.2595 ms 2.2624 ms]
                        change: [-0.1964% +0.0095% +0.2161%] (p = 0.94 > 0.05)
                        No change in performance detected.

P.S. @huitseeker, I saw your WIP branch related to parallel rlc. Please, notify me when you land that, so I can simplify C_P computing by utilising you work

Copy link
Contributor

@adr1anh adr1anh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some (duplicated) comments to give pointers about optimizations. If you prefer to tackle them as another PR feel free to mark them as resolved and I'll approve this one. Otherwise, happy to give this another look later.

src/provider/shplonk.rs Outdated Show resolved Hide resolved
src/provider/shplonk.rs Outdated Show resolved Hide resolved
src/provider/shplonk.rs Outdated Show resolved Hide resolved
src/provider/shplonk.rs Outdated Show resolved Hide resolved
src/provider/shplonk.rs Outdated Show resolved Hide resolved
src/provider/shplonk.rs Outdated Show resolved Hide resolved
src/provider/shplonk.rs Outdated Show resolved Hide resolved
src/provider/shplonk.rs Outdated Show resolved Hide resolved
src/provider/shplonk.rs Show resolved Hide resolved
src/provider/shplonk.rs Outdated Show resolved Hide resolved
}

let C_P: E::G1 = pi.comms.iter().map(|comm| comm.to_curve()).rlc(&q);
// TODO: revisit C_P computing later with progress in https://github.com/huitseeker/arecibo/tree/rayon-parscan
let q_powers = HyperKZG::<E, NE>::batch_challenge_powers(q, comms.len());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can make spartan::powers pub (in crate):

Suggested change
let q_powers = HyperKZG::<E, NE>::batch_challenge_powers(q, comms.len());
let q_powers = crate::spartan::powers(q, comms.len());

Comment on lines 294 to 299
#[allow(clippy::redundant_closure)]
let C_P: E::G1 = comms
.par_iter()
.zip_eq(q_powers.par_iter())
.fold(|| E::G1::identity(), |acc, (comm, q_i)| acc + *comm * q_i)
.sum::<E::G1>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[allow(clippy::redundant_closure)]
let C_P: E::G1 = comms
.par_iter()
.zip_eq(q_powers.par_iter())
.fold(|| E::G1::identity(), |acc, (comm, q_i)| acc + *comm * q_i)
.sum::<E::G1>();
let C_P: E::G1 = comms
.par_iter()
.zip_eq(q_powers.par_iter())
.fold(E::G1::identity, |acc, (comm, q_i)| acc + *comm * q_i)
.sum::<E::G1>();

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be irrelevant after rebasing to latest dev which includes #330

adr1anh
adr1anh previously approved these changes Feb 20, 2024
Copy link
Contributor

@adr1anh adr1anh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Feel free to apply the suggestions by @huitseeker (will re-approve if necessary)

Copy link
Member

@huitseeker huitseeker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@storojs72 storojs72 added this pull request to the merge queue Feb 20, 2024
Merged via the queue into dev with commit 1f453c9 Feb 20, 2024
9 checks passed
@storojs72 storojs72 deleted the issue-303 branch February 20, 2024 14:32
github-actions bot pushed a commit that referenced this pull request Jul 3, 2024
* avoid explicit compression/decompression

* update version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Shplonk improvements
3 participants