Feat/scroll mem reduction

[CPerezz]

Results so far

In main.

Start: Proof Gen for K=21 GWC, plonk_api test
End: Proof Gen for K=21 GWC, plonk_api test ..................................129.805s
Start: Proof Gen for K=21 SHPLONK, plonk_api test
End: Proof Gen for K=21 SHPLONK, plonk_api test ..............................128.257s

Dhat output for K = 21 GWC

dhat: Total:     49,691,685,732 bytes in 24,022 blocks
dhat: At t-gmax: 9,932,123,677 bytes in 142 blocks
dhat: At t-end:  5,744 bytes in 3 blocks

Dhat output for K = 21 SHPLONK

dhat: Total:     51,002,396,268 bytes in 24,416 blocks
dhat: At t-gmax: 9,932,123,677 bytes in 142 blocks
dhat: At t-end:  5,744 bytes in 3 blocks

In feat/scroll_mem_opt

Start:   Proof Gen for K=21 GWC, `plonk_api` test
End:     Proof Gen for K=21 GWC, `plonk_api` test ..................................196.833s
Start:   Proof Gen for K=21 SHPLONK, `plonk_api` test
End:     Proof Gen for K=21 SHPLONK, `plonk_api` test ..............................198.223s

Dhat output for K = 21 GWC

dhat: Total:     67,626,846,484 bytes in 6,325,485 blocks
dhat: At t-gmax: 5,066,733,341 bytes in 160 blocks
dhat: At t-end:  5,744 bytes in 3 blocks

Dhat output for K = 21 SHPLONK

dhat: Total:     68,937,557,020 bytes in 6,325,879 blocks
dhat: At t-gmax: 5,771,389,277 bytes in 291 blocks
dhat: At t-end:  5,744 bytes in 3 blocks

[CPerezz]

Current status:

Testing plonk_api test and failing for both IPA, GWC and SHPLONK.
It's weird as I checked the diff against taikoxyz#6 and axiom-crypto#17 and they're identical except from some changes in generics I made.

Next things to analyze:

• I haven't added any bugs in the plonk_api test by mistake when adding the profiling features.
• Start to look deeper on whether permutation or lookup can be the cause (by removing them from the test CS).
• If none of them are, then we will have a bigger issue as the error is related to the change but only us have it. None of the other forks do. Which makes it quite a bone to chew 😞

EDIT: Had the brilliant idea of running all tests.

test plonk::circuit::tests::iter_product ... ok
test plonk::circuit::tests::iter_sum ... ok
test plonk::tests::add_trivial_to_inv0_rational ... ok
test plonk::tests::sub_rational_from_inv0_rational ... ok
test plonk::tests::mul_rational_by_inv0_rational ... ok
test plonk::tests::add_rational_to_inv0_rational ... ok
test plonk::tests::sub_trivial_from_inv0_rational ... ok
test arithmetic::test_lagrange_interpolate ... ok
test poly::domain::tests::test_coeff_to_extended_part ... ok
test poly::domain::tests::test_rotate ... ok
test poly::domain::tests::test_l_i ... ok
test plonk::proptests::operation_commutativity ... ok
test poly::domain::tests::test_lagrange_vecs_to_extended ... FAILED
test poly::kzg::commitment::test::test_parameter_serialisation_roundtrip ... ok
test poly::kzg::commitment::test::test_commit_lagrange ... ok
test poly::ipa::msm::tests::msm_arithmetic ... ok
test poly::multiopen_test::test::test_roundtrip_gwc ... ok
test poly::multiopen_test::test::test_roundtrip_shplonk ... ok
test poly::ipa::commitment::test::test_commit_lagrange_epaffine ... ok
test poly::ipa::commitment::test::test_commit_lagrange_eqaffine ... ok
test poly::multiopen_test::test::test_roundtrip_ipa_keccak ... ok
test poly::multiopen_test::test::test_roundtrip_ipa ... ok
test poly::ipa::commitment::test::test_opening_proof ... ok
test poly::domain::tests::bench_lagrange_vecs_to_extended ... ok
test poly::kzg::multiopen::shplonk::proptests::test_intermediate_sets ... ok
test plonk::circuit::compress_selectors::tests::test_selector_combination ... ok
test poly::domain::tests::bench_coeff_to_extended_parts ... ok

Seems we have a possible candidate.

[CPerezz]

Although plonk_api test now passes. One thing to notice is that none of Scroll, Axiom nor Taiko have the shuffle API. Hence they simply got ridd of it and did not need to adapt it to the new Evaluator.

So now is time to integrate the shuffle API with the new Evaluator stack and see if we can make it work.
Ideally, this shouldn't be too hard as considering that the shuffle API isn't expected to be a gigantic argument with lots of instantiations, the idea is that we can simply do FFT/IFFT of the entire thing at once and merge it later with $g(x)$ later to construct $h(x)$