feat: Execution Trace

[ledwards2225]

Introduces an ExecutionTrace class responsible for translating raw circuit data into the "execution trace polynomials", which I define as the wire, selector and permutation argument polynomials (sigma/id). This work was previously split across several different methods in the Composers/ProverInstance and much of the logic was duplicated between Plonk and Honk. That core logic is now shared across Plonk/Honk for all flavors. It is encapsulated into a single call of the form Trace::generate(circuit, proving_key); which takes the raw circuit data contained in circuit and populates the provided proving_key with wire, selector and sigma/id polynomials.

The main work of the ExecutionTrace is to "sort" the relatively un-ordered raw circuit data. For example, for GUH, the ordering of the execution trace is (1) zero row, (2) goblin ecc op gates, (3) public inputs, and (4) conventional gates. Each one of these can be described as an ExecutionTraceBlock which is a {wires, selectors} pair. E.g. the conventional gates block is just the existing {builder.wires, builder.selectors}. The public inputs block has all of its selectors equal to zero, the first two wires equal to builder.public_inputs, and the remaining wires identically zero. And so on. Eventually, each gate type will have its own ExecutionTraceBlock. (I've rephrased the goblin ecc op gates as an ExecutionTraceBlock as an example). These blocks are constructed by the builder and "sorting" is achieved in ExecutionTrace by simply arranging the blocks as desired then constructing polynomials.

From here, achieving a sorted execution trace should amount to introducing one ExecutionTraceBlock for each gate type in the builders, populating the appropriate block in each gate construction function (instead of piling everything into a single wires, selectors), then arranging those blocks to define the execution trace polynomials in ExecutionTrace.

Closes AztecProtocol/barretenberg#808
Closes AztecProtocol/barretenberg#859

[AztecBot]

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 19d2bbea and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	54,212	214,436	855,332
l1_rollup_calldata_gas	259,832	1,019,840	4,058,612
l1_rollup_execution_gas	354,018	1,139,288	4,280,013
l2_block_processing_time_in_ms	1,267	4,748 (-1%)	18,874 (-1%)
note_successful_decrypting_time_in_ms	221 (+1%)	613	2,170 (-3%)
note_trial_decrypting_time_in_ms	29.6 (-1%)	48.8 (-4%)	182 (-3%)
l2_block_building_time_in_ms	21,362	84,440	342,752
l2_block_rollup_simulation_time_in_ms	16,009	63,317	258,545
l2_block_public_tx_process_time_in_ms	5,326	21,047	83,953 (-1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	16,255	31,215 (-2%)
note_history_successful_decrypting_time_in_ms	1,615 (+15%)	2,727 (+1%)
note_history_trial_decrypting_time_in_ms	129 (-23%)	233 (+4%)
node_database_size_in_bytes	18,686,032	35,930,192 (-1%)
pxe_database_size_in_bytes	29,923	59,478

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	333	44,664	27,457
private-kernel-ordering	214	45,777	14,627
base-rollup	1,745	175,344	933
root-rollup	82.6 (-2%)	4,192	825
private-kernel-inner	444	73,099	27,457
public-kernel-app-logic	275	32,254	25,379
merge-rollup	7.80 (-1%)	2,712	933

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	2 leaves	8 leaves	16 leaves	32 leaves	128 leaves	64 leaves	512 leaves	1024 leaves	2048 leaves	8192 leaves
batch_insert_into_append_only_tree_16_depth_ms	10.2 (-1%)	10.8 (+2%)	12.8	17.2	23.0	63.9	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.9	17.5	23.0	31.6	47.0	143	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.593 (-1%)	0.602 (+2%)	0.543	0.532	0.479	0.441	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	N/A	N/A	N/A	74.6	47.7 (-1%)	240	461	907 (+1%)	3,552
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	N/A	N/A	N/A	159	96.0	543	1,055	2,079	8,223
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.461	0.488 (-1%)	0.437	0.432	0.431 (+1%)	0.428
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	N/A	N/A	N/A	105 (+1%)	57.6	355	687	1,375	5,419
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	N/A	N/A	N/A	197	104	691	1,363	2,707	10,771
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.495	0.503 (+1%)	0.481 (+1%)	0.472	0.475	0.472
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	N/A	64.4	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	N/A	110	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	N/A	0.556	N/A	N/A	N/A	N/A	N/A	N/A

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	15,388	36,001

Transaction processing duration by data writes.

Metric	0 new commitments	1 new commitments
tx_pxe_processing_time_ms	575 (-1%)	1,506

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	0.0284	655

[codygunton]

You walked me through this and we discussed a few follow-on changes that will happen after review. Had a look myself too--LGTM

[codygunton]

Maybe should go in the generate function?