perf(batch): optimize proof materialization pipeline

[ryan-wong-coder]

Linked Issue

N/A

Summary

• remove duplicate L3 signing and repeated Merkle/tree persistence from async batch planning
• add durable bounded materialization workers with retryable manifest states
• add incremental L2-to-L3 index promotion, Pebble tree tiles, batched L4 appends, and concurrent L5 workers
• add benchmark profiles, metrics, regression tests, and performance documentation

Required Format

• Branch name uses an approved feat/ prefix.
• PR title and commit use Conventional Commits.

Scope

• This PR solves one clearly scoped performance and reliability task.
• This PR excludes unrelated local files and generated artifacts.
• User-facing documentation describes implemented behavior only.

Proof, Storage, And Compatibility

• L1/L2/L3/L4/L5 cryptographic proof formats are unchanged.
• L4 remains batch-root inclusion in the Global Log; L5 remains STH anchoring.
• Public HTTP, gRPC, SDK, and proof schemas are unchanged.
• Pebble durable storage boundaries are unchanged.

Pebble uses storage schema v2 for 512-entry batch-tree tiles and refuses legacy populated stores. There is intentionally no automatic migration because no production dataset needs preservation. WAL replay now understands preparing/prepared/failed materialization states.

Validation

• go test ./...
• go vet ./...
• go test -race ./...
• go test -count=1 -tags=integration ./...
• go test -count=1 -tags=e2e ./...
• all shipped and new benchmark configs pass trustdb config validate
• frontend jobs are left to GitHub CI because frontend sources are unchanged

Local Apple M1 benchmark highlights:

• CommitBatchIndexes/1024: about 23.6 ms -> 1.0-1.3 ms; about 3.34 MB -> 0.79 MB
• full L2 plan + compact tree snapshot: about 1.46-1.57 ms, 1.06 MB
• time-only materialized artifacts: about 33-36 ms / 12.9 MB -> 26-29 ms / 4.7 MB
• 8192-record Pebble batch tree: about 24,575 objects -> 40 tiles

Risk And Rollback

• rollback by reverting this commit and using a fresh pre-v2 proofstore
• existing legacy Pebble stores fail fast with an actionable schema error instead of being modified
• full 32-vCPU dual-machine throughput acceptance remains a deployment benchmark, documented in the new performance report