Simplex consensus performance update#129
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR is a broad performance/observability update spanning Simplex consensus integration, VM/block primitives, and node telemetry/tools. It introduces new resolver plumbing for Simplex (to reduce parent-state stalls), switches several hot byte paths to use SmallData/slice-based extensions, and adds multiple operational metrics + log-analysis helpers.
Changes:
- Add a validator-side Simplex state resolver cache, plus new consensus callbacks/API (
on_candidate_observed,Session::start(prev_blocks, min_masterchain_block_id),ensure_candidate_available) and tests. - Optimize BOC/cell/slice handling (e.g.,
SmallData,get_slice(offset,bits),stream_read/read_to_storage) and update call sites across node/vm/executor/block code. - Improve observability and tooling: RocksDB memory usage telemetry, account-state cache bytes metric, Prometheus republishing for per-session Simplex metrics, and new allocation timeline scripts.
Reviewed changes
Copilot reviewed 124 out of 127 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/vm/tests/test_stack_manipulation.rs | Avoid Vec::append temps; use slice extends |
| src/vm/tests/common/test_framework.rs | Accept SmallData-compatible bytecode; faster byte extraction |
| src/vm/src/tests/test_smart_contract_info.rs | Update get_slice API usage |
| src/vm/src/executor/dump.rs | Convert SmallData to Vec where needed |
| src/vm/src/executor/crypto.rs | Convert SmallData to Vec for signature verification |
| src/vm/src/executor/bls.rs | Switch message extraction to SmallData |
| src/tl/ton_api/tl/ton_api.tl | Update Simplex DB schema TL definitions |
| src/node/validator-session/src/session.rs | Update Session::start signature + imports |
| src/node/tests/test_run_net_py/test_run_net.py | Derive validator key expiry from wall-clock |
| src/node/tests/test_run_net/test_run_net_ci.sh | Make debug dump resilient without gdb |
| src/node/tests/alloc_timeline/alloc_timeline.sh | New log-to-allocation timeline helper (bash) |
| src/node/tests/alloc_timeline/alloc_timeline.py | New log-to-allocation timeline helper (python) |
| src/node/tests/alloc_timeline/README.md | Document alloc timeline tools |
| src/node/storage/src/tests/test_dynamic_boc_rc_db.rs | Update BOC reader API in tests |
| src/node/storage/src/lib.rs | Add RocksDB memory usage telemetry structs/metrics |
| src/node/storage/src/dynamic_boc_rc_db.rs | Reduce duplicate counter-load logic; telemetry timings |
| src/node/storage/src/db/rocksdb.rs | Expose approximate RocksDB memory usage |
| src/node/storage/src/archives/epoch.rs | Sum RocksDB memory usage over epochs |
| src/node/storage/src/archives/db_provider.rs | Extend provider trait with memory usage aggregation |
| src/node/storage/src/archives/archive_manager.rs | Expose archive RocksDB memory usage |
| src/node/storage/src/archive_shardstate_db.rs | Expose shardstate RocksDB memory usage |
| src/node/src/validator/validator_session_listener.rs | Handle on_candidate_observed; queue new action |
| src/node/src/validator/validator_manager.rs | Clarify applied-top semantics in docs |
| src/node/src/validator/validate_query.rs | Add engine-or-cache prev-state waiting + materialization |
| src/node/src/validator/tests/test_state_resolver_cache.rs | New unit tests for resolver cache behavior |
| src/node/src/validator/out_msg_queue_cleaner.rs | Update get_slice API usage |
| src/node/src/validator/mod.rs | Export state_resolver_cache module |
| src/node/src/validator/fabric.rs | Wire resolver cache through validation/collation entrypoints |
| src/node/src/validator/consensus.rs | Update pipeline-context semantics; new Session::start signature |
| src/node/src/types/top_block_descr.rs | Switch to stream_read for BOC parsing |
| src/node/src/types/accounts.rs | Storage-stat recalculation changes + timing logs |
| src/node/src/tests/test_helper.rs | Adjust ValidateQuery ctor args; add resolver cache in tests |
| src/node/src/tests/test_control.rs | Adapt to SmallData return from get_bytestring |
| src/node/src/shard_states_keeper.rs | Update BOC read APIs; reduce cursor usage |
| src/node/src/shard_blocks_intershard.rs | Tighten MC seqno checks + improve error messaging |
| src/node/src/network/liteserver.rs | Track account-state cache bytes for telemetry |
| src/node/src/internal_db/mod.rs | Aggregate RocksDB memory usage across DBs |
| src/node/src/engine_traits.rs | Add new telemetry/alloc counters (cache/metrics) |
| src/node/src/engine.rs | Publish new telemetry metrics; simplex Prometheus descriptions |
| src/node/src/config.rs | Clarify ADNL-key teardown ordering in comments |
| src/node/src/collator_test_bundle.rs | Wire new telemetry fields + resolver cache in bundles |
| src/node/src/block_proof.rs | Use direct-slice BOC reading |
| src/node/src/block.rs | Use direct-slice BOC reading |
| src/node/simplex/tests/test_validation.rs | Update session start args + explicit parent handling |
| src/node/simplex/tests/test_restart.rs | Update session start args + explicit parent handling |
| src/node/simplex/tests/test_collation.rs | Update session start args + explicit parent handling |
| src/node/simplex/src/tests/test_restart.rs | Extend bootstrap ordering assertions (final/skip certs, finalize chain) |
| src/node/simplex/src/tests/test_prometheus_publisher.rs | New tests for Prometheus republisher |
| src/node/simplex/src/tests/test_misbehavior.rs | Add SkipAfterFinalize test coverage |
| src/node/simplex/src/tests/test_database.rs | Add final/skip cert persistence + bootstrap tests |
| src/node/simplex/src/tests/test_candidate_resolver.rs | Extend resolver state fields; add ban-state tests |
| src/node/simplex/src/prometheus_publisher.rs | New per-session metrics republisher |
| src/node/simplex/src/misbehavior.rs | Add SkipAfterFinalize conflict reason |
| src/node/simplex/src/lib.rs | API/docs updates; add Prometheus labels config; new ensure-candidate API |
| src/node/simplex/Cargo.toml | Bump simplex crate version to 0.7.0 |
| src/node/simplex/CHANGELOG.md | Add 0.7.0 release notes |
| src/node/consensus-common/src/utils.rs | Make MetricUsage public; add enumerate_with_usage |
| src/node/consensus-common/src/lib.rs | Extend APIs: explicit parent vec, resolver flags/options, new session start signature |
| src/node/consensus-common/src/compression.rs | Use direct-slice BOC reading |
| src/node/bin/hardfork.rs | Wire resolver cache for collation path |
| src/node/bin/benchmark.rs | Update BOC read-to-storage API |
| src/node-control/contracts/src/config_contract/config_impl.rs | Adapt to SmallData bytestring API |
| src/executor/src/transaction_executor.rs | Update slice API + use precalculated storage stat |
| src/executor/src/tests/test_transaction_executor_with_real_data.rs | Update storage-stat API calls in tests |
| src/executor/src/tests/test_tr_phases.rs | Update storage-stat API calls in tests |
| src/executor/src/tests/test_tick_tock_transaction.rs | Update storage-stat API calls in tests |
| src/executor/src/tests/test_ordinary_transaction.rs | Update storage-stat API calls in tests |
| src/executor/src/tests/test_ordinary_rawreserve.rs | Update storage-stat API calls in tests |
| src/executor/src/tests/test_ordinary_libs_and_code.rs | Update storage-stat API calls in tests |
| src/executor/src/tests/test_ordinary_freeze.rs | Update storage-stat API calls in tests |
| src/executor/src/tests/common/mod.rs | Update storage-stat API calls in shared helpers |
| src/emulator/src/lib.rs | Update storage-stat API calls |
| src/block/src/wrappers.rs | Switch SHA256/SHA512 helpers to OpenSSL; adjust Ed25519 expand |
| src/block/src/tests/test_utils.rs | Update storage-stat API calls in tests |
| src/block/src/tests/test_types.rs | Adapt to SmallData bytestring API in tests |
| src/block/src/tests/test_storage_stat.rs | Rename/update storage-stat calc calls |
| src/block/src/tests/test_merkle_update.rs | Disable env var set in test |
| src/block/src/tests/test_merkle_proof.rs | Switch to stream_read in tests |
| src/block/src/tests/test_master.rs | Use EmptyValue in publishers hashmap |
| src/block/src/tests/test_crypto.rs | Switch tests to new digest helpers |
| src/block/src/tests/test_config_params.rs | Use EmptyValue in mandatory params |
| src/block/src/tests/test_boc_compression.rs | Switch to stream_read in test |
| src/block/src/tests/test_accounts.rs | Switch to stream_read; update storage-stat API calls |
| src/block/src/storage_stat.rs | Optimize storage-stat dict update (AHashMap, multiset changes) |
| src/block/src/shard.rs | Update storage-stat API usage |
| src/block/src/master.rs | Use EmptyValue in publishers hashmap |
| src/block/src/lib.rs | Introduce EmptyValue instead of () TLB true marker |
| src/block/src/dictionary/tests/test_hashmapaug.rs | Update multiset tests to new key type |
| src/block/src/dictionary/tests/test_dictionary.rs | Update multiset tests; add wrong-bitlen rejection test |
| src/block/src/dictionary/hashmapaug.rs | Change multiset signature to FixedBitsKey |
| src/block/src/dictionary/hashmap.rs | Change multiset signature; fast-path empty get |
| src/block/src/config_params.rs | Use EmptyValue for HashmapE true markers |
| src/block/src/cell/tests/test_slice.rs | Add regression/hardening tests for slice offset overflow/OOB |
| src/block/src/cell/tests/test_cell.rs | Remove Cell BOC draft test (draft API changed) |
| src/block/src/cell/slice.rs | Change get_slice signature; add overflow checks; SmallData bytestrings |
| src/block/src/cell/mod.rs | Add combined hash+depth path; new BOC draft reading flow |
| src/block/src/cell/builder.rs | Make SmallData public |
| src/block/src/boc_compression.rs | Switch to stream_read after decompress |
| src/block/src/accounts.rs | Rename storage-stat methods; add precalc accessor |
| src/block/Cargo.toml | Add OpenSSL dependency |
| src/block-json/src/tests/test_serialize.rs | Update storage-stat API calls |
| src/block-json/src/deserialize.rs | Update storage-stat API calls |
| src/adnl/tests/test_overlay.rs | Disable env var set in test |
| src/adnl/src/quic/stat.rs | Add QUIC accept/delayed-accept counters; improve snapshot API |
| src/adnl/src/overlay/mod.rs | Refactor overlay maintenance loop timeouts; improve logs; remove unused helper |
| src/Makefile | Add test_quic_* target under adnl tests |
| src/Cargo.lock | Lockfile updates (simplex version, openssl dep) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+26
to
+27
| grep -E "Engine::run|telemetry$|${FILTER}" "$INPUT" \ | ||
| | awk -v filter="$FILTER" -v full="$FULL" ' |
There was a problem hiding this comment.
FILTER is documented as a substring match, but it’s interpolated into a grep -E regex. If the filter contains regex metacharacters (e.g. ., [, (), the script will match the wrong metrics or grep can error out. Consider removing ${FILTER} from the grep -E stage and letting the AWK index(name, filter) do the substring filtering, or escape the filter before passing it to grep/use a fixed-string match.
| VerifyingKey::from(&exp_key) | ||
| }; | ||
| // sha2::Sha512 is required by raw_sign's API, can't be replaced with openssl. | ||
| // ed25519 dalek mandatory use sha2 |
There was a problem hiding this comment.
Comment grammar: "ed25519 dalek mandatory use sha2" is ungrammatical and a bit unclear. Consider rephrasing to something like "ed25519-dalek requires sha2 for raw_sign" so it reads cleanly and is easier to understand later.
LenyKholodov
approved these changes
Apr 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.