feat: [DSM-142] Use CanisterStates in ReplicatedState#10287
Open
alin-at-dfinity wants to merge 3 commits into
Open
feat: [DSM-142] Use CanisterStates in ReplicatedState#10287alin-at-dfinity wants to merge 3 commits into
CanisterStates in ReplicatedState#10287alin-at-dfinity wants to merge 3 commits into
Conversation
Switches `ReplicatedState::canister_states` from a flat
`BTreeMap<CanisterId, Arc<CanisterState>>` to `CanisterStates`,
exposing the hot/cold partition to the rest of the system and
migrating every caller.
`ReplicatedState` changes:
* `canister_states` field is now `CanisterStates`.
* Drop `canisters_iter_mut()`. Round-level callers move to
`hot_canisters_iter_mut()` (skips the long tail of cold
canisters); bulk callers move to `canisters_for_each_mut` /
`canisters_try_for_each_mut`, which iterate every canister and
re-establish the partition afterwards.
* Add `hot_canisters_iter()` for read-only hot-only iteration.
* Add `repartition_canister_states()`, called from
`StateManager::commit_and_certify` after
`flush_checkpoint_ops_and_page_maps` to drive canisters that went
quiet during the round back into `cold` before checkpointing, so
that replicas continuing through a checkpoint and replicas
(re)starting from it agree on the partition.
* `take_canister_states` / `put_canister_states` now exchange the
`CanisterStates` directly instead of going through a flat
`BTreeMap` round-trip.
* Aggregator delegations: `total_compute_allocation`, `memory_taken`,
`total_canister_memory_usage`,
`guaranteed_response_message_memory_taken`,
`best_effort_message_memory_taken`, `callback_count` now delegate
to `CanisterStates` and run in `O(|hot|)`.
`state_manager`:
* `commit_and_certify` calls `state.repartition_canister_states()`
after `flush_checkpoint_ops_and_page_maps` and before tip
handover.
* `validate_eq_canister_states` calls
`CanisterStates::validate_strict_split` on the reference state to
verify that the persisted partition matches what
`CanisterStates::new` would produce on a fresh load.
* `flush_checkpoint_ops_and_page_maps` and
`switch_to_checkpoint` switch from `canisters_iter_mut` to
`canisters_for_each_mut` / `canisters_try_for_each_mut`.
* Bench: `bench_traversal` likewise.
`execution_environment`:
* `scheduler.rs`: scheduler hot-only iteration where appropriate
(`add_heartbeat_and_global_timer_tasks`,
`purge_expired_ingress_messages`, the
`ongoing_long_install_code` check); migrate
`charge_canisters_for_resource_allocation_and_usage` and the
log-memory-store migration loop to `canisters_for_each_mut`.
* `round_schedule.rs`: `partition_canisters_to_cores` now takes /
returns a `CanisterStates`; idle canisters are dropped before the
main hot-canister iteration.
* `query_handler.rs`, `execution_environment.rs`: callers updated.
* `canister_manager/tests.rs`, scheduler tests
(`scheduling.rs`, `metrics.rs`, `dts.rs`, `ecdsa.rs`,
`round_schedule/tests.rs`, `test_utilities.rs`, `tests.rs`)
updated.
* `benches/scheduler.rs`: updated.
`canonical_state`:
* `lazy_tree_conversion.rs`: new `CanisterStatesFork<'_>` that
presents a `CanisterStates` as a `LazyFork` over the merged
hot+cold pools in `CanisterId` order.
`canister_sandbox`:
* `sandboxed_execution_controller.rs`: switch
`evict_sandbox_processes` to per-id `state.canister_state(id)` /
`state.canister_priority(id)` lookups (also enables removing the
bulk `canister_accumulated_priorities` method). This duplicates
the standalone "perf: Look up sandbox scheduler priorities per
canister" PR; whichever lands first, the other becomes a no-op.
`messaging`:
* `stream_handler/tests.rs`: pre-heat `LOCAL_CANISTER` in the
`out_of_memory` reject-signal test so that the expected and
inducted states share the same hot/cold partition.
* `stream_builder/tests.rs`, `state_machine/tests.rs`,
`tests/common/mod.rs`: caller updates.
`replicated_state` queues and system_state:
* `CanisterQueues` / `SystemState` `local_canisters` parameter type
flips from `&BTreeMap<CanisterId, Arc<CanisterState>>` to
`&CanisterStates` (no behavioural change; queues only need
`contains_key`).
* `replicated_state.rs` deletes the now-unused
`canister_accumulated_priorities` method.
`metrics.rs`:
* `check_dts` walks `hot_canisters_iter()` (only hot canisters can
have non-empty task queues).
* `check_subnet_memory_usage` switches to
`CanisterStates::memory_taken()` for `O(|hot|)` aggregation.
`test_utilities` and `state_tool`:
* `test_utilities/execution_environment`, `test_utilities/state`,
and `state_tool/src/commands/canister_metrics.rs` updated to use
the new iteration APIs.
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Switches
ReplicatedState::canister_statesfrom a flatBTreeMap<CanisterId, Arc<CanisterState>>toCanisterStates, exposing the hot/cold partition to the rest of the system and migrating every caller.ReplicatedStatechanges:canister_statesfield is nowCanisterStates.canisters_iter_mut(). Round-level callers move tohot_canisters_iter_mut()(skips the long tail of cold canisters); bulk callers move tocanisters_for_each_mut/canisters_try_for_each_mut, which iterate every canister and re-establish the partition afterwards.hot_canisters_iter()for read-only hot-only iteration.repartition_canister_states(), called fromStateManager::commit_and_certifyafterflush_checkpoint_ops_and_page_mapsto drive canisters that went quiet during the round back intocoldbefore checkpointing, so that replicas continuing through a checkpoint and replicas (re)starting from it agree on the partition.take_canister_states/put_canister_statesnow exchange theCanisterStatesdirectly instead of going through a flatBTreeMapround-trip.total_compute_allocation,memory_taken,total_canister_memory_usage,guaranteed_response_message_memory_taken,best_effort_message_memory_taken,callback_countnow delegate toCanisterStatesand run inO(|hot|).state_manager:commit_and_certifycallsstate.repartition_canister_states()afterflush_checkpoint_ops_and_page_mapsand before tip handover.validate_eq_canister_statescallsCanisterStates::validate_strict_spliton the reference state to verify that the persisted partition matches whatCanisterStates::newwould produce on a fresh load.flush_checkpoint_ops_and_page_mapsandswitch_to_checkpointswitch fromcanisters_iter_muttocanisters_for_each_mut/canisters_try_for_each_mut.bench_traversallikewise.execution_environment:scheduler.rs: scheduler hot-only iteration where appropriate (add_heartbeat_and_global_timer_tasks,purge_expired_ingress_messages, theongoing_long_install_codecheck); migratecharge_canisters_for_resource_allocation_and_usageand the log-memory-store migration loop tocanisters_for_each_mut.round_schedule.rs:partition_canisters_to_coresnow takes / returns aCanisterStates; idle canisters are dropped before the main hot-canister iteration.query_handler.rs,execution_environment.rs: callers updated.canister_manager/tests.rs, scheduler tests (scheduling.rs,metrics.rs,dts.rs,ecdsa.rs,round_schedule/tests.rs,test_utilities.rs,tests.rs) updated.benches/scheduler.rs: updated.canonical_state:lazy_tree_conversion.rs: newCanisterStatesFork<'_>that presents aCanisterStatesas aLazyForkover the merged hot+cold pools inCanisterIdorder.messaging:stream_builder/tests.rs,state_machine/tests.rs,tests/common/mod.rs: caller updates.replicated_statequeues and system_state:CanisterQueues/SystemStatelocal_canistersparameter type flips from&BTreeMap<CanisterId, Arc<CanisterState>>to&CanisterStates(no behavioral change; queues only needcontains_key).metrics.rs:check_dtswalkshot_canisters_iter()(only hot canisters can have non-empty task queues).check_subnet_memory_usageswitches toCanisterStates::memory_taken()forO(|hot|)aggregation.test_utilitiesandstate_tool:test_utilities/execution_environment,test_utilities/state, andstate_tool/src/commands/canister_metrics.rsupdated to use the new iteration APIs.