Refactor: Move in-memory store into IndexerState - #1330
Merged
Conversation
Flatten the Ctx wrapper (config, persistence, inMemoryStore) directly onto IndexerState and drop the construction-only registrations field, which is now passed straight to ChainManager.makeFromDbState. EventProcessing and startServer take the persistence/config they use instead of a Ctx. Delete Ctx.res. https://claude.ai/code/session_01UnGJ5W6o9QKzV7JJz9q8Qi
Flatten the in-memory store fields (entity/effect tables, pending-write queue, chain metadata staging) onto IndexerState and drop InMemoryStore.t. The InMemoryStore module now holds store operations over IndexerState.t; the loop mutex/progress counters (isProcessing, processedBatchesCount) become plain state fields. A single onError replaces the store's separate write-error handler, wrapping write failures with the batch-write message inline. Extract the contractRegister context out of UserContext into ContractRegisterContext so ChainFetcher's fetch-time registration no longer pulls in the store, breaking the IndexerState -> ChainManager -> ChainFetcher -> UserContext -> IndexerState cycle. https://claude.ai/code/session_01UnGJ5W6o9QKzV7JJz9q8Qi
Move the storage write loop and its capacity/flush coordination out of InMemoryStore into a dedicated Writing module, a peer to ChainFetching/ BatchProcessing/Rollback: runWriteLoop, schedule (formerly kick), runOneWrite, drainBatchRun, snapshotEffects, awaitCapacity, flush, commitBatch, setChainMeta and the commit-waiter coordination. InMemoryStore keeps the entity/effect table primitives plus prepareRollbackDiff and setBatchDcs. Callers commit/flush/stage through Writing; the write fiber is still kicked by data availability so the backpressure path is unchanged. https://claude.ai/code/session_01UnGJ5W6o9QKzV7JJz9q8Qi
Add IndexerState.resi with an abstract type t. Other modules no longer touch fields directly: they read through accessors and change state through named domain operations rather than raw setters — queueProcessedBatch, drainBatchRun, takeRollback, markCommitted, beginRollbackDiff, recordWriteFailure, begin/endWriteFiber, wakeCommitWaiters/addCommitWaiter, stageChainMeta and takeChainMetaSnapshot. Writing keeps the async write-loop orchestration and InMemoryStore the entity/effect table primitives, both operating on the opaque state through these operations. https://claude.ai/code/session_01UnGJ5W6o9QKzV7JJz9q8Qi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates the in-memory store (entity/effect tables, batch queue, write coordination) into
IndexerStateand extracts write-loop logic into a newWritingmodule. This eliminates the intermediateCtxwrapper and simplifies state threading throughout the codebase.Key Changes
IndexerState expansion: Moved
EntityTables,effectCacheInMemTable, and all write-related fields fromInMemoryStoreintoIndexerState. Added read accessors and domain operations (queueProcessedBatch,drainBatchRun,dropCommitted, etc.) to encapsulate state mutations.InMemoryStore simplification: Reduced to a thin utility module with only effect/entity table accessors (
getInMemTable,getEffectInMemTable,hasEffectOutput, etc.). Removed initialization, batch queuing, and write-loop logic.Writing module: New module owns the write loop (
runOneWrite,runWriteLoop,schedule) and capacity/flush coordination (getChangesCount,waitForCommit,commitBatch,dropCommitted). CallsIndexerStatedomain operations to mutate state.Ctx removal: Deleted
Ctx.res(was a wrapper holdingconfig,persistence,inMemoryStore,registrations). Callers now threadIndexerStatedirectly or access fields via accessors.ContractRegisterContext extraction: Moved contract-register handler context logic from
UserContextinto a newContractRegisterContextmodule, keeping it independent ofIndexerStateso fetch-time registration doesn't pull the full state.Accessor pattern:
IndexerStateexposes read-only accessors for all fields (e.g.,config,persistence,allEntities,committedCheckpointId) and write operations through named functions (e.g.,setChainManager,beginProcessing,markCommitted).Notable Implementation Details
IndexerState.tremains opaque in the interface (.resi); all mutations route through explicit operations, preventing accidental direct field access.Writing, keepingIndexerStatefocused on state representation.InMemoryStorefunctions that read state now takeIndexerState.tinstead of the oldInMemoryStore.t, maintaining the same interface but operating on the consolidated state.IndexerStatedirectly instead of buildingCtx+InMemoryStore.https://claude.ai/code/session_01UnGJ5W6o9QKzV7JJz9q8Qi