feat: resync block header / configurable resync and reindex purge options#257
Merged
BlobMaster41 merged 16 commits intomainfrom Mar 12, 2026
Merged
feat: resync block header / configurable resync and reindex purge options#257BlobMaster41 merged 16 commits intomainfrom
BlobMaster41 merged 16 commits intomainfrom
Conversation
Introduce a DEV.RESYNC_BLOCK_HEIGHTS mode to allow re-processing blocks while only purging and reimporting block headers/witnesses. Changes include: - Config: add RESYNC_BLOCK_HEIGHTS and RESYNC_BLOCK_HEIGHTS_UNTIL defaults and validations; adjust PROCESS_ONLY_X_BLOCK and INDEXER.START_INDEXING_UTXO when resync mode is enabled; force OP_NET.REINDEX from block 0 in resync mode. - Safety: BlockIndexer now validates that resync ranges do not cross OPNet activation (to avoid invalid headers) and uses revertBlockHeadersOnly when resync mode is enabled; otherwise it reverts full data as before. - Indexing flow: skip persisting/deleting transactions and conflict removal when resync mode is active (Block.insertPartialTransactions, Block.finalizeBlock, IndexingTask) to avoid touching non-header state. - VM: skip MLDSA/state storage during resync in VMManager. - Storage: add abstract revertBlockHeadersOnly to VMStorage and implement it in VMMongoStorage to delete only block headers and witnesses. - Witness handling: BlockWitnessManager gains a bounded queue, per-block and global validation limits, concurrency control and cleanup of validation counters to prevent overload during resync/validation bursts. Overall this enables a lighter-weight resync that preserves non-header data while safely reprocessing headers and witnesses, and adds guardrails to prevent invalid resyncs across OPNet activation.
Add an automatic exit when DEV.RESYNC_BLOCK_HEIGHTS is set and the configured number of blocks have been reprocessed (logs a completion message and calls process.exit). Change block witness handlers from async Promise<void> to synchronous void across ClientPeerNetworking, OPNetPeer and BlockWitnessManager to simplify call sites and remove unnecessary async/await. Also tighten BlockWitnessManager queue handling (guard against undefined shift results, reformat imports, annotate catch, and small signature/spacing cleanups) and adjust some concurrency/limit constants formatting.
Reformat code for consistency: consolidate and wrap imports, remove stray blank lines, and adjust line breaks and type annotations across multiple files. Add/align explicit imports (e.g. PackageResult, TestMempoolAcceptResult in BitcoinRPCThread) and tidy message/type interface declarations. These are stylistic and type-signature formatting changes only — no functional logic changes.
BlobMaster41
added
bug
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new dev “resync headers-only” mode to reprocess a bounded block range while preserving non-header state, and introduces witness-validation backpressure/concurrency controls to avoid overload during resync/validation bursts.
Changes:
- Add
DEV.RESYNC_BLOCK_HEIGHTS/DEV.RESYNC_BLOCK_HEIGHTS_UNTILconfig defaults + validations, and wire resync mode into indexer startup/reindex flow. - Introduce
VMStorage.revertBlockHeadersOnly()and implement it inVMMongoStorageto purge only block headers + witnesses. - Adjust indexing/VM/witness flows to avoid touching transaction/state persistence during resync; add bounded witness validation queue + concurrency limits.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/src/vm/storage/databases/VMMongoStorage.ts | Implements headers+witnesses-only purge for resync mode. |
| src/src/vm/storage/VMStorage.ts | Adds abstract revertBlockHeadersOnly() API. |
| src/src/vm/VMManager.ts | Skips MLDSA/state storage during resync. |
| src/src/threading/interfaces/thread-messages/messages/api/PackageRequest.ts | Formatting/typing cleanup for RPC message interfaces. |
| src/src/threading/interfaces/thread-messages/messages/api/MempoolTransactionNotification.ts | Formatting/typing cleanup for mempool notification message types. |
| src/src/poc/peer/OPNetPeer.ts | Makes onBlockWitness callback synchronous and updates wiring. |
| src/src/poc/networking/p2p/BlockWitnessManager.ts | Adds bounded queue + per-block/global validation limits and concurrency control. |
| src/src/poc/networking/client/ClientPeerNetworking.ts | Updates onBlockWitness callback type and destroy cleanup. |
| src/src/poc/mempool/verificator/bitcoin/v2/BitcoinTransactionVerificatorV2.ts | Import formatting cleanup. |
| src/src/poc/mempool/manager/Mempool.ts | Import formatting cleanup. |
| src/src/db/repositories/PublicKeysRepository.ts | Import formatting cleanup. |
| src/src/config/interfaces/IBtcIndexerConfig.ts | Extends DEV config with resync flags and makes PROCESS_ONLY_X_BLOCK mutable. |
| src/src/config/BtcIndexerConfigLoader.ts | Adds resync defaults, validation, and config adjustments when enabled. |
| src/src/blockchain-indexer/rpc/thread/BitcoinRPCThread.ts | Consolidates imports + minor formatting. |
| src/src/blockchain-indexer/processor/tasks/IndexingTask.ts | Skips tx deletion/conflict removal during resync. |
| src/src/blockchain-indexer/processor/block/Block.ts | Skips tx persistence and OPNet tx saving during resync. |
| src/src/blockchain-indexer/processor/BlockIndexer.ts | Validates resync range vs OPNet activation and calls headers-only revert in resync mode. |
| src/src/api/websocket/handlers/HandlerRegistry.ts | Import + formatting cleanup. |
| src/src/api/routes/api/v1/transaction/BroadcastTransactionPackage.ts | Formatting cleanup and minor refactors. |
| src/src/api/routes/api/v1/transaction/BroadcastTransaction.ts | Formatting cleanup for notification broadcast. |
| src/src/api/json-rpc/types/interfaces/results/transactions/BroadcastTransactionPackageResult.ts | Removes stray leading blank line. |
| src/src/api/json-rpc/types/interfaces/params/transactions/BroadcastTransactionPackageParams.ts | Formatting cleanup for params interface. |
| src/src/api/ServerThread.ts | Import formatting cleanup. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
BlockIndexer: tighten checks around OPNet resyncing. Add explicit handling for the case where OPNet is enabled from block 0 (throwing a clear error that resync is not allowed), and keep the existing guard that RESYNC_BLOCK_HEIGHTS_UNTIL must be less than the OPNet activation block. This prevents attempts to resync OPNet-enabled blocks which would produce invalid block headers.
Introduce configurable resync and reindex purge options and implement batched DB purging to avoid MongoDB timeouts. Adds new config keys (RESYNC_BLOCK_HEIGHTS, RESYNC_BLOCK_HEIGHTS_UNTIL, REINDEX_PURGE_UTXOS, REINDEX_PURGE_PUBLIC_KEYS, REINDEX_BATCH_SIZE) and validation logic in the config loader; updates example/sample config files. Adds range-deletion methods across multiple repositories (transactions, unspent txs, contracts, pointers, block headers/witnesses, epochs, epoch submissions, reorgs, MLDSA public keys) and hooks them into VMMongoStorage so purges occur in configurable batches and respect the purge flags. BlockIndexer now validates that the requested resync height does not exceed the highest indexed block and provides a clearer resync-complete log message.
BlobMaster41
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 36 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Several cross-cutting edits: - BtcIndexerConfigLoader: Added type check and a >0 validation for OP_NET.REINDEX_BATCH_SIZE with clear error messages to prevent invalid config values. - BlockIndexer: Updated the RESYNC COMPLETE banner to use dashed separators for a cleaner notice. - VMMongoStorage: Replaced two separate warns with a consolidated important notice when REINDEX_PURGE_UTXOS or REINDEX_PURGE_PUBLIC_KEYS is false, explicitly stating which items will not be purged and how to enable a full purge. - RoswellConsensus: Changed regtest BLOCK default from 0n to 100n. - OPNetPeer: Simplified the default onBlockWitness handler from an async noop to a synchronous noop. These changes improve config safety, clarify runtime notices, and adjust defaults/handlers for expected behavior.
Cleanup and refactor across multiple modules: - Core: removed duplicate start() definition and kept a single startup flow that initializes DB, identity and threads. - PeerChecker: moved logColor, removed duplicate methods, and added external APIs: isPeerUnreachable, resetDialFailures, and recordExternalDialFailure to centralize dial-failure tracking and notify on unreachable peers. - EpochManager: deduplicated compareBytes by consolidating the helper into a single private method. - Several files: normalized and compacted import formatting (GetPendingTransaction, GetLatestPendingTransactions, MempoolTransactionConverter, TransactionRepository, OPNetPathFinder, VMMongoStorage) to improve consistency. These changes are primarily organizational, remove duplicated code, and expose peer failure utilities for use by external components.
Remove the REINDEX_PURGE_PUBLIC_KEYS config option and related logic across the codebase. Updated docker and sample config files to drop the setting, removed the property from the IBtcIndexerConfig interface, removed the default and validation in BtcIndexerConfigLoader, and simplified VMMongoStorage to always purge MLDSA public keys during reindex. Adjusted the reindex purge notice message to reference only UTXO purge. This simplifies reindex purge behaviour by making public-key purge unconditional.
Replace uses of sodium.sodium_malloc with Node Buffer.alloc across EncryptemClient.ts, EncryptemServer.ts, and KeyPairGenerator.ts. Allocations for auth keys, nonces, cipher/decrypted buffers, signatures, and seeds now use Buffer.alloc while preserving existing sodium.randombytes_* and crypto_* calls. Also fix an incorrect reference in EncryptemServer.generateNonce to use this.sodium.crypto_box_NONCEBYTES. No functional changes to cryptographic logic; this simplifies memory handling by using Node Buffers instead of libsodium's sodium_malloc API.
Ensure upperBound is at least blockId and introduce an initial unbounded purge pass to remove orphaned data above the known height. Add sequential (dev) and parallel (non-dev via Promise.safeAll) delete routines that call delete*FromBlockHeight(...) for wide deletions, then perform a batched reverse walk from upperBound down to blockId to delete data in ranges. Also refactor range variables (from/to) and replace previous batchEnd usage; include MLDSA public keys and other repositories in both unbounded and batched purges.
Introduce MongoDBConfigurationDefaults (readPreference: PRIMARY_PREFERRED, directConnection: true, connect/socket timeouts, appName) and pass it into ConfigurableDBManager instances across the codebase to unify Mongo client options. Updated files: BlockIndexer, DBManager, MempoolManager (bitcoin-mempool), Mempool (manager), SSH, and VMMongoStorage. Also includes minor import/formatting tweaks.
Add comprehensive reorganization and startup purge test suite for BlockIndexer. New tests under tests/reorg cover revertChain behavior, startup purge/init logic, edge-cases, observer/watchdog interactions and mempool preservation. The changes introduce many test files and test helpers/mocks (VM storage, ChainObserver, BlockFetcher, ReorgWatchdog, EpochReindexer, config and others) to validate execution order, chainReorged lifecycle, stopAllTasks, thread/plugin notifications, error propagation, argument forwarding, processing-error interactions, concurrent revert protection, REINDEX/RESYNC/EPOCH_REINDEX startup flows and OPNet validation. Also add mock modules and setup utilities used across the suite.
Ensure UTXO state is restored correctly during live chain reorganisations by always purging UTXOs in live reorg paths. VMStorage.revertDataUntilBlock now accepts an optional purgeUtxos flag; BlockIndexer passes true for live reorgs. VMMongoStorage respects the override (falling back to Config when undefined). Tests updated and extended to assert the new behavior (including new cases for purgeUtxosOverride), plus various test cleanup and minor formatting fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introduce a DEV.RESYNC_BLOCK_HEIGHTS mode to allow re-processing blocks while only purging and reimporting block headers/witnesses. Changes include:
Overall this enables a lighter-weight resync that preserves non-header data while safely reprocessing headers and witnesses, and adds guardrails to prevent invalid resyncs across OPNet activation.
Introduce configurable resync and reindex purge options and implement batched DB purging to avoid MongoDB timeouts. Adds new config keys (RESYNC_BLOCK_HEIGHTS, RESYNC_BLOCK_HEIGHTS_UNTIL, REINDEX_PURGE_UTXOS, REINDEX_PURGE_PUBLIC_KEYS, REINDEX_BATCH_SIZE) and validation logic in the config loader; updates example/sample config files. Adds range-deletion methods across multiple repositories (transactions, unspent txs, contracts, pointers, block headers/witnesses, epochs, epoch submissions, reorgs, MLDSA public keys) and hooks them into VMMongoStorage so purges occur in configurable batches and respect the purge flags. BlockIndexer now validates that the requested resync height does not exceed the highest indexed block and provides a clearer resync-complete log message.
Type of Change
Checklist
Build & Tests
npm installcompletes without errorsnpm run buildcompletes without errorsnpm testpasses all testsCode Quality
Documentation
Security
OP_NET Node Specific
Testing
Consensus Impact
Related Issues
By submitting this PR, I confirm that my contribution is made under the terms of the project's license.