Skip to content

Conversation

@chedieck
Copy link
Collaborator

Related to #1037

Description

Optimize syncing by avoiding unnecessary address refetching, preventing duplicate price connections, refining logs, and having the initial sync block addresses message processing one-by-one, not all at the same time.

Also have Supertokens core fixed in version 10.0.0.

Test plan

Run in this branch make dev-from-dump using prod's DB as dump.sql in the repo root folder.

Check the logs. Make a TX to a address and see it appear and say it is waiting for syncing to unblock it. See when that address is synced, and the new TX message being processed.

Maybe run something like select count(*) from Address where syncing = 0; in the DB, while the sync is happening, to see how many addresses have been cleared so far (and will now process new txs normally)

@Klakurka Klakurka requested a review from Copilot August 11, 2025 21:19
@Klakurka Klakurka added the enhancement (behind the scenes) Stuff that users won't see label Aug 11, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes the blockchain synchronization process by implementing selective address blocking during initial sync, preventing duplicate price connections, and refining logging. The key improvement is changing from blocking all addresses during initialization to blocking addresses individually while they're being synced.

  • Addresses are now blocked individually during sync rather than all at once during initialization
  • Price connection process is moved to after initialization to prevent duplicates
  • Enhanced logging for better debugging and monitoring of sync progress

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
services/chronikService.ts Main sync optimization logic, address blocking changes, and initialization flow updates
services/addressService.ts Added batch address syncing function for efficient database operations
services/transactionService.ts Enhanced logging for price connection process
tests/unittests/chronikService.test.ts Updated test to match new Address parameter type
docker-compose*.yml Fixed Supertokens core version to 10.0.0
constants/index.ts Increased initialization delay from 500ms to 2000ms

return
}
while (this.mempoolTxsBeingProcessed > MAX_MEMPOOL_TXS_TO_PROCESS_AT_A_TIME) {
while (this.mempoolTxsBeingProcessed >= MAX_MEMPOOL_TXS_TO_PROCESS_AT_A_TIME) {
Copy link

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The condition changed from > to >= will cause the system to wait when exactly MAX_MEMPOOL_TXS_TO_PROCESS_AT_A_TIME transactions are being processed, potentially reducing throughput unnecessarily. The original > condition was correct as it allows processing up to the maximum limit.

Suggested change
while (this.mempoolTxsBeingProcessed >= MAX_MEMPOOL_TXS_TO_PROCESS_AT_A_TIME) {
while (this.mempoolTxsBeingProcessed > MAX_MEMPOOL_TXS_TO_PROCESS_AT_A_TIME) {

Copilot uses AI. Check for mistakes.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this.mempoolTxsBeingProcessed === MAX_MEMPOOL_TXS_TO_PROCESS_AT_A_TIME, it means we already have the max being processed, so it should keep waiting.

Comment on lines +728 to 729
this.initializing = false
await connectAllTransactionsToPrices()
Copy link

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting this.initializing = false before calling connectAllTransactionsToPrices() could cause race conditions. If WebSocket messages arrive after initialization is marked complete but before price connections finish, they may not wait for address syncing to complete properly.

Suggested change
this.initializing = false
await connectAllTransactionsToPrices()
await connectAllTransactionsToPrices()
this.initializing = false

Copilot uses AI. Check for mistakes.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the reason to wait missed txs to be synced with prices to processes txs that have arrived during the sync. Only reason could be too many DB connections, as far as I can see, which I don't think is reason enough, but I could be wrong

chedieck and others added 2 commits August 11, 2025 18:25
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@Klakurka Klakurka self-requested a review August 11, 2025 22:13
Klakurka
Klakurka previously approved these changes Aug 11, 2025
Copy link
Member

@Klakurka Klakurka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works great! It's really fast locally so I want to see how it is in prod.

@Klakurka
Copy link
Member

Just some intergation test failures.

@Klakurka Klakurka merged commit 97d3b32 into master Aug 11, 2025
2 checks passed
@Klakurka Klakurka linked an issue Aug 14, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement (behind the scenes) Stuff that users won't see

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Supertokens version needs to be set to v10.0.0 build successfully

3 participants