[IND-545]: Batch Process Vulcan messages

[Christopher-Li]

Changelist

Batch Process Vulcan messages. It doesn't seem like this change makes much of an impact on processing, I believe it's because Vulcan is reading from multiple partitions and the benefit would be improved if there the same number of partitions as instances of vulcan.

See research here

Test Plan

Tested in Staging and internal mainnet

Author/Reviewer Checklist

• If this PR has changes that result in a different app state given the same prior state and transaction list, manually add the state-breaking label.
• If the PR has breaking postgres changes to the indexer add the indexer-postgres-breaking label.
• If this PR isn't state-breaking but has changes that modify behavior in PrepareProposal or ProcessProposal, manually add the label proposal-breaking.
• If this PR is one of many that implement a specific feature, manually label them all feature:[feature-name].
• If you wish to for mergify-bot to automatically create a PR to backport your change to a release branch, manually add the label backport/[branch-name].
• Manually add any of the following labels: refactor, chore, bug.

[linear]

IND-545 Process Vulcan Messages in Batch

[coderabbitai]

Walkthrough

The recent updates across the Kafka-related parts of the indexer project focus on enhancing message handling and batch processing capabilities. Key changes include the renaming of a function to better reflect its updated purpose, an increase in the default heartbeat interval for Kafka consumers, and the introduction of batch processing features. These modifications aim to improve efficiency and flexibility in processing Kafka messages, aligning with evolving requirements for scalability and performance in message-driven architectures.

Changes

File(s)	Change Summary
.../kafka/__tests__/consumer.test.ts, .../kafka-controller.ts in ender, scripts, and socks services	Renamed addOnMessageFunction to updateOnMessageFunction to align with its enhanced functionality.
.../kafka/src/config.ts	Modified default values for KAFKA_CONNECTION_TIMEOUT_MS, KAFKA_SESSION_TIMEOUT_MS, KAFKA_REBALANCE_TIMEOUT_MS, and KAFKA_HEARTBEAT_INTERVAL_MS for better readability and potential impact on Kafka-related components.
.../kafka/src/consumer.ts	Extended functionality to support batch processing with new functions and variables for handling batch messages.
.../vulcan/src/config.ts	Introduced new configuration options for batch processing, including enabling batch processing and setting commit frequencies.
.../vulcan/src/helpers/kafka/kafka-controller.ts, .../vulcan/src/index.ts, .../vulcan/src/lib/on-batch.ts	Added batch processing functionality based on the BATCH_PROCESSING_ENABLED configuration, including new batch handling and committing logic.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

---

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit-tests for this file.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit tests for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository from git and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit tests.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• The JSON schema for the configuration file is available here.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

[coderabbitai]

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 39046a5 and 66771e0.

Files selected for processing (10)

• indexer/packages/kafka/tests/consumer.test.ts (2 hunks)
• indexer/packages/kafka/src/config.ts (1 hunks)
• indexer/packages/kafka/src/consumer.ts (3 hunks)
• indexer/services/ender/src/helpers/kafka/kafka-controller.ts (2 hunks)
• indexer/services/scripts/src/print-block.ts (2 hunks)
• indexer/services/socks/src/lib/message-forwarder.ts (2 hunks)
• indexer/services/vulcan/src/config.ts (2 hunks)
• indexer/services/vulcan/src/helpers/kafka/kafka-controller.ts (2 hunks)
• indexer/services/vulcan/src/index.ts (1 hunks)
• indexer/services/vulcan/src/lib/on-batch.ts (1 hunks)

Additional comments: 10

indexer/services/ender/src/helpers/kafka/kafka-controller.ts (1)

• 23-23: Renaming addOnMessageFunction to updateOnMessageFunction aligns with the intention to standardize function naming across the service. Ensure all references and documentation are updated to reflect this change.

indexer/services/vulcan/src/config.ts (1)

• 24-30: The introduction of BATCH_PROCESSING_ENABLED, KAFKA_BATCH_PROCESSING_COMMIT_FREQUENCY, and KAFKA_BATCH_PROCESSING_COMMIT_FREQUENCY_MS with default values enhances the configurability of batch processing. Ensure these new configurations are documented in the service's configuration guide.

indexer/packages/kafka/__tests__/consumer.test.ts (1)

• 29-29: Updating the function name in the test case from addOnMessageFunction to updateOnMessageFunction ensures consistency with the source code changes. Verify that all related test cases have been updated accordingly.

indexer/packages/kafka/src/config.ts (1)

• 24-24: Modifying the KAFKA_HEARTBEAT_INTERVAL_MS default value from 2000ms to 5000ms could affect consumer behavior, especially in environments with network latency issues. Ensure this change is documented and communicated to users who might need to adjust their configurations accordingly.

indexer/services/vulcan/src/helpers/kafka/kafka-controller.ts (1)

• 25-39: The conditional logic to enable batch processing based on the BATCH_PROCESSING_ENABLED configuration is a good approach. Ensure that the onBatch and onMessage functions are thoroughly tested in both batch processing enabled and disabled scenarios to confirm correct behavior.

indexer/services/vulcan/src/index.ts (1)

• 40-40: Modifying startService to accept the BATCH_PROCESSING_ENABLED configuration as a parameter and passing it to startConsumer aligns with the introduction of batch processing functionality. Ensure that all calls to startService throughout the codebase are updated to pass this configuration.

indexer/services/vulcan/src/lib/on-batch.ts (1)

• 11-92: The onBatch function implements logic for processing batches of Kafka messages, including error handling for empty batches and committing offsets at a configurable frequency. Ensure that performance testing is conducted, especially for large batches, to optimize the commit frequency and minimize processing delays.

indexer/packages/kafka/src/consumer.ts (1)

• 87-102: Extending the Kafka consumer to support batch processing with a configurable flag is a significant enhancement. Ensure that the onBatchFunction and onMessageFunction are thoroughly tested in isolation and in combination to confirm that batch and individual message processing work as expected under various scenarios.

indexer/services/scripts/src/print-block.ts (1)

• 69-69: Updating the function name to updateOnMessageFunction in the script ensures consistency with the Kafka package changes. Verify that the script's functionality related to Kafka message handling remains unaffected by this change.

indexer/services/socks/src/lib/message-forwarder.ts (1)

• 66-66: The update to use updateOnMessageFunction in the MessageForwarder class aligns with the renaming in the Kafka package. Ensure that the message forwarding functionality is thoroughly tested to confirm that messages are correctly handled and forwarded.

[coderabbitai]

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 66771e0 and 8436f71.

Files selected for processing (4)

• indexer/packages/kafka/src/config.ts (1 hunks)
• indexer/services/ender/src/helpers/kafka/kafka-controller.ts (2 hunks)
• indexer/services/vulcan/src/config.ts (1 hunks)
• indexer/services/vulcan/src/lib/on-batch.ts (1 hunks)

Files skipped from review as they are similar to previous changes (4)

• indexer/packages/kafka/src/config.ts
• indexer/services/ender/src/helpers/kafka/kafka-controller.ts
• indexer/services/vulcan/src/config.ts
• indexer/services/vulcan/src/lib/on-batch.ts