feat: introduce random sleep before clickhouse loads

[Sidddddarth]

Description

Customers facing issues due to continuous inserts on clickhouse. This introduces a misc.SleepCtx before every load anywhere between (0, maxDelay] with maxDelay defaulting to 5s.

Linear Ticket

Fix Clickhouse "too many parts" errors while inserting data

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

Summary by CodeRabbit

• New Features

  • Implemented a feature to introduce a random delay in data loading processes to optimize performance under varying system loads.

• Refactor

  • Enhanced the data loading mechanism with a new configuration option for managing operation timing.

[coderabbitai]

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Walkthrough

The Clickhouse integration code has been updated to include a new feature that introduces a random delay when loading tables. This is achieved by importing the math/rand package, adding a new field to the Clickhouse struct to handle the delay, initializing this field with a delay calculation function, and invoking a sleep function with the calculated delay within the loadTable method.

Changes

File	Change Summary
.../clickhouse/clickhouse.go	Added import of math/rand, a new randomLoadDelay field to the Clickhouse struct, initialized this field, and added a call to misc.SleepCtx with a random delay in the loadTable method.

---

Tips

Chat with CodeRabbit Bot (@coderabbitai)

• If you reply to a review comment from CodeRabbit, the bot will automatically respond.
• To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment
• Note: Review comments are made on code diffs or files, not on the PR overview.
• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Note: For conversation with the bot, please use the review comments on code diffs or files.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• The JSON schema for the configuration file is available here.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

[Sidddddarth]

@coderabbitai review

[coderabbitai]

Review Status

Actionable comments generated: 2

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 0f202e8 and e10def0.

Files selected for processing (1)

• warehouse/integrations/clickhouse/clickhouse.go (4 hunks)

Additional comments: 7

warehouse/integrations/clickhouse/clickhouse.go (7)

• 11-17: The import of "math/rand" is consistent with the PR objective to introduce random sleep delays.

• 161-165: The addition of the randomLoadDelay field to the Clickhouse struct aligns with the PR objective to introduce a random sleep delay before load operations.

• 236-247: The initialization of the randomLoadDelay field with a function that calculates a random delay based on configuration parameters is consistent with the PR objective.

• 504-507: The call to misc.SleepCtx with the randomLoadDelay function within the loadTable method is consistent with the PR objective to introduce a random sleep delay before load operations.

• 246-246: The use of rand.Float64() to generate a random delay factor is correct and aligns with the PR objective.

• 504-507: The use of the randomLoadDelay function within the loadTable method to introduce a delay before the load operation is consistent with the PR objective.

• 504-507: The use of misc.SleepCtx suggests that the random delay respects the context's deadline or cancellation, which is a good practice for long-running operations.

[codecov]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (09a1ab0) 72.70% compared to head (b5b9ef1) 72.61%.

Files	Patch %	Lines
warehouse/integrations/clickhouse/clickhouse.go	71.42%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4193      +/-   ##
==========================================
- Coverage   72.70%   72.61%   -0.09%     
==========================================
  Files         387      387              
  Lines       56458    56471      +13     
==========================================
- Hits        41048    41009      -39     
- Misses      13043    13091      +48     
- Partials     2367     2371       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.