Skip to content

Conversation

@batrov
Copy link

@batrov batrov commented May 27, 2025

Add concurrent processing during compass sink

@haveiss haveiss requested a review from Copilot May 27, 2025 07:53
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces concurrent processing in the Compass sink to handle record sinking concurrently.

  • Added sync.WaitGroup and error channel for concurrent error handling
  • Wraps record processing in a goroutine for parallel execution

@batrov batrov force-pushed the concurrent-compass-sink branch from 2ecec21 to da32851 Compare May 27, 2025 08:10
@haveiss haveiss requested a review from Copilot May 27, 2025 08:14
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces concurrent processing for sinking records to the compass service. Key changes include:

  • Employing golang.org/x/sync/errgroup to process records concurrently.
  • Removing error wrapping in favor of returning raw errors.
  • Updating the test’s expected error message accordingly.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
plugins/sinks/compass/sink_test.go Updated expected error message to match changes in error propagation.
plugins/sinks/compass/sink.go Introduced concurrent processing using errgroup and modified error returns.

@batrov batrov force-pushed the concurrent-compass-sink branch 2 times, most recently from ae4d388 to c67c063 Compare May 27, 2025 08:25
@haveiss haveiss requested a review from Copilot May 27, 2025 08:25
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Add concurrent processing to the Compass sink using errgroup, update tests for the new error flow, and ensure CI artifacts are uniquely named per job.

  • Introduce an errgroup.Group to run send calls in parallel in Sink.
  • Simplify error propagation by returning underlying errors directly.
  • Parameterize coverage artifact names in the GitHub Actions workflow with ${{ strategy.job-index }}.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
plugins/sinks/compass/sink.go Use errgroup for concurrent sinking and simplify error returns
plugins/sinks/compass/sink_test.go Adjust expected error message to match the simplified error path
.github/workflows/test.yml Add job-index suffix to coverage artifact names
Comments suppressed due to low confidence (1)

plugins/sinks/compass/sink.go:99

  • [nitpick] The variable name errCh is ambiguous; consider renaming to group or eg to clarify that it’s an errgroup.Group.
errCh := errgroup.Group{}

@batrov batrov force-pushed the concurrent-compass-sink branch from c67c063 to f594bcf Compare May 27, 2025 08:32
@batrov batrov force-pushed the concurrent-compass-sink branch from f594bcf to ecf1a1f Compare May 27, 2025 08:33
@batrov batrov merged commit 1fb081f into main May 27, 2025
96 of 105 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants