Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] - Concurrent processing of binary diffs during git scans #2236

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ahrav
Copy link
Collaborator

@ahrav ahrav commented Dec 16, 2023

Description:

Refactor Git repo scanning to use workers for asynchronous processing of binary diffs.

  • Introduce a buffered binaryDiffChan to distribute work
  • Start N goroutine workers to handle binary diffs
  • Scan commits and staged changes concurrently using WaitGroup
  • Close diff channel when producers finish
  • Synchronize consumer completion before exiting
  • Propagate first error to cancel context
  • Return early if error occurs processing diffs (this preserves existing functionality)

High level design:
binary-diff-workers

This refactors the scanning logic to utilize concurrency and avoid blocking on large binaries. Multiple goroutines can now process diffs asynchronously through the buffered channel.

The context cancellation allows all workers to exit cleanly if an error happens. And synchronization points ensure proper shutdown.

This should help increase overall scanning throughput, while maintaining existing control flow around error handling.

Checklist:

  • Tests passing (make test-community)?
  • Lint passing (make lint this requires golangci-lint)?

@ahrav ahrav marked this pull request as ready for review December 17, 2023 18:14
@ahrav ahrav requested a review from a team as a code owner December 17, 2023 18:14
@ahrav ahrav marked this pull request as draft December 19, 2023 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant