-
Notifications
You must be signed in to change notification settings - Fork 301
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(batchrouter): honour upload frequency when limitsReached if destination is failing #3874
Conversation
router/batchrouter/handle.go
Outdated
@@ -245,7 +248,8 @@ func (brt *Handle) getWorkerJobs(partition string) (workerJobs []*DestinationJob | |||
if batchDest, ok := destinationsMap[destID]; ok { | |||
var processJobs bool | |||
brt.lastExecTimesMu.Lock() | |||
if limitsReached && !brt.forceHonorUploadFrequency { // if limits are reached, process all jobs regardless of their upload frequency |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
**Note: ** forceHonorUploadFrequency
is being removed in #3871
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #3874 +/- ##
==========================================
+ Coverage 68.81% 68.84% +0.03%
==========================================
Files 351 351
Lines 52746 52752 +6
==========================================
+ Hits 36299 36319 +20
+ Misses 14130 14112 -18
- Partials 2317 2321 +4
☔ View full report in Codecov by Sentry. |
@@ -116,6 +116,9 @@ type Handle struct { | |||
lastExecTimesMu sync.RWMutex | |||
lastExecTimes map[string]time.Time | |||
|
|||
failingDestinationsMu sync.RWMutex | |||
failingDestinations map[string]bool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
failingDestinations map[string]bool | |
failingDestinations map[string]struct{} |
Is it possible to use empty structs here, since currently it is being used as set only?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then we would need to delete from the set when the destination starts succeeding. Both are equally valid solutions in my head. I went with the one that requires the less amount of lines :)
…tination is failing
e8e3a93
to
1a6bacb
Compare
Description
When a destination is failing there is no point in keep retrying jobs continuously even if there are a lot of jobs in the queue (limitsReached in jobsdb query).
Linear Ticket
PIPE-283
Security