Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement queue-side coalescing (superseding without worker support) #93

Closed
wants to merge 1 commit into from

Conversation

djmitche
Copy link
Contributor

@djmitche djmitche commented Feb 7, 2018

We extend tasks with:

{
  stack: '<name>',  // requires scope: 'queue:stack:<name>'
  rank: 0, // integer task rank (higher means supersedes), defaults to zero
  ... // existing task fields
  payload: {...},
}

Tasks without task.stack will behave as if they were assigned an exclusive stack name.

claimWork will pick tasks by:

  1. Find the highest priority task (FIFO if there are multiple tasks with same priority)
  2. If another task with same task.stack and higher task.rank exists, it supersedes the task picked...

In-tree the decision task will use a per-branch string for task.stack if coalescing is supported, and task.rank = push_log_id, to ensure that newer pushes supersedes old pushes.
Example: task.stack = 'gecko:mozilla-central:<label>' where <label> is the task-label.

Note: implementation of query logic like this requires a more advanced database like postgres.

@djmitche
Copy link
Contributor

The current coalescing supports a minimum stack depth and a minimum stack age, both of which serve to prevent tasks from being coalesced when not really necessary (e.g., if two pushes land one after the other, but otherwise we are well within capacity).

@djmitche
Copy link
Contributor

RFC #65 is the postgres migration, a prerequisite to this issue.

@djmitche
Copy link
Contributor

Closing as this hasn't seen any updates in a while. It's always easy to re-open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant