Skip to content

Commit

Permalink
llbsolver: fix possible deadlock in history listen
Browse files Browse the repository at this point in the history
The events for currently active builds were sent through
pubsub channel instead of directly to the current request,
like it was done for completed builds for example.

This meant that if there are more active builds running
than the pubsub channel buffer (32) the sends will block.
Because the history API mutex is held in this process it
will eventually block the requests for builds that try to
update their history records.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 1ed9931)
  • Loading branch information
tonistiigi authored and crazy-max committed Nov 29, 2023
1 parent 646e71a commit 5f41ca6
Showing 1 changed file with 10 additions and 1 deletion.
11 changes: 10 additions & 1 deletion solver/llbsolver/history.go
Original file line number Diff line number Diff line change
Expand Up @@ -762,21 +762,30 @@ func (h *HistoryQueue) Listen(ctx context.Context, req *controlapi.BuildHistoryR
}()
}

// make a copy of events for active builds so we don't keep a lock during grpc send
actives := make([]*controlapi.BuildHistoryEvent, 0, len(h.active))

for _, e := range h.active {
if req.Ref != "" && e.Ref != req.Ref {
continue
}
if _, ok := h.deleted[e.Ref]; ok {
continue
}
sub.send(&controlapi.BuildHistoryEvent{
actives = append(actives, &controlapi.BuildHistoryEvent{
Type: controlapi.BuildHistoryEventType_STARTED,
Record: e,
})
}

h.mu.Unlock()

for _, e := range actives {
if err := f(e); err != nil {
return err
}
}

if !req.ActiveOnly {
events := []*controlapi.BuildHistoryEvent{}
if err := h.opt.DB.View(func(tx *bolt.Tx) error {
Expand Down

0 comments on commit 5f41ca6

Please sign in to comment.