feat: add configurable limit on concurrent bulk dispatch goroutines

[ycombinator]

What is the problem this PR solves?

During a 30k Serverless scale test, 22 of 39 fleet-server pods were OOMKilled. Analysis of the captured pod logs showed:

• An upgrade + policy reassignment storm caused checkin durations to spike from normal (~ms) to 5-45 seconds average.
• This created ~479 concurrent checkins per pod, each blocked in dispatch() waiting to enqueue onto the bulk engine's channel (capacity 32).
• Elasticsearch was not the bottleneck — ES had ~88% free heap after GC with sub-30ms pause times.
• Each blocked goroutine holds its stack (~4-8KB) plus the bulkT object. With no upper bound on concurrent dispatches, goroutines piled up until pods hit their memory limit (~154 Mi) and were killed.

How does this PR solve the problem?

This adds an optional cap on concurrent dispatch goroutines to bound memory usage.

The limit check runs at the top of dispatch(), before blocking on the channel:

fleet-server/internal/pkg/bulk/engine.go

Lines 608 to 622 in 1f456fb

	// Check pending dispatch limit before blocking on the channel.
	if limit := b.opts.maxPendingDispatches; limit > 0 {
	pending := b.pendingDispatches.Add(1)
	defer b.pendingDispatches.Add(-1)
	if pending > int64(limit) {
	zerolog.Ctx(ctx).Warn().
	Str("mod", kModBulk).
	Str("action", blk.action.String()).
	Int64("pending", pending).
	Int("limit", limit).
	Msg("Dispatch rejected: too many pending")
	b.freeBlk(blk)
	return respT{err: ErrTooManyDispatches}
	}
	}

When the limit is reached, the dispatch is rejected immediately with ErrTooManyDispatches, which maps to HTTP 429 so agents retry on their next checkin interval:

fleet-server/internal/pkg/api/error.go

Lines 181 to 189 in 1f456fb

	{
	bulk.ErrTooManyDispatches,
	HTTPErrResp{
	http.StatusTooManyRequests,
	"TooManyDispatches",
	"too many pending dispatches",
	zerolog.WarnLevel,
	},
	},

The limit is configurable via max_pending_dispatches in the bulk config:

fleet-server/internal/pkg/config/input.go

Line 52 in 1f456fb

MaxPendingDispatches int `config:"max_pending_dispatches"`

The default is 0 (no limit) so existing deployments are unaffected. Operators opt in by setting a value appropriate for their deployment size.

How to test this PR locally

go test -race ./internal/pkg/bulk/ -run TestDispatch -v

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works

Related issues

• Relates fix: free bulkT on dispatch abort to prevent memory leak under load #6747

🤖 Generated with Claude Code

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[michel-laterman]

We really should be consistent with the type of maxPendingBulkDispatches; either have it as an int64 in the implementation + config or just an int

Also the new tests don't follow our existing test structures with the use of the require test package

[michel-laterman]

As the linter says, errors.Is should be used.
Or the require package for the error check

[ycombinator]

Fixed in 9548f86.

[michel-laterman]

We should add something to AGENTS.md that gets claude to use the require package

[ycombinator]

Done in a separate PR: #6756

[michel-laterman]

Shouldn't this be int64?

[ycombinator]

Fixed in b0f71eb.