Summary
Allow tasks to be tagged with a group key, with per-group concurrency limits enforced by the scheduler alongside the global limit.
Motivation
When multiple sync profiles target the same S3 endpoint, the global concurrency limit alone isn't enough. Three profiles with 4 concurrent transfers each would hit the same bucket with 12 simultaneous connections, potentially triggering 503 SlowDown responses or overwhelming the endpoint. We need the ability to say "max 6 concurrent tasks against this specific endpoint, regardless of how many profiles use it."
This is a general pattern beyond S3 — any shared resource (database, API endpoint, external service) benefits from scoped concurrency limits.
Proposed Behavior
TaskSubmission accepts an optional group: String (e.g. an endpoint URL, API host, resource identifier)
SchedulerConfig accepts per-group concurrency limits:
let config = SchedulerConfig::builder()
.max_concurrent_tasks(16) // global
.group_concurrency("s3://us-east-1.amazonaws.com", 6)
.group_concurrency("s3://play.min.io", 4)
.default_group_concurrency(8) // for groups without explicit limits
.build();
- The scheduler enforces both the global limit and the per-group limit — a task only runs when it has a slot in both
- Groups can be added/updated at runtime (new endpoints discovered dynamically)
SchedulerSnapshot / TypeStats expose per-group active/pending counts
Design Considerations
- Group keys should be arbitrary strings to stay generic (not tied to any domain like S3)
- Priority scheduling should still work within groups — a high-priority task in a full group should preempt a low-priority one in the same group
- Consider whether group limits should be hard caps or soft targets that the backpressure system can influence
- A task without a group is only subject to the global limit (backwards compatible)
Summary
Allow tasks to be tagged with a group key, with per-group concurrency limits enforced by the scheduler alongside the global limit.
Motivation
When multiple sync profiles target the same S3 endpoint, the global concurrency limit alone isn't enough. Three profiles with 4 concurrent transfers each would hit the same bucket with 12 simultaneous connections, potentially triggering
503 SlowDownresponses or overwhelming the endpoint. We need the ability to say "max 6 concurrent tasks against this specific endpoint, regardless of how many profiles use it."This is a general pattern beyond S3 — any shared resource (database, API endpoint, external service) benefits from scoped concurrency limits.
Proposed Behavior
TaskSubmissionaccepts an optionalgroup: String(e.g. an endpoint URL, API host, resource identifier)SchedulerConfigaccepts per-group concurrency limits:SchedulerSnapshot/TypeStatsexpose per-group active/pending countsDesign Considerations