Skip to content

Feature: Task groups with shared concurrency limits #10

@deepjoy

Description

@deepjoy

Summary

Allow tasks to be tagged with a group key, with per-group concurrency limits enforced by the scheduler alongside the global limit.

Motivation

When multiple sync profiles target the same S3 endpoint, the global concurrency limit alone isn't enough. Three profiles with 4 concurrent transfers each would hit the same bucket with 12 simultaneous connections, potentially triggering 503 SlowDown responses or overwhelming the endpoint. We need the ability to say "max 6 concurrent tasks against this specific endpoint, regardless of how many profiles use it."

This is a general pattern beyond S3 — any shared resource (database, API endpoint, external service) benefits from scoped concurrency limits.

Proposed Behavior

  • TaskSubmission accepts an optional group: String (e.g. an endpoint URL, API host, resource identifier)
  • SchedulerConfig accepts per-group concurrency limits:
    let config = SchedulerConfig::builder()
        .max_concurrent_tasks(16)  // global
        .group_concurrency("s3://us-east-1.amazonaws.com", 6)
        .group_concurrency("s3://play.min.io", 4)
        .default_group_concurrency(8)  // for groups without explicit limits
        .build();
  • The scheduler enforces both the global limit and the per-group limit — a task only runs when it has a slot in both
  • Groups can be added/updated at runtime (new endpoints discovered dynamically)
  • SchedulerSnapshot / TypeStats expose per-group active/pending counts

Design Considerations

  • Group keys should be arbitrary strings to stay generic (not tied to any domain like S3)
  • Priority scheduling should still work within groups — a high-priority task in a full group should preempt a low-priority one in the same group
  • Consider whether group limits should be hard caps or soft targets that the backpressure system can influence
  • A task without a group is only subject to the global limit (backwards compatible)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions