[Platform] Support per-stream retry/resume and stream subset triggering for connection syncs

[Chris van den Hurk (sun-cv)]

Support per-stream retry/resume and stream subset triggering for connection syncs

Problem / Current Behavior

A connection is the unit of execution: if any single stream in a multi-stream connection fails, the only way to recover is a full-connection re-sync. This re-runs every stream that already succeeded, which is wasteful and, for sources with a shared account-level rate limit, can consume the entire rate-limit budget before the failed stream even gets retried. There's currently no way to retry only the failed streams, trigger a subset of streams on demand, or control the order streams run in within a connection.

Requested Enhancement

1. Retry/resume only the streams that failed on the next sync attempt, instead of re-running the whole connection.
2. Ability to trigger a sync for a named subset of streams.
3. Stream-level ordering/prioritization within a connection, so critical streams run first when time or rate-limit budget is constrained.

Use Case

Some source APIs impose a low, shared rate limit across all streams (e.g., a fixed number of export jobs per hour, shared across 10+ streams), often within a limited sync window (e.g., an overnight batch window). In this situation, one stream failing forces a full-connection retry that re-consumes rate-limit allowance already spent on successful streams — sometimes making it impossible to complete a sync within the available window at all. Per-stream retry and prioritization would let syncs recover from partial failures without wasting the source's rate-limit budget.

Additional Context

This applies broadly to any source connector where the underlying API enforces a shared, low, account-level rate limit across many streams rather than per-stream limits.

[Airbyte Support Bot (airbyte-support-bot)]

Hi Chris van den Hurk (@sun-cv), thank you for the detailed write-up!

The use case is clear: with a shared, low, account-level rate limit across many streams, a full-connection retry can burn budget already spent on streams that succeeded, which is especially painful inside a fixed sync window. We've logged this and escalated it to our team for evaluation — tracked internally at airbytehq/oncall#13294. Note that community requests don't carry an SLA and we can't commit to a timeline.

A couple of things that may help in the meantime:

Possible interim approaches

• Split streams across multiple connections. Grouping streams into separate connections from the same source makes the connection the retry unit you actually want: a failure only re-runs the streams in that group, and you can stagger schedules so high-priority groups run first in your window. This is the closest available approximation of both per-stream retry and prioritization today.
• Check whether the connector exposes concurrency or rate-limit settings. Some sources allow lowering concurrency so a single stream is less likely to exhaust the account-level budget.
• Review sync mode and checkpointing. Incremental streams resume from their last saved state rather than re-reading everything, which reduces the cost of a re-run even when the whole connection restarts.

If you can share which source connector and Airbyte version (OSS/Self-Managed or Cloud) you're hitting this with, that context would help the team weigh the request. Please mask or remove any sensitive information (API keys, passwords, tokens, connection strings) before sharing configuration or logs.

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.