Declarative connector gets stuck with no logs after 58k records

[MiguelRiosLo]

I have a custom low-code connector (YAML/manifest) for the Jira/Atlassian Assets API, which syncs with an MSSQL target, implemented in Airbyte. Synchronization stops advancing steadily after approximately 57,900 records in total, always loading 946 MB and always while processing one of several streams that follow the same form: a SubstreamPartitionRouter that expands to one HTTP GET request per primary record (/object/{id}/history), with NoPagination in the secondary retriever and concurrency_level.max_concurrency: 1.

There is no error, warning or registry output once the connector reaches this point. Sync simply stops moving forward and remains locked without syncing any additional logs. I manually stopped work after waiting 20–25 minutes with no activity, but it looks like it could stay in this state indefinitely (even for days) if left running.

I am using Airbyte's free OSS resources. Could this have any impact on this behavior? For example, could limited resources or resource throttling cause the connector to appear locked while waiting for an HTTP response?

The only message displayed after manually stopping work is:

"Terminal workload; leaving."

This message comes from the workload orchestrator/launcher, not the connector or destination. As far as I can tell, the connector seems to get stuck waiting for one of its HTTP requests to complete, and nothing in the manifest scheme seems to allow setting an HTTP request timeout that would raise the issue as a visible error.

Has anyone experienced a similar issue with declarative connectors or SubstreamPartitionRouter flows?

Does anyone have any idea what could be causing the connector to jam without causing logs or errors?

Are there any known limitations or timeout/resource issues when running Airbyte OSS with the free/default resources that could explain this behavior?

Any suggestions or ideas would be greatly appreciated. Thanks!

[Airbyte Support Bot (airbyte-support-bot)]

Hi MiguelRiosLo, thanks for the detailed write-up!

We're looking into this. Nothing is confirmed yet, but a few observations and things that would help narrow it down.

The detail that stands out most is that the stall is reproducible at roughly the same point every time (~57.9k records / ~946 MB). A hang that always lands on the same data volume tends to point at a resource ceiling or downstream backpressure rather than a single unlucky HTTP request, which would be expected to vary run to run.

Suggested things to check

1. Container memory limits. If the source or destination pod is being killed or is thrashing near its memory limit, the sync can appear frozen with no connector-level error. Check the pod's memory usage/limits and kubectl describe pod (or docker stats for docker-compose deployments) for OOM or eviction events around the time it stalls.
2. Isolate source vs. destination. Run the same streams against a throwaway destination (for example a local JSON/file destination) with the same catalog. If it sails past 946 MB, the stall is more likely on the MSSQL side or in the platform's record pipe rather than in the manifest.
3. Test the source alone. Running the source image directly with read --config ... --catalog ... and piping to a file removes the destination and the orchestrator from the picture entirely.
4. Platform-side logs. The connector logs won't show much for this class of stall. The workload launcher / orchestrator pod logs, and the destination container logs, are usually where the useful signal is.
5. Narrow the substream. If you can reproduce with a single one of those /object/{id}/history streams and a reduced parent set, that makes it far easier for us to investigate.

To help us dig in, could you share:

• Your Airbyte OSS version and deployment method (abctl, Helm/Kubernetes, docker-compose).
• The source-declarative-manifest / CDK version the custom connector is running on.
• The destination connector name and version.
• Whether the stall point moves if you change the stream selection, or whether it is always at the same record/byte count.
• Any platform-side logs (workload launcher, orchestrator, destination container) from the stall window, pasted inline in a fenced code block.

Please make sure to mask or remove any sensitive information (API keys, passwords, tokens, connection strings, hostnames) before sharing logs or manifest excerpts.

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.

[Airbyte Support Bot (airbyte-support-bot)]

We've escalated this to our team for investigation: airbytehq/oncall#13238

The details requested above (OSS version, CDK / source-declarative-manifest version, destination version, and any platform-side logs from the stall window) would still help a lot — please add them here and they'll be picked up.

[MiguelRiosLo]

Hi!

We managed to isolate this to something much more specific. We ran a sync selecting only one stream (no other streams active), so this is a clean, minimal reproduction with zero prior sync activity to confound it.

Versions (from this run's logs):

• source-declarative-manifest: 7.24.0
• destination-mssql: 2.2.19
• container-orchestrator: 2.0.2-alpha-c905e75
• Platform workload heartbeat: interval=10s, timeout=600s

What the logs show:

1. Sync starts at 13:33:59, syncing only this stream.
2. Between 13:34:00 and 13:39:51 (~5m51s), the connector logs 9,999 "slice:" lines - one per object_id - e.g. slice:{"object_id": "100812", "parent_slice": {}}. This is the SubstreamPartitionRouter enumerating the parent stream and generating one partition per object. No records are emitted during this phase, which is expected.
3. Immediately after the last slice is logged (13:39:51), the connector goes completely silent. No HTTP request logs to /object/{id}/history, no RECORD messages, no errors, nothing - for 10+ minutes straight (until our log capture ends at 13:49:58). The only lines in that window are internal "No checkpoints to flush" messages every ~60s from the CheckpointManager.
4. Given the platform's workload heartbeat timeout is 600s (10 minutes) and we're well past that with zero heartbeat-eligible activity (no RECORD/STATE messages), this run was very likely killed shortly after our log capture ends, the same "Workload in terminal state; exiting" pattern we reported before.

Why we think this is significant: this reproduction has zero records synced before the stall - it's not a matter of accumulating volume or resource exhaustion over a long-running sync (which was our previous working theory). The connector appears to hang specifically in the transition between "partition/slice generation for the substream is complete" and "the first child HTTP request is issued." With ~10,000 partitions generated via SubstreamPartitionRouter (this stream retriever uses NoPagination against GET /object/{id}/history, one request per object_id, partitioned from the parent stream), that transition appears to never happen, or takes far longer than the 600s heartbeat timeout allows.

This also lines up with what we saw in the full multi-stream sync: the *_objects streams (which use real pagination) completed and flushed batches normally; the stall consistently began once the sync reached the *_object_history streams - i.e. at the same slice-generation-to-first-request transition, just reached later because other streams were processed first.

We don't have infrastructure-level access (only the manifest YAML and SSH to the host, no kubectl/docker beyond that), so we can't inspect the source connector's internal state (thread pool, executor queue) during that silent window. Given how cleanly this now reproduces with a single isolated stream, would it be possible to get guidance on:

• Whether there are known issues with SubstreamPartitionRouter + NoPagination handling large partition counts (~10k) in source-declarative-manifest 7.24.0
• Any debug/verbose logging flag we could enable from the connector config to see what's happening between slice generation and the first child request
• Whether increasing the platform's heartbeat timeout (though we understand this needs platform-level flags.yaml access, not something we can set from the manifest) would at least let the sync survive long enough to confirm it eventually starts, or whether it's a true deadlock that time alone won't resolve.