Skip to content

Comments

fix(connectors): prevent state loss on source connector restart#2743

Merged
hubcio merged 1 commit intomasterfrom
fix/connector-restart-state-persistence
Feb 16, 2026
Merged

fix(connectors): prevent state loss on source connector restart#2743
hubcio merged 1 commit intomasterfrom
fix/connector-restart-state-persistence

Conversation

@hubcio
Copy link
Contributor

@hubcio hubcio commented Feb 16, 2026

Source handler tasks were fire-and-forget (JoinHandle dropped),
so the tokio runtime could cancel them mid-state-save during
shutdown. FileStateProvider::save() uses a non-atomic truncate
then write sequence - cancellation between the two left the
state file empty. On restart the connector loaded no tracking
offsets and re-polled all rows from the source database.

Retain handler JoinHandles and await them (with timeout) after
dropping flume senders but before shutting down Iggy clients.
Also add sync_all() after write_all() in save() to ensure
data reaches disk before returning.

Source handler tasks were fire-and-forget (JoinHandle dropped),
so the tokio runtime could cancel them mid-state-save during
shutdown. FileStateProvider::save() uses a non-atomic truncate
then write sequence — cancellation between the two left the
state file empty. On restart the connector loaded no tracking
offsets and re-polled all rows from the source database.

Retain handler JoinHandles and await them (with timeout) after
dropping flume senders but before shutting down Iggy clients.
Also add sync_all() after write_all() in save() to ensure
data reaches disk before returning.
@hubcio hubcio merged commit 44be42f into master Feb 16, 2026
33 checks passed
@hubcio hubcio deleted the fix/connector-restart-state-persistence branch February 16, 2026 12:03
@codecov
Copy link

codecov bot commented Feb 16, 2026

Codecov Report

❌ Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.97%. Comparing base (88a598a) to head (a4d2285).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
core/connectors/runtime/src/state.rs 0.00% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2743      +/-   ##
============================================
+ Coverage     68.96%   68.97%   +0.01%     
  Complexity      641      641              
============================================
  Files           736      736              
  Lines         60898    60910      +12     
  Branches      57311    57323      +12     
============================================
+ Hits          41996    42011      +15     
+ Misses        16858    16853       -5     
- Partials       2044     2046       +2     
Flag Coverage Δ
rust 70.21% <73.33%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
core/connectors/runtime/src/main.rs 86.09% <100.00%> (+0.12%) ⬆️
core/connectors/runtime/src/source.rs 66.29% <100.00%> (+0.56%) ⬆️
core/connectors/runtime/src/state.rs 56.00% <0.00%> (-4.87%) ⬇️

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants