Skip to content

rust(bug): Improve how sift-stream handles the flow config cache#401

Merged
tsift merged 3 commits into
mainfrom
rust/sift-stream-flow-config-cache-improvements
Nov 24, 2025
Merged

rust(bug): Improve how sift-stream handles the flow config cache#401
tsift merged 3 commits into
mainfrom
rust/sift-stream-flow-config-cache-improvements

Conversation

@tsift
Copy link
Copy Markdown
Contributor

@tsift tsift commented Nov 24, 2025

Improves how sift-stream handles the flow config cache internally.

  • On initialization, if no flows are provided, sift-stream syncs with everything Sift has.
  • On initialization, if flows are provided, sift-stream only syncs those with Sift.
  • When using "add_new_flows", any flows that exist in the flow config cache are filtered out.

Note, the initialization behavior w/r/t Sift is unchanged, sift-stream simply no longer will throw away the resulting flow configs when none are provided during initialization.

These changes improve the restart behavior of sift-stream or restarts of upstream callers. For example, if an upstream entity is responsible for streaming flow "foo" and always attempts to add flow "foo" when starting, that entity disconnecting and reconnecting would previously always hit an "already exists" error for subsequent calls to "add_new_flow". The same scenario would occur if sift-stream itself needed to be recreated, since effectively the upstream entity would perform the same actions.

@tsift tsift requested a review from nathan-sift November 24, 2025 19:24
Comment thread rust/crates/sift_stream/src/stream/mode/ingestion_config.rs Outdated
@tsift tsift requested a review from nathan-sift November 24, 2025 21:04
@tsift tsift merged commit 79ed6f9 into main Nov 24, 2025
13 checks passed
@tsift tsift deleted the rust/sift-stream-flow-config-cache-improvements branch November 24, 2025 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants