Skip to content

rust(feat): Adds FlowDescriptor and FlowBuilder to improve performance#396

Merged
tsift merged 5 commits into
mainfrom
rust/flow-descriptor-and-flow-builder
Nov 22, 2025
Merged

rust(feat): Adds FlowDescriptor and FlowBuilder to improve performance#396
tsift merged 5 commits into
mainfrom
rust/flow-descriptor-and-flow-builder

Conversation

@tsift
Copy link
Copy Markdown
Contributor

@tsift tsift commented Nov 20, 2025

Creates two new public structs, the FlowDescriptor and the FlowBuilder. These are intended to abstract the complete FlowConfig and ChannelConfig to the minimum required for ingesting data once registered to Sift.

Internally, the FlowDescriptor will have an index map of some key that maps to the index of that channel within the flow; it will also hold the data type of that field to improve validation while building a flow.

The FlowDescriptor can be created manually, or by converting a FlowConfig. Here, we are using strings as the "key", though the key can be arbitrary, so long as it is unique per channel. For maximum performance, the returned indices for each registered channel should then be used when setting the value in order to avoid a hash lookup when building the flow below:

let mut flow_descriptor = FlowDescriptor::new("ingestion_config_id", "my_flow_name");
let my_channel_idx = flow_descriptor.add("my_channel_key", ChannelDataType::String);
let my_other_channel_idx = flow_descriptor.add("my_other_channel_key", ChannelDataType::Uint64);
let descriptor = FlowDescriptor::try_from((INGESTION_CONFIG_ID, flow)).unwrap();

Then to build a flow using the FlowBuilder, you can either use the "key" to set the value, or the index returned when registering the channel in the FlowDescriptor:

let mut flow_builder = FlowBuilder::new(&mut flow_descriptor);
flow_builder.set(my_channel_idx, "my_value".to_string());
flow_builder.set_with_key("my_other_channel_key", 123_u64);

@tsift tsift requested a review from nathan-sift November 20, 2025 21:38
@tsift tsift marked this pull request as ready for review November 21, 2025 15:59
Comment thread rust/crates/sift_stream/src/stream/mode/ingestion_config.rs Outdated
Comment thread rust/crates/sift_stream/src/stream/flow/mod.rs
@tsift tsift requested a review from nathan-sift November 22, 2025 00:43
@tsift tsift merged commit 833f092 into main Nov 22, 2025
13 checks passed
@tsift tsift deleted the rust/flow-descriptor-and-flow-builder branch November 22, 2025 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants