Version 0.15.0

Major changes

Reservoir — New Storage Backend

A brand-new storage backend called Reservoir has been introduced and is now the recommended storage solution for all deployments.
Reservoir is designed for cloud-native workloads and provides a multi-tier architecture:

• In-Flight Buffer — volatile RAM layer for Read-Your-Own-Writes consistency.
• Local Disk Cache — immutable, CRC-verified cache of 64 MB blobs fetched on-demand from remote storage.
• Persistent Storage — authoritative state in cloud object storage (Azure Blob Storage, local disk, etc.).

Key design highlights include page packing into large immutable files to minimize cloud API calls, CRC32/CRC64 checksums at page and file level for data integrity, snapshot checkpoints for fast recovery, and automatic compaction of fragmented files.

Legacy storage backends (FasterKV, temporary file cache, SQL Server) remain available but are no longer recommended for new deployments. FasterKV has been moved to its own NuGet package (FlowtideDotNet.Storage.FasterKV) and is no longer included by default.

To learn more, visit State Persistence docs.

StarRocks Connector

A new connector for StarRocks has been added, starting with a sink operator. The StarRocks sink writes data using the built-in Stream Load mechanism and requires the target table to have a primary key defined for upsert and delete support.

Two execution modes are available:

• OnCheckpoint — data is committed transactionally at checkpoint boundaries.
• OnWatermark — data is flushed on each watermark for lower latency (at-least-once semantics).

To learn more, visit StarRocks Connector docs.

OpenLineage Support

Flowtide now has built-in support for reporting data lineage events to an OpenLineage-compatible endpoint over HTTP. When enabled, the reporter automatically sends lineage events as the stream transitions between states (starting, running, completed, or failed). Schema information can optionally be included in the events.

To learn more, visit OpenLineage docs.

INSERT OVERWRITE for Delta Lake

The Delta Lake connector now supports INSERT OVERWRITE {tableName} as an alternative to INSERT INTO. This replaces all content in the delta table with the new data from the stream on the initial load. After the initial overwrite, subsequent inserts, updates, and deletes behave normally. The existing data is committed as remove operations in the delta log, so the full table history is preserved.

Additionally, column mapping is now supported when writing to delta tables.

SpiceDB materialize_permission Table Function

The SpiceDB connector has been refactored and now exposes a materialize_permission SQL table function for easily denormalizing permissions directly in your queries. The function automatically fetches the schema from SpiceDB at plan-build time, removing the need for manual schema loading.

SELECT subject_type, subject_id, relation, resource_type, resource_id
FROM spicedb.materialize_permission('document', 'view')

To learn more, visit SpiceDB Connector docs.

.NET 10 Support

Flowtide now targets .NET 10 in addition to .NET 8, allowing projects on the latest runtime to benefit from the newest performance improvements and language features.

Minor changes

New SQL Functions

Several new scalar and aggregate functions have been added:

• datediff — calculates the difference between two dates in a specified date part.
• round_calendar — rounds a timestamp to a calendar boundary (e.g., start of hour, day, month).
• xxhash64 — computes an xxHash64 hash of input values.
• log10 — computes the base-10 logarithm.
• count_distinct — aggregate function that counts distinct values.
• avg (window function) — average is now available as a window function. A bug with using window functions inside recursive CTEs has also been fixed.

Generic Data Sink: Fetch Existing Data

The generic data sink now allows fetching existing data from the target on initial startup. This is used to compare against incoming data to avoid duplicates and to produce deletes for data that is no longer present in the stream result.

SharePoint Source Refactored to Column-Based Data

The SharePoint source has been rebuilt to use column-based data and now supports resync operations.

JsonElement Support in C# Object Converter

The C# object converter now supports JsonElement as a type, allowing direct mapping of JSON data when converting objects to columns.

Precompiled Frontend

The ASP.NET Core monitoring UI now ships with a precompiled frontend. Node.js is no longer required to build the project.

Kafka Improvements

• Tombstone events are now skipped during initial load when loading existing data from Kafka.
• The Kafka sink can once again handle deleted rows during initial batch processing.

Delta Lake Source Fixes

• The delta lake source can now read tables that only have a checkpoint as the first entry.
• A fix ensures correct difference detection in column batch reads.

Memory Optimization for Window Functions

Memory allocation on the managed heap has been reduced for MIN/MAX window functions, decreasing GC pressure.

Bug Fixes

• Union column insert range — fixed incorrect count calculation when null values are present in the range.
• C# Object Sink — fixed an exception when removing a row with a non-nullable field.
• SQL case sensitivity — single identifiers in SQL are no longer case sensitive.
• Surrogate key / upper metadata — upper and surrogate key int64 now return correct types for metadata.
• Plan serialization — list and map types now serialize correctly to JSON when serializing the plan.
• Delta load task reset — fixed an issue where the delta load task was not reset when a full load followed a delta load.

What's Changed

• Bump Microsoft.Data.SqlClient from 6.1.1 to 6.1.2 by @dependabot[bot] in #889
• changed versioning builder to allow combinations by @bpfz in #891
• [Snyk] Upgrade @docusaurus/core from 3.9.0 to 3.9.1 by @Ulimo in #893
• Fasterkv dynamic directory by @Ulimo in #892
• Add connector for starrocks, starting with sink by @Ulimo in #895
• Change deserialize methods to use ReadOnlySequence instead of ReadOnlyMemory by @Ulimo in #896
• Add permissions in benchmark pipeline by @Ulimo in #902
• Bump Apache.Arrow from 16.1.0 to 22.1.0 by @dependabot[bot] in #897
• Bump Apache.Arrow.Compression from 19.0.0 to 22.1.0 by @dependabot[bot] in #898
• Bump Aspire.Hosting.AppHost from 9.1.0 to 9.5.2 by @dependabot[bot] in #899
• Bump Aspire.Hosting.Azure and Aspire.Hosting.Azure.Storage by @dependabot[bot] in #901
• Skip tombstone events in initial load when loading existing data from Kafka by @Ulimo in #903
• Check that there is a difference of data in column batch read in delta by @Ulimo in #904
• [Snyk] Upgrade next from 14.2.32 to 14.2.33 by @Ulimo in #905
• Add packaging step for Starrocks connector by @Ulimo in #906
• Fix so types list and map serialize correctly to json when serializing plan by @Ulimo in #907
• Bump Apache.Arrow.Compression from 19.0.0 to 22.1.0 by @dependabot[bot] in #908
• Bump Aspire.Hosting.Elasticsearch from 9.1.0-preview.1.25121.10 to 9.5.0 by @dependabot[bot] in #909
• Bump Aspire.Hosting.MongoDB from 9.1.0 to 9.5.2 by @dependabot[bot] in #910
• Bump Aspire.Hosting.SqlServer from 9.1.0 to 9.5.2 by @dependabot[bot] in #911
• Bump Aspire.Hosting.Testing from 8.1.0 to 9.5.2 by @dependabot[bot] in #912
• [Snyk] Upgrade @docusaurus/core from 3.9.1 to 3.9.2 by @Ulimo in #913
• Add support for column mapping when writing to delta table by @Ulimo in #914
• Bump Aspire.Microsoft.Data.SqlClient from 9.0.0 to 9.5.2 by @dependabot[bot] in #915
• Bump AspNetCore.HealthChecks.Publisher.ApplicationInsights from 8.0.1 to 9.0.0 by @dependabot[bot] in #916
• Bump BenchmarkDotNet and BenchmarkDotNet.Diagnostics.Windows by @dependabot[bot] in #919
• Bump Bogus from 35.6.2 to 35.6.5 by @dependabot[bot] in #920
• Bump Azure.Monitor.OpenTelemetry.Exporter and OpenTelemetry.Extensions.Hosting by @dependabot[bot] in #921
• Add support for datediff function by @Ulimo in #922
• Add missing permissions to release workflow by @Copilot in #924
• Update Node.js version in GitHub Actions workflow by @Ulimo in #923
• C# Object Sink: Fix removing a row with non nullable field throws exception by @Ulimo in #925
• Bump js-yaml from 4.1.0 to 4.1.1 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot[bot] in #926
• Bump Aspire.Hosting.AppHost from 9.5.2 to 13.0.0 by @dependabot[bot] in #928
• Bump Aspire.Hosting.Azure and Aspire.Hosting.Azure.Storage by @dependabot[bot] in #930
• Bump Aspire.Hosting.Elasticsearch from 9.5.0 to 13.0.0 by @dependabot[bot] in #931
• [Snyk] Upgrade @docusaurus/preset-classic from 3.9.0 to 3.9.2 by @Ulimo in #927
• Bump node-forge from 1.3.1 to 1.3.2 in /docs by @dependabot[bot] in #934
• Bump Aspire.Hosting.AppHost from 13.0.0 to 13.0.1 by @dependabot[bot] in #935
• Bug fix in union column insert range, fix wrong count calculation if null is in the range by @Ulimo in #941
• Bump Aspire.Hosting.Azure and Aspire.Hosting.Azure.Storage by @dependabot[bot] in #937
• Bump Aspire.Hosting.MongoDB from 9.5.2 to 13.0.1 by @dependabot[bot] in #938
• Bump Aspire.Hosting.SqlServer from 9.5.2 to 13.0.1 by @dependabot[bot] in #939
• Bump mdast-util-to-hast from 13.2.0 to 13.2.1 in /docs by @dependabot[bot] in #944
• Bump Aspire.Hosting.Testing from 9.5.2 to 13.0.1 by @dependabot[bot] in #940
• Bump express from 4.21.2 to 4.22.1 in /docs by @dependabot[bot] in #945
• Fix so upper and surrogate key int64 returns correct types for metadata by @Ulimo in #943
• Add support for jsonelement type in C# object converter to column by @Ulimo in #942
• Bump Aspire.Hosting.AppHost from 13.0.1 to 13.0.2 by @dependabot[bot] in #946
• Bump Aspire.Hosting.Azure and Aspire.Hosting.Azure.Storage by @dependabot[bot] in #948
• Bump Aspire.Hosting.MongoDB from 13.0.1 to 13.0.2 by @dependabot[bot] in #949
• Bump Aspire.Hosting.SqlServer from 13.0.1 to 13.0.2 by @dependabot[bot] in #950
• Bump Aspire.Hosting.Testing from 13.0.1 to 13.0.2 by @dependabot[bot] in #951
• Bump next from 14.2.33 to 14.2.35 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot[bot] in #952
• Bump Aspire.Microsoft.Data.SqlClient from 9.5.2 to 13.0.2 by @dependabot[bot] in #953
• Bump Azure.Monitor.OpenTelemetry.Exporter and OpenTelemetry.Extensions.Hosting by @dependabot[bot] in #954
• Bump BenchmarkDotNet and BenchmarkDotNet.Diagnostics.Windows by @dependabot[bot] in #956
• Bump Bogus from 35.6.1 to 35.6.5 by @dependabot[bot] in #957
• Bump Confluent.Kafka from 2.10.0 to 2.12.0 by @dependabot[bot] in #958
• Bump Aspire.Hosting.AppHost from 13.0.2 to 13.1.0 by @dependabot[bot] in #959
• Bump Aspire.Hosting.Azure and Aspire.Hosting.Azure.Storage by @dependabot[bot] in #961
• Bump Aspire.Hosting.MongoDB from 13.0.2 to 13.1.0 by @dependabot[bot] in #962
• Bump Aspire.Hosting.SqlServer from 13.0.2 to 13.1.0 by @dependabot[bot] in #963
• Bump Aspire.Hosting.Testing from 13.0.2 to 13.1.0 by @dependabot[bot] in #964
• Allow fetching existing data in generic data sink by @Ulimo in #968
• Bump Aspire.Microsoft.Data.SqlClient from 13.0.2 to 13.1.0 by @dependabot[bot] in #965
• Bump coverlet.collector from 6.0.2 to 6.0.4 by @dependabot[bot] in #966
• Bump CsvHelper from 33.0.1 to 33.1.0 by @dependabot[bot] in #967
• Bump qs from 6.14.0 to 6.14.1 in /docs by @dependabot[bot] in #970
• Bump coverlet.collector from 6.0.0 to 6.0.4 by @dependabot[bot] in #971
• Bump CsvHelper from 31.0.0 to 33.1.0 by @dependabot[bot] in #972
• Bump Fody and InlineIL.Fody by @dependabot[bot] in #976
• Enable kafka sink to handle deleted rows again in initial batch by @Ulimo in #977
• Remove unnecessary tests in sql server sink by @Ulimo in #978
• Add xxhash64 scalar function by @Ulimo in #979
• Reduce memory allocation on managed heap for min max by in window functions by @Ulimo in #983
• Bump Confluent.Kafka from 2.12.0 to 2.13.0 by @dependabot[bot] in #980
• Bump Google.Protobuf from 3.26.1 to 3.33.3 by @dependabot[bot] in #981
• Bump Grpc.Net.Client from 2.61.0 to 2.76.0 by @dependabot[bot] in #982
• Bump Aspire.Hosting.Elasticsearch from 13.0.0 to 13.1.0 by @dependabot[bot] in #984
• Bump Google.Protobuf from 3.33.3 to 3.33.4 by @dependabot[bot] in #985
• Bump Grpc.Tools from 2.61.0 to 2.76.0 by @dependabot[bot] in #986
• Increase fasterKV default page size by @Ulimo in #988
• Fix so delta lake source can read tables that only have a checkpoint as first entry by @Ulimo in #987
• Bump lodash from 4.17.21 to 4.17.23 in /docs by @dependabot[bot] in #989
• Bump Microsoft.AspNetCore.Mvc.Testing from 8.0.14 to 8.0.23 by @dependabot[bot] in #990
• Bump Microsoft.Azure.Cosmos from 3.47.2 to 3.57.0 by @dependabot[bot] in #991
• Bump Google.Protobuf from 3.33.4 to 3.33.5 by @dependabot[bot] in #997
• Bump Azure.Monitor.OpenTelemetry.Exporter from 1.5.0 to 1.6.0 by @dependabot[bot] in #995
• Bump next from 14.2.35 to 16.1.5 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot[bot] in #994
• Bump webpack from 5.100.2 to 5.105.0 in /docs by @dependabot[bot] in #998
• Add average window function, and fix using window functions in recursive CTEs by @Ulimo in #1004
• Add count distinct aggregate function by @Ulimo in #1005
• Fix starrocks sink to use database casing of column names by @Ulimo in #1006
• Add round_calendar function by @Ulimo in #1007
• Add log10 function by @Ulimo in #1008
• Add Addition function as a column based function by @Ulimo in #1009
• Add .net 10 as a target framework by @Ulimo in #1010
• Substrait serialization and deserialization fixes by @Ulimo in #1012
• Bump Microsoft.EntityFrameworkCore.Design and Microsoft.EntityFrameworkCore.Relational by @dependabot[bot] in #1003
• Update packages in docs by @Ulimo in #1013
• Update packages for the stream UI by @Ulimo in #1014
• Update GitHub Actions test workflow to include permissions by @Ulimo in #1015
• Change release action to use .net 10 by @Ulimo in #1016
• Update test command to include framework version by @Ulimo in #1017
• Update elasticsearch client to 9.3 by @Ulimo in #1018
• [DeltaLake] Fix various sink issues by @Ulimo in #1019
• Update substrait proto to latest version by @Ulimo in #1020
• Add support for overwrite in SQL and in the write relation by @Ulimo in #1021
• Add support for INSERT OVERWRITE for delta tables by @Ulimo in #1022
• Bump qs from 6.14.1 to 6.14.2 in /docs by @dependabot[bot] in #1023
• Bump Aspire.Hosting.AppHost from 13.1.0 to 13.1.1 by @dependabot[bot] in #1024
• Bump Aspire.Hosting.Azure and Aspire.Hosting.Azure.Storage by @dependabot[bot] in #1026
• Bump Aspire.Hosting.MongoDB from 13.1.0 to 13.1.1 by @dependabot[bot] in #1027
• Bump Aspire.Hosting.SqlServer from 13.1.0 to 13.1.1 by @dependabot[bot] in #1028
• Enable overwrite for delta lake and fix tests for delta connector by @Ulimo in #1030
• Bump Aspire.Microsoft.Data.SqlClient from 13.1.0 to 13.1.1 by @dependabot[bot] in #1029
• Move bitmap list from core to storage data structures by @Ulimo in #1032
• Bump Aspire.Hosting.Testing from 13.1.0 to 13.1.1 by @dependabot[bot] in #1033
• Bump Confluent.Kafka from 2.13.0 to 2.13.1 by @dependabot[bot] in #1034
• Bump coverlet.collector from 6.0.4 to 8.0.0 by @dependabot[bot] in #1035
• Bump Microsoft.AspNetCore.Mvc.Testing from 8.0.23 to 8.0.24 by @dependabot[bot] in #1036
• Bump Microsoft.EntityFrameworkCore.Design and Microsoft.EntityFrameworkCore.Relational by @dependabot[bot] in #1037
• Bump Aspire.Hosting.AppHost from 13.1.1 to 13.1.2 by @dependabot[bot] in #1038
• Bump Aspire.Hosting.MongoDB from 13.1.1 to 13.1.2 by @dependabot[bot] in #1041
• Bump Aspire.Hosting.Azure and Aspire.Hosting.Azure.Storage by @dependabot[bot] in #1040
• Bump Aspire.Hosting.SqlServer from 13.1.1 to 13.1.2 by @dependabot[bot] in #1042
• Bump Aspire.Microsoft.Data.SqlClient from 13.1.1 to 13.1.2 by @dependabot[bot] in #1043
• Bump svgo from 3.3.2 to 3.3.3 in /docs by @dependabot[bot] in #1044
• Add xml documentation in base project by @Ulimo in #1045
• Move all vertices into a common namespace by @Ulimo in #1046
• Move streamstatevalue from context to engine root folder by @Ulimo in #1047
• Bump Aspire.Hosting.Testing from 13.1.1 to 13.1.2 by @dependabot[bot] in #1048
• Refactor SpiceDb connector by @Ulimo in #1053
• Add setting for max file size, this is calculated when data is added into the batch by @Ulimo in #1058
• SpiceDB Denormalization: Add ConvertAsync to help fetch schema after schemaservice is internal by @Ulimo in #1059
• Add support for connector provided table functions, start with spicedb materialize permissions by @Ulimo in #1060
• Add Reservoir a new storage solution for state by @Ulimo in #1061
• Bump Confluent.Kafka from 2.13.1 to 2.13.2 by @dependabot[bot] in #1049
• Bump Google.Protobuf from 3.33.5 to 3.34.0 by @dependabot[bot] in #1051
• Bump Microsoft.Azure.Cosmos from 3.57.0 to 3.57.1 by @dependabot[bot] in #1052
• Bump Elastic.Clients.Elasticsearch from 9.3.0 to 9.3.2 by @dependabot[bot] in #1054
• Bump Microsoft.AspNetCore.Mvc.Testing from 8.0.24 to 8.0.25 by @dependabot[bot] in #1055
• Bump next from 16.1.6 to 16.1.7 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot[bot] in #1056
• Bump flatted from 3.3.3 to 3.4.2 in /src/FlowtideDotNet.AspNetCore/ClientApp by @dependabot[bot] in #1057
• Bump brace-expansion from 1.1.12 to 1.1.13 in /docs by @dependabot[bot] in #1062
• Move FasterKV to its own project so its not inccluded by default by @Ulimo in #1063
• Add Azure Blobs packaging step to release workflow by @Ulimo in #1064
• Bug fix: Fix that delta load task was not reset when a full load followed a delta load by @Ulimo in #1065
• Fix so azure blobs get a stream with length to allow less calls to azure API by @Ulimo in #1067
• Fix single identifiers being case sensitive in sql by @Ulimo in #1066
• Fix code quality, make fields readonly by @Ulimo in #1068
• Code quality: remove unused containers by @Ulimo in #1069
• Code quality: Fix futile synchronization on field by @Ulimo in #1070
• Code quality: Useless assignment to local variable by @Ulimo in #1071
• Code quality: Fix disposables by @Ulimo in #1072
• Refactor sharepoint source to use column based data, also resync support by @Ulimo in #1073
• Add OpenLineage support by @Ulimo in #1074
• Regenerate lock file for docs by @Ulimo in #1075
• Upgrade frontend package and regenerate yarn.lock by @Ulimo in #1076
• Include precompiled frontend, no longer require node to build the project by @Ulimo in #1078

New Contributors

• @Copilot made their first contribution in #924

Full Changelog: v0.14.4...v0.15.0