v0.21.0

What's Changed

• pipeline-manager: remove service probing by @snkas in #2002
• Add tracing for dbsp steps and general docs for monitoring feldera by @gz in #1999
• [SQL] Improved monotonicity analysis for filter operators by @mihaibudiu in #2006
• DBSP: Asof-join operator. by @ryzhyk in #1998
• Merge new WebConsole layout prototype by @Karakatiza666 in #1994
• Automate Flink testing in CI by @aehmttw in #2005
• dbsp: test using window to implement SQL now. by @ryzhyk in #2015
• demo: Hopsworks Tiktok RecSys pipeline by @abhizer in #1992
• [SQL] Fix bugs in implementations of LIMIT by @mihaibudiu in #2020
• Add docs for the new WebConsole project by @Karakatiza666 in #2021
• Disable sveltekit build by @Karakatiza666 in #2024
• Re-enable and fix sveltekit build by @Karakatiza666 in #2025
• Disable coloring in maven builds by @mihaibudiu in #2026
• [SQL] Fix bug for primary key fields that need cloning by @mihaibudiu in #2029
• Update README.md by @gz in #2035
• [SQL] Preliminary implementation of the now() function by @mihaibudiu in #2019
• WebConsole: Fix pipeline relation metrics not aggregated … by @Karakatiza666 in #2036
• [SQL] Fix bugs in window bound computations for INTERVAL types by @mihaibudiu in #2034
• [SQL] Fix handling of monotonicity for tables with primary keys by @mihaibudiu in #2041
• Make sure dbsp and adapters metrics verison match. by @gz in #2044
• Update grafana_dashboard.json with some storage metrics by @gz in #2046
• [SQL] Correctly handle nullable types in ARG_MAX by @mihaibudiu in #2047
• [SQL] Use waterline and not waterline_monotonic by @mihaibudiu in #2048
• Fix bun build in deploy/Dockerfile by @Karakatiza666 in #2050
• Generalize SQL benchmarks to make it easier to add more by @aehmttw in #2013
• Reset min-storage-bytes to default by @aehmttw in #2053
• [SQL] Have the compiler synthesize the 'NOW' table by @mihaibudiu in #2049
• [SQL] Improved monotonicity analysis for joins followed by filters by @mihaibudiu in #2014
• Make Flink run all queries by @aehmttw in #2054
• TikTok demo: Fixes and refactoring. by @ryzhyk in #2055
• dbsp_adapters: Wait a bit before disconnecting on pause in url adapter. by @blp in #2057
• Improve profiling output to make storage performance easier to understand by @blp in #2059
• nexmark: Use same metrics version as the rest of the crates. by @blp in #2061
• dbsp: Name background threads so system tools can distinguish them. by @blp in #2062
• Update program.sql by @gz in #2064
• Fix port in documentation. by @gz in #2065
• [SQL] Two optimizations by @mihaibudiu in #2068
• Storage optimizations by @blp in #2066
• Revert "WebConsole: Fix pipeline relation metrics not accumulated when there are multiple connectors per relation" by @lalithsuresh in #2069
• Benchmark with 100M events by @aehmttw in #2077
• Add metric for records dropped due to lateness by @mihaibudiu in #2074
• Read input connector names from SQL by @aehmttw in #2076

Full Changelog: v0.20.0...v0.21.0

v0.20.0

What's Changed

• Post release v0.19.0 by @gz in #1944
• Improve py docs by @abhizer in #1942
• dbsp: Fix build of galen benchmark. by @blp in #1928
• py: max_queue_records param to connect_source_kafka by @ryzhyk in #1948
• Improved monotonicity analysis by @mihaibudiu in #1949
• Switch binary download port from 9090 to 8085. by @gz in #1951
• pipeline-manager: SQL MAP type by @snkas in #1941
• benchmark: API key and Kafka option arguments by @snkas in #1954
• [SQL] Do not sort collection after TOP K unless it is needed at the output by @mihaibudiu in #1958
• Fix for numpy regression. by @ryzhyk in #1961
• Update earthly version/install. by @gz in #1960
• build(deps): bump scikit-learn from 1.2.1 to 1.5.0 in /demo/demo_notebooks by @dependabot in #1888
• [SQL] Reorganize compilation stages and fix 3 bugs by @mihaibudiu in #1966
• [SQL] support MATERIALIZED table and views by @mihaibudiu in #1959
• [SQL] Improve monotonicity analysis by @mihaibudiu in #1970
• Do not automatically distinct ouputs. by @ryzhyk in #1972
• Switch from log to tracing. by @gz in #1976
• [SQL] Improve monotonicity for aggregations by @mihaibudiu in #1974
• ci: run python tests by @abhizer in #1873
• Nexmark sql output by @blp in #1979
• Minor profiling-related improvements by @ryzhyk in #1971
• WebConsole: Drop all Data Services functionality by @Karakatiza666 in #1978
• Post-fix for unsuccessful rebase in PR1971. by @ryzhyk in #1981
• Expose dbsp metrics in pipeline metrics endpoint. by @gz in #1969
• Fix docker command by @aehmttw in #1984
• Use a single tokio runtime instead of n. by @gz in #1983
• docs: feldera enterprise by @snkas in #1991
• docs: remove deploy from sidebar and fix secret management links by @snkas in #1993
• Improve Kafka input performance for small messages by @blp in #1967
• docs: eks cluster storage class check by @snkas in #1996
• dbsp: Change threshold for spilling to storage from rows to bytes. by @blp in #1917
• Make profiles downloadable via API by @blp in #1988
• benchmark: Make poller threads configurable for SQL benchmarks. by @blp in #1997
• Enable playwright tests by @Karakatiza666 in #1952
• Add script to run flink on CI by @aehmttw in #1985
• [SQL] a few small optimizations: combine join/filter/map and map/mapindex by @mihaibudiu in #1986
• Disable navigation from materialized to non-materialized view by @Karakatiza666 in #2001
• release: bump project version to 0.20.0 by @gz in #2007

Full Changelog: v0.19.0...v0.20.0

v0.19.0

What's Changed

• [SQL] Support for SQL ARRAY_AGG aggregate function by @mihaibudiu in #1862
• SQL benchmarks graph by @aehmttw in #1857
• [SQL] Do not attempt to simplify expressions that will cause panics a… by @mihaibudiu in #1869
• [SQL] Hopping and tumbling windows using table functions by @mihaibudiu in #1855
• Per-thread Storage cache. by @gz in #1853
• avro: make serializing schema ID configurable by @abhizer in #1871
• [SQL] Move simulator to separate project by @mihaibudiu in #1877
• [SQL] Fix bug in dead code elimination by @mihaibudiu in #1879
• Upgrade delta-rs to 0.18. by @ryzhyk in #1878
• [SQL] Second attempt to fix build by @mihaibudiu in #1880
• join_flatmap by @ryzhyk in #1881
• benchmark: Handle case where Kafka requires different access `host:po… by @blp in #1876
• [SQL] Use join_flatmap when possible by @mihaibudiu in #1882
• Log SQL metrics by @aehmttw in #1875
• pipeline-manager: reduce logging in integration tests by @snkas in #1885
• [SQL] Support hopping windows for nullable timestamp columns by @mihaibudiu in #1887
• [SQL] Monotonicity analysis for hopping windows by @mihaibudiu in #1889
• Run rust tests separately by @aehmttw in #1893
• [SQL] Do not assume that user-defined types are structs by @mihaibudiu in #1897
• [SQL] Introduce syntax to specify connector properties in SQL by @mihaibudiu in #1896
• Avro: Serialize non-nullable fields as nullable. by @ryzhyk in #1892
• Add a dialog to configure pipeline's runtime resources by @Karakatiza666 in #1027
• Add health popup containing all system errors to the header by @Karakatiza666 in #1856
• [SQL] Remove unsupported SQL construct (FILTER and OVER) from documentation by @mihaibudiu in #1900
• py: SQLContext.register_type by @ryzhyk in #1899
• pipeline-manager: demos supplied via local directory by @snkas in #1890
• py: add register_local_view by @abhizer in #1884
• [SQL] Fix bug in CSE pass by @mihaibudiu in #1903
• SQL storage metrics by @aehmttw in #1886
• Adds a new spine that does compaction in a thread. by @gz in #1863
• [SQL] Prevent SQRT(negative value) in STDDEV due to unstable FP computations by @mihaibudiu in #1908
• py: add_lateness. by @ryzhyk in #1910
• [SQL] Give warnings for LATENESS statements without corresponding views by @mihaibudiu in #1909
• [SQL] Preliminary support for MAP types by @mihaibudiu in #1905
• dbsp: Fix hidden dependency on vec-based batches in input_upsert. by @blp in #1913
• Avoid head-of-line blocking in async spine. by @gz in #1912
• Log more SQL metrics by @aehmttw in #1915
• Convert all DBSP batches to possibly use storage by @blp in #1867
• Sampled histograms by @blp in #1916
• dbsp_adapters: Fix typo in comment. by @blp in #1918
• Handle negative fuel in merger.rs. by @ryzhyk in #1923
• Integration test logging by @ryzhyk in #1921
• Switch from crossbeam-channel to std::sync::mpsc::channel. by @gz in #1924
• Enable storage in testing by @gz in #1927
• py: implement SQLContext.wait_for_completion by @abhizer in #1872
• [SQL] proper unnest check for null arguments by @mihaibudiu in #1930
• docs: simplify the fraud detection preamble by @lalithsuresh in #1929
• Bump delta by @ryzhyk in #1934
• py: add hopsworks integration demo by @abhizer in #1901
• Fix UI regressions by @Karakatiza666 in #1940
• Release v0.19.0 by @gz in #1943

New Contributors

• @aehmttw made their first contribution in #1857

Full Changelog: v0.18.0...v0.19.0

v0.18.0

Added

• [SQL] CREATE LOCAL VIEW statement to declare views which are not outputs. (#1851)
• Documentation: added the fraud detection use case to the docs section of the website.
• API: storage class can be configured via pipeline resources (#1829)
• [Python] Added support for Kafka connector via Python SDK (#1807)
• [Python] Added support for HTTP GET connector via Python SDK
• [Python] Add support for Avro format (#1836)
• [Python] allow specifying pipeline resource configuration and compilation profile via SQLContext (#1835)
• [Python] Add function to wait for the pipeline to become idle (#1841)
• WebConsole: DeltaLake input (#1744) and output (#1584) conenctors
• WebConsole: Add output buffer configuration UI (#1591)

Fixed

• WebConsole: Fix crash when displaying anonymous output columns (#1757)

v0.17.0

Added

• Added backpressure mode to the /egress endpoint, which applies backpressure on the pipeline if the HTTP client cannot keep up
  with the output instead of dropping data. (#1780)
• /heap_profile endpoint that generates the heap profile of a pipeline in the pprof format. (#1767)
• Python SDK for Feldera (#1745).

v0.16.0

Added

• [SQL] Support for user-defined functions implemented in SQL (#1714)
• [Kafka] Allow specifying Kafka headers as part of output Kafka connector
  config (#1718)
• [SQL] Support for LEAD/LAG window aggregates (#1706)
• [SQL] Support for WATERMARK table column annotations (#1694)
• Added a new config flag tcp_metrics_exporter which allows to inspect
  the pipeline metrics for debugging purposes with the metrics-observer
  CLI. (#1710)
• Ability to selectively enable storage per-pipeline. (#1692)

Changed

• [Docs] Re-organize documentation sections (#1734)

v0.15.0

[0.15.0] - 2024-04-30

Added

• Initial support for Avro as an output format (#1673)

Changed

• Fixed a bug where sometimes the wrong profile for compilation is chosen by the manager. (#1696)

v0.14.0

Added

• [SQL] ARG_MIN, ARG_MAX aggregation functions
  (#1619)
• SQL: Support for ARRAYS_OVERLAP function (#1570)
• Connectors: Initial Delta Table output connector support
  (#1639)

Changed

• [SQL]: ARRAY_CONTAINS, ARRAY_POSITION, ARRAY_REMOVE functions
  for arguments of different types (#1636)

v0.13.0

Added

• Output buffering mechanism to consolidate multiple output
  updates over time.
  (#1558)
• pipeline-manager: allow programs to be compiled in unoptimized or
  optimized modes, which tradeoff compilation speed for runtime speed
  (#1524)
• WebConsole: Support NULL values on Data Inspection and Insertion page (#1392)

Changed

• pipeline-manager: connector transport configuration in the API is now
  strongly typed, and the transport names have the _input and _output
  suffix added (#1532)

Fixed

• WebConsole: State shows InQueue when program is empty (#1443)

v0.12.0

Changed

• dbsp: Major refactoring of the dbsp crate to improve compilation speed of SQL programs (#1516)

Added

• SQL: Support for GROUPING, GROUPING_ID, and GROUPING SETS (#1527)
• SQL: Support for aggregation functions: STDDEV,
  STDDEV_POP, STDDEV_SAMP (#1539)
• SQL: support for SORT_ARRAY function (#1490)
• SQL: support for ARRAY_REVERSE function (#1491)
• SQL: support for ARRAY_MAX, ARRAY_MIN functions (#1492)
• SQL: support for ARRAY_COMPACT function (#1493)
• SQL: support for ARRAY_PREPEND function (#1496)
• adapters: add an input connector for Amazon S3 (#1485)
• pipeline-manager: add option to pick a compilation profile (#1513)
• SQL: support for ARRAY_CONTAINS function (#1499)
• SQL: support for ARRAY_REPEAT function (#1497)
• SQL: support for ROUND(DOUBLE, digit), TRUNCATE(DOUBLE, digit) (#1512)
• SQL: support for ARRAY_REMOVE function (#1498)
• SQL: support for ARRAY_DISTINCT function (#1515)
• Added basic support to ingest and export data in the parquet format (#1510)
• WebConsole: Services page, Kafka service edit dialog (#1233)
• adapters: add an input connector for Amazon S3 (#1485)