Skip to content

Releases: spiceai/spiceai

v0.13.0-alpha

21 May 03:24
67d034f
Compare
Choose a tag to compare

Spice v0.13-alpha (May 20, 2024)

The v0.13.0-alpha release significantly improves federated query performance and efficiency with Query Push-Down. Query push-down allows SQL queries to be directly executed by underlying data sources, such as joining tables using the same data connector. Query push-down is supported for all SQL-based and Arrow Flight data connectors. Additionally, runtime metrics, including query duration, collected and accessed in the spice.runtime.metrics table. This release also includes a new FTP/SFTP data connector and improved CSV support for the S3 data connector.

Highlights

  • Federated Query Push-Down (#1394): All SQL and Arrow Flight data connectors support federated query push-down.

  • Runtime Metrics (#1361): Runtime metric collection can be enabled using the --metrics flag and accessed by the spice.runtime.metrics table.

  • FTP & SFTP data connector (#1355) (#1399): Added support for using FTP and SFTP as data sources.

  • Improved CSV support (#1411) (#1414): S3/FTP/SFTP data connectors support CSV files with expanded CSV options.

Contributors

What's Changed

Full Changelog: v0.12.2-alpha...v0.13.0-alpha

v0.12.2-alpha

14 May 00:49
78d749f
Compare
Choose a tag to compare

Spice v0.12.2-alpha (May 13, 2024)

The v0.12.2-alpha release introduces data streaming and key-pair authentication for the Snowflake data connector, enables general append mode data refreshes for time-series data, improves connectivity error messages, adds nested folders support for the S3 data connector, and exposes nodeSelector and affinity keys in the Helm chart for better Kubernetes management.

Highlights

  • Improved Connectivity Error Messages: Error messages provide clearer, actionable guidance for misconfigured settings or unreachable data connectors.

  • Snowflake Data Connector Improvements: Enables data streaming by default and adds support for key-pair authentication in addition to passwords.

  • API for Refresh SQL Updates: Update dataset Refresh SQL via API.

  • Append Data Refresh: Append mode data refreshes for time-series data are now supported for all data connectors. Specify a dataset time_column with refresh_mode: append to only fetch data more recent than the latest local data.

  • Docker Image Update: The spiceai/spiceai:latest Docker image now includes the ODBC data connector. For a smaller footprint, use spiceai/spiceai:latest-slim.

  • Helm Chart Improvements: nodeSelector and affinity keys are now supported in the Helm chart for improved Kubernetes deployment management.

Breaking Changes

  • API to trigger accelerated dataset refreshes has changed from POST /v1/datasets/:name/refresh to POST /v1/datasets/:name/acceleration/refresh to be consistent with the Spicepod.yaml structure.

Contributors

What's Changed

  • Fix list type support in spark connect by @y-f-u in #1341
  • Add nested folder support in S3 Parquet connector by @y-f-u in #1342
  • Improves S3 connector using DataFusion ListingTable table provider by @y-f-u in #1326
  • Update ROADMAP May 6, 2024 by @lukekim in #1315
  • List flightsql and snowflake as supported connectors in README.md by @sgrebnov in #1317
  • Helm chart for v0.12.1-alpha by @ewgenius in #1323
  • Read sqlite_file param and use it as path by @Sevenannn in #1309
  • Compile spiced with release feature in docker image by @ewgenius in #1324
  • Add support for Snowflake key-pair authentication by @sgrebnov in #1314
  • Wrap postgres errors in common DataConnectorError by @ewgenius in #1327
  • Fix TPCH tests runner by @sgrebnov in #1330
  • Spice CLI support for Snowflake key-pair auth by @sgrebnov in #1325
  • sql_provider_datafusion: Support TimestampMicrosecond, Date32, Date64 by @mach-kernel in #1329
  • Resolve dangling reference for SQLite by @Sevenannn in #1312
  • Select columns from Spark Dataframe according to projected_schema by @Sevenannn in #1336
  • Expose nodeselector and affinity keys in Helm chart by @mach-kernel in #1338
  • Use streaming for Snowflake queries by @sgrebnov in #1337
  • Publish ODBC images by @mach-kernel in #1271
  • Include Postgres acceleration engine to types support tests by @sgrebnov in #1343
  • Refactor dataconnector providers getters to return common DataConnectorResult and DataConnectorError by @ewgenius in #1339
  • s3 csv support to validate the listing table extensibility by @y-f-u in #1344
  • Move model code into separate, feature-flagged crate by @Jeadie in #1335
  • Initial setup for federated queries by @phillipleblanc in #1350
  • Refactor dbconnection errors, and catch invalid postgres table name case by @ewgenius in #1353
  • Rename default datafusion catalog to "spice", add internal "spice.runtime" schema by @ewgenius in #1359
  • Add API to set Refresh SQL for accelerated table by @sgrebnov in #1356
  • Set next version to v0.12.2 by @phillipleblanc in #1367
  • Upgrade to DataFusion 38 by @phillipleblanc in #1368
  • Incremental append based on time column by @y-f-u in #1360
  • Update README.md to include correct output when running show tables from quickstart by @digadeesh in #1371

Full Changelog: v0.12.1-alpha...v0.12.2-alpha

v0.12.1-alpha

06 May 23:25
e287869
Compare
Choose a tag to compare

Spice v0.12.1-alpha (May 6, 2024)

The v0.12.1-alpha release introduces a new Snowflake data connector, support for UUID and TimestampTZ types in the PostgreSQL connector, and improved error messages across all data connectors. The Clickhouse data connector enables data streaming by default. The public SQL interface now restricts DML and DDL queries. Additionally, accelerated tables now fully support NULL values, and issues with schema conversion in these tables have been resolved.

Highlights

  • Snowflake Data Connector: Initial support for Snowflake as a data source.

  • Clickhouse Data Streaming: Enables data streaming by default, eliminating in-memory result collection.

  • Read-only SQL Interface: Disables DML (INSERT/UPDATE/DELETE) and DDL (CREATE/ALTER TABLE) queries for improved data source security.

  • Error Message Improvements: Improved the error messages for commonly encountered issues with data connectors.

  • Accelerated Tables: Supports NULL values across all data types and fixes schema conversion errors for consistent type handling.

Contributors

What's Changed

Full Changelog: v0.12.0-alpha...v0.12.1-alpha

v0.12-alpha

30 Apr 00:27
eec359d
Compare
Choose a tag to compare

Spice v0.12-alpha (Apr 29, 2024)

The v0.12-alpha release introduces Clickhouse and Apache Spark data connectors, adds support for limiting refresh data periods for temporal datasets, and includes upgraded Spice Client SDKs compatible with Spice OSS.

Highlights

  • Clickhouse data connector: Use Clickhouse as a data source with the clickhouse: scheme.

  • Apache Spark Connect data connector: Use Apache Spark Connect connections as a data source using the spark: scheme.

  • Refresh data window: Limit accelerated dataset data refreshes to the specified window, as a duration from now configuration setting, for faster and more efficient refreshes.

  • ODBC data connector: Use ODBC connections as a data source using the odbc: scheme. The ODBC data connector is currently optional and not included in default builds. It can be conditionally compiled using the odbc cargo feature when building from source.

  • Spice Client SDK Support: The official Spice SDKs have been upgraded with support for Spice OSS.

Breaking Changes

  • Refresh interval: The refresh_interval acceleration setting and been changed to refresh_check_interval to make it clearer it is the check versus the data interval.

Contributors

New Contributors

  • Thanks to @mach-kernel who made their first contribution in #1204 by adding the ODBC data connector!

What's Changed

Full Changelog: v0.11.1-alpha...v0.12.0-alpha

0.11.1-alpha

22 Apr 23:40
1489ba7
Compare
Choose a tag to compare

Spice v0.11.1-alpha (Apr 22, 2024)

The v0.11.1-alpha release introduces retention policies for accelerated datasets, native Windows installation support, and integration of catalog and schema settings for the Databricks Spark connector. Several bugs have also been fixed for improved stability.

Highlights

  • Retention Policies for Accelerated Datasets: Automatic eviction of data from accelerated time-series datasets when a specified temporal column exceeds the retention period, optimizing resource utilization.

  • Windows Installation Support: Native Windows installation support, including upgrades.

  • Databricks Spark Connect Catalog and Schema Settings: Improved translation between DataFusion and Spark, providing better Spark Catalog support.

Contributors

New in this release

What's Changed

Full Changelog: v0.11.0-alpha...v0.11.1-alpha

Spice.ai v0.11.0-alpha

15 Apr 23:51
56523dc
Compare
Choose a tag to compare

The Spice v0.11.0-alpha release significantly improves the Databricks data connector with Databricks Connect (Spark Connect) support, adds the DuckDB data connector, and adds the AWS Secrets Manager secret store. In addition, enhanced control over accelerated dataset refreshes, improved SSL security for MySQL and PostgreSQL connections, and overall stability improvements have been added.

Highlights in v0.11.0-alpha

DuckDB data connector: Use DuckDB databases or connections as a data source.

AWS Secrets Manager Secret Store: Use AWS Secrets Managers as a secret store.

Custom Refresh SQL: Specify a custom SQL query for dataset refresh using refresh_sql.

Dataset Refresh API: Trigger a dataset refresh using the new CLI command spice refresh or via API.

Expanded SSL support for Postgres: SSL mode now supports disable, require, prefer, verify-ca, verify-full options with the default mode changed to require. Added pg_sslrootcert parameter for setting a custom root certificate and the pg_insecure parameter is no longer supported.

Databricks Connect: Choose between using Spark Connect or Delta Lake when using the Databricks data connector for improved performance.

Improved SSL support for Postgres: ssl mode now supports disable, require, prefer, verify-ca, verify-full options with default mode changed to require.
Added pg_sslrootcert parameter to allow setting custom root cert for postgres connector, pg_insecure parameter is no longer supported as redundant.

Internal architecture refactor: The internal architecture of spiced was refactored to simplify the creation data components and to improve alignment with DataFusion concepts.

New Contributors

@edmondop's first contribution github.com/spiceai/spiceai/pull/1110!

Contributors

New in this release

Full Changelog: v0.10.2-alpha...v0.11.0-alpha

Spice.ai v0.10.2-alpha

09 Apr 00:00
ccf8615
Compare
Choose a tag to compare

The v0.10.2-alpha release adds the MySQL data connector and makes external data connections more robust on initialization.

Highlights in v0.10.2-alpha

  • MySQL data connector: Connect to any MySQL server, including SSL support.

  • Data connections verified at initialization: Verify endpoints and authorization for external data connections (e.g. databricks, spice.ai) at initialization.

New Contributors

Contributors

New in this release

Full Changelog: v0.10.1-alpha...v0.10.2-alpha

Spice.ai v0.10.1-alpha

01 Apr 23:37
b900221
Compare
Choose a tag to compare

The v0.10.1-alpha release focuses on stability, bug fixes, and usability by improving error messages when using SQLite data accelerators, improving the PostgreSQL support, and adding a basic Helm chart.

Highlights in v0.10.1-alpha

Improved PostgreSQL support for Data Connectors TLS is now supported with PostgreSQL Data Connectors and there is improved VARCHAR and BPCHAR conversions through Spice.

Improved Error messages Simplified error messages from Spice when propagating errors from Data Connectors and Accelerator Engines.

Spice Pods Command The spice pods command can give you quick statistics about models, dependencies, and datasets that are loaded by the Spice runtime.

Contributors

New in this release

  • Adds Basic Helm Chart for spiceai (#1002)
  • Adds Support for spice login in environments with no browser. (#994)
  • Adds TLS support in Postgres connector. (#970)
  • Fixes Improve Postgres VARCHAR and BPCHAR conversion. (#993)
  • Fixes spice pods Returns incorrect counts. (#998)
  • Fixes Return friendly error messages for unsupported types in sqlite. (#982)
  • Fixes Pass Tonic errors when receiving errors from dependencies. (#995)

Spice.ai v0.10-alpha

28 Mar 00:00
bf97452
Compare
Choose a tag to compare

Announcing the release of Spice.ai v0.10-alpha! 🎉

The Spice.ai v0.10-alpha release focused on additions and updates to improve stability, usability, and the overall Spice developer experience.

Highlights in v0.10-alpha

Public Bucket Support for S3 Data Connector: The S3 Data Connector now supports public buckets in addition to buckets requiring an access id and key.

JDBC-Client Connectivity: Improved connectivity for JDBC clients, like Tableau.

User Experience Improvements:

  • Friendlier error messages across the board to make debugging and development better.
  • Added a spice login postgres command, streamlining the process for connecting to PostgreSQL databases.
  • Added PostgreSQL connection verification and connection string support, enhancing usability for PostgreSQL users.

Grafana Dashboard: Improving the ability to monitor Spice deployments, a standard Grafana dashboard is now available.

Contributors

New in this release

  • Fixes Gracefully handle Arrow Flight DoExchange connection resets
  • Adds Grafana Dashboard
  • Adds Flight SQL CommandGetTableTypes Command support (improves JDBC-client connectivity)
  • Adds Friendlier error messages
  • Adds spice login postgres command
  • Adds PostgreSQL connection verification
  • Adds PostgreSQL connection string support
  • Adds Linux aarch64 build
  • Updates Improves spice status with dataset metrics
  • Updates CLI REPL improved show tables output
  • Updates CLI REPL limit output to 500 rows
  • Updates Improved README.md with architecture diagram updates
  • Updates Improved CI run time.
  • Updates Use macOS hosted Actions runner

Spice.ai v0.9.1-alpha

25 Mar 23:55
4008d30
Compare
Choose a tag to compare

The v0.9.1 release focused on stability, bug fixes, and usability by adding spice CLI commands for listing Spicepods (spice pods), Models (spice models), Datasets (spice datasets), and improved status (spice status) details. In addition, the Arrow Flight SQL (flightsql) data connector and SQLite (sqlite) data store were added.

Highlights in v0.9.1-alpha

FlightSQL data connector: Arrow Flight SQL can now be used as a connector for federated SQL query.

SQLite data backend: SQLite can now be used as a data store for acceleration.

Contributors

New in this release

  • Adds FlightSQL data connector (flightsql).
  • Adds SQLite data store, supports both in-memory and file based (sqlite).
  • Adds support for date, varchar, bpchar, and primitive list types for the PostgreSQL data connector and data store.
  • Adds spice pods, spice status, spice datasets, and spice models CLI commands.
  • Adds GET /v1/spicepods API for listing loaded Spicepods.
  • Adds spiced Docker CI build and release.
  • Adds E2E tests for release installation and local acceleration.
  • Adds E2E tests and instructions to run basic TPC-H benchmark tests.
  • Adds linux/arm64 binary build.
  • Fixes spice sql REPL panics when query result is too large. (#875)
  • Fixes --access-secret in spice s3 login. (#894)
  • Fixes version check upgrade logic.