v0.12.2-alpha

Spice v0.12.2-alpha (May 13, 2024)

The v0.12.2-alpha release introduces data streaming and key-pair authentication for the Snowflake data connector, enables general append mode data refreshes for time-series data, improves connectivity error messages, adds nested folders support for the S3 data connector, and exposes nodeSelector and affinity keys in the Helm chart for better Kubernetes management.

Highlights

• Improved Connectivity Error Messages: Error messages provide clearer, actionable guidance for misconfigured settings or unreachable data connectors.

• Snowflake Data Connector Improvements: Enables data streaming by default and adds support for key-pair authentication in addition to passwords.

• API for Refresh SQL Updates: Update dataset Refresh SQL via API.

• Append Data Refresh: Append mode data refreshes for time-series data are now supported for all data connectors. Specify a dataset time_column with refresh_mode: append to only fetch data more recent than the latest local data.

• Docker Image Update: The spiceai/spiceai:latest Docker image now includes the ODBC data connector. For a smaller footprint, use spiceai/spiceai:latest-slim.

• Helm Chart Improvements: nodeSelector and affinity keys are now supported in the Helm chart for improved Kubernetes deployment management.

Breaking Changes

• API to trigger accelerated dataset refreshes has changed from POST /v1/datasets/:name/refresh to POST /v1/datasets/:name/acceleration/refresh to be consistent with the Spicepod.yaml structure.

Contributors

• @mach-kernel
• @y-f-u
• @sgrebnov
• @ewgenius
• @Jeadie
• @Sevenannn
• @digadeesh
• @phillipleblanc
• @lukekim

What's Changed

• Fix list type support in spark connect by @y-f-u in #1341
• Add nested folder support in S3 Parquet connector by @y-f-u in #1342
• Improves S3 connector using DataFusion ListingTable table provider by @y-f-u in #1326
• Update ROADMAP May 6, 2024 by @lukekim in #1315
• List flightsql and snowflake as supported connectors in README.md by @sgrebnov in #1317
• Helm chart for v0.12.1-alpha by @ewgenius in #1323
• Read sqlite_file param and use it as path by @Sevenannn in #1309
• Compile spiced with release feature in docker image by @ewgenius in #1324
• Add support for Snowflake key-pair authentication by @sgrebnov in #1314
• Wrap postgres errors in common DataConnectorError by @ewgenius in #1327
• Fix TPCH tests runner by @sgrebnov in #1330
• Spice CLI support for Snowflake key-pair auth by @sgrebnov in #1325
• sql_provider_datafusion: Support TimestampMicrosecond, Date32, Date64 by @mach-kernel in #1329
• Resolve dangling reference for SQLite by @Sevenannn in #1312
• Select columns from Spark Dataframe according to projected_schema by @Sevenannn in #1336
• Expose nodeselector and affinity keys in Helm chart by @mach-kernel in #1338
• Use streaming for Snowflake queries by @sgrebnov in #1337
• Publish ODBC images by @mach-kernel in #1271
• Include Postgres acceleration engine to types support tests by @sgrebnov in #1343
• Refactor dataconnector providers getters to return common DataConnectorResult and DataConnectorError by @ewgenius in #1339
• s3 csv support to validate the listing table extensibility by @y-f-u in #1344
• Move model code into separate, feature-flagged crate by @Jeadie in #1335
• Initial setup for federated queries by @phillipleblanc in #1350
• Refactor dbconnection errors, and catch invalid postgres table name case by @ewgenius in #1353
• Rename default datafusion catalog to "spice", add internal "spice.runtime" schema by @ewgenius in #1359
• Add API to set Refresh SQL for accelerated table by @sgrebnov in #1356
• Set next version to v0.12.2 by @phillipleblanc in #1367
• Upgrade to DataFusion 38 by @phillipleblanc in #1368
• Incremental append based on time column by @y-f-u in #1360
• Update README.md to include correct output when running show tables from quickstart by @digadeesh in #1371

Full Changelog: v0.12.1-alpha...v0.12.2-alpha