v4.0.0

The v4.0.0 release introduces a lazy DataFrame API and expression DSL, adds asynchronous queries, search, natural-language-to-SQL, runtime status, active-query management, and mTLS support, modernizes the runtime requirements, and bumps core dependencies to their latest major versions.

What's New

DataFrame API and Expression DSL

SpiceDataFrame is a new lazy SQL builder. Each operation returns a new DataFrame holding a SQL fragment; terminal operations (collect, to_pandas, to_polars, to_arrow, count, show) ship the compiled SQL to the runtime over the existing Flight transport — there is no client-side execution.

from spicepy import Client, col
from spicepy import functions as F

client = Client()
trips = client.table("taxi_trips")

result = (
    trips
    .filter(col("trip_distance") > 1.0)
    .group_by(col("payment_type"))
    .aggregate(
        F.count().alias("n"),
        F.sum(col("total_amount")).alias("revenue"),
        F.avg(col("trip_distance")).alias("avg_distance"),
    )
    .sort(col("revenue").desc())
    .limit(10)
    .to_pandas()
)

Entry points: client.table(name), client.from_sql(query), client.from_arrow(table), client.from_pandas(df), client.from_pydict(data). DataFrame operations include select, with_column(s), drop, rename, cast, filter/where, limit, head, sort/order_by, distinct, union, intersect, except_, join, cross_join, group_by(...).aggregate(...), schema, explain, to_sql, and the materializers (collect/to_arrow/to_pandas/to_polars/to_pylist/to_pydict/count/show).

Expr is the companion expression type — build one with col(name), lit(value), case(), or any function in spicepy.functions (aggregates, math, strings, date/time, null handling, window functions). Operators (+ - * / % == != < <= > >= & | ~) compose without evaluating. Anything not covered by the DSL is still reachable by writing SQL directly via client.sql(...)/client.from_sql(...).

Also new alongside the DataFrame API: catalog introspection (client.catalogs(), .schemas(), .tables(), .describe(), .get_schema(), .explain(), .show()) and streaming writers (client.write_parquet/csv/json(...)).

Search

search() finds documents similar to a piece of text, for datasets with an embedding column and a loaded embedding model:

result = client.search(
    'tokyo plane tickets',
    datasets=['app_messages'],
    limit=3,
    additional_columns=['timestamp'],
)

print(f'{len(result)} matches in {result.duration_ms}ms')
for match in result:
    print(match.score, match.dataset, match.matches, match.data)

Supplying keywords adds a lexical pass that the runtime blends with the vector scores into a hybrid ranking.

Natural Language to SQL (Nsql)

nsql() answers a question in natural language — the configured LLM generates SQL, the runtime runs it read-only, and both the rows and the generated query come back:

result = client.nsql('top 5 customers by revenue', datasets=['sales'])
print('generated SQL:', result.sql)
for row in result:
    print(row)

nsql_generate_sql() generates the SQL without running it — inspect or edit it, or run it through client.sql(...) for Arrow-typed results instead of nsql()'s decoded JSON rows.

Mutual TLS (mTLS)

Client accepts PEM certificate file paths for custom server verification and mutual TLS:

client = Client(
    flight_url="grpc+tls://my-spice-host:50051",
    tls_root_cert="./certs/ca.pem",
    tls_client_certificate="./certs/client.pem",
    tls_client_key="./certs/client.key",
)

tls_client_certificate and tls_client_key must be provided together. mTLS is an Enterprise feature of the Spice.ai runtime, which must be configured with client_auth_mode: request or required.

Runtime Health and Status

is_ready() reports whether the runtime is ready to serve queries. runtime_status() reports each runtime connection (http, flight, metrics, opentelemetry) individually, when you need to know which component isn't ready:

for component in client.runtime_status():
    print(f"{component.name} ({component.endpoint}): {component.status}")

A status added by a future runtime is preserved as a plain string rather than raising.

Active Query Management

list_active_queries() reports the synchronous queries running in the caller's scope, and cancel_active_query(query_id) stops one — the runtime doesn't hand a query's id back to the client that submitted it, so listing is the only way to find the id cancellation needs.

for query in client.list_active_queries():
    print(f"{query.query_id} [{query.protocol}] {query.sql_preview}")

queries = client.list_active_queries()
if queries:
    client.cancel_active_query(queries[0].query_id)

Asynchronous Queries

query() and query_with_params() submit a query as a background job over the runtime's /v1/queries REST API and return a QueryJob immediately, instead of streaming results over Flight. Use these for long-running queries you want to poll, wait on, or cancel independently of the connection that submitted them:

job = client.query("SELECT * FROM taxi_trips WHERE trip_distance > 50")
print(job.query_id, job.status())

result = job.results()  # waits for completion, then fetches and concatenates all pages
print(result.row_count)
for row in result:
    print(row)

QueryJob also exposes wait(poll_interval=0.5, timeout=None) and cancel(). client.list_queries(...) lists async jobs — a separate set from the synchronous queries list_active_queries() reports.

The previous synchronous, streaming behavior of query()/query_with_params() is now sql()/sql_with_params(). See Breaking Changes below.

Fixed

A new live-runtime integration suite (tests/test_integration_local.py, #190) caught four real SpiceDataFrame/Expr bugs that unit tests mocking the runtime couldn't:

• DataFrame.drop(), .rename(), .cast(), and .unnest() never worked against the real runtime — they compiled to * EXCLUDE (...) / * REPLACE (...) star modifiers the runtime's SQL parser rejects outright. They now resolve the frame's column list with a zero-row query and rewrite the projection explicitly.
• DataFrame.sort().limit() silently returned unordered rows — limit() wrapped the sorted frame in a subquery, and the planner is free to discard a subquery's ordering. LIMIT now shares the ORDER BY's query level.
• Expr.over(order_by=col(x)) (a bare column/expression, not a list) hung forever — Expr supports __getitem__ for array indexing, so iterating a bare Expr yields index expressions endlessly. Scalar partition_by/order_by are now boxed into a list.

If you use any of these operations, upgrade — no code changes needed, they just work correctly now.

Breaking Changes

query() and query_with_params() now submit SQL for asynchronous execution and return a QueryJob, instead of streaming results directly. The previous synchronous, streaming behavior is now sql()/sql_with_params():

# Before (v3.x)
reader = client.query("SELECT * FROM taxi_trips")

# After (v4.0.0)
reader = client.sql("SELECT * FROM taxi_trips")

Async queries additionally require the runtime to be running in distributed/scheduler mode; calling query()/query_with_params() against a single-node runtime now returns an error explaining that, rather than the query results.

Spice.ai Cloud endpoints are now region-specific. The legacy region-agnostic hostnames grpc+tls://flight.spiceai.io and https://data.spiceai.io are no longer valid — pass both flight_url and http_url explicitly, using your app's region (e.g. us-east-1, us-west-2, eu-west-1):

client = Client(
    api_key="API_KEY",
    flight_url="grpc+tls://us-east-1-prod-aws-flight.spiceai.io",
    http_url="https://us-east-1-prod-aws-data.spiceai.io",
)

config.DEFAULT_FLIGHT_URL/DEFAULT_HTTP_URL have been updated accordingly, but only for us-east-1 — if your app lives in a different region you must pass flight_url/http_url explicitly rather than relying on the default.

Client(http_url=...) now defaults to the local runtime (http://127.0.0.1:8090), matching flight_url's existing local default. Previously it defaulted to the (now invalid) cloud HTTP hostname — anyone relying on that default to reach cloud rather than local must now pass http_url explicitly.

Minimum Python version is now 3.11 (previously 3.9/3.10 were supported).

Core dependency floors moved to current majors — upgrade these in your environment before or together with spicepy: pyarrow>=25.0.1, pandas>=3.0.5, plus certifi and requests floor bumps.

Everything else is additive: existing refresh_dataset() code runs unchanged, and existing query()/query_with_params() code keeps compiling but now behaves differently — see the async-queries breaking change above.

What's Changed

• Add DataFrame API, Expr DSL, functions module, and SDK ergonomics by @lukekim in #151
• feat: add mTLS client certificate support by @phillipleblanc in #156
• feat: expand DataFrame/Expr analytics coverage by @lukekim in #167
• feat: array & struct support + DataFrame unnest / joins / describe by @lukekim in #168
• feat: notebook/Arrow interop, DataFrame ergonomics, and function breadth by @lukekim in #169
• perf: stream parameterized query results instead of buffering them by @lukekim in #172
• feat: add runtime_status and is_ready in #173
• feat: add search for the runtime's /v1/search endpoint in #177
• feat: add list_active_queries and cancel_active_query for running queries in #185
• feat: add nsql and nsql_generate_sql for the runtime's /v1/nsql endpoint in #187
• docs: add TLS/mTLS section to README, fix params extra install command by @sgrebnov in #188
• test: live-runtime integration suite; fix DataFrame SQL the runtime rejects in #190
• feat: async queries via /v1/queries; rename sync query()/query_with_params() to sql()/sql_with_params() in #191
• Prepare v4.0.0 release by @lukekim in #155
• fix(test): require a credential before running the cloud tests in #184
• ci: unbreak CI — inline the test dataset, and ignore PLR0917 from ruff 0.16 by @lukekim in #176
• chore(deps): upgrade all dependencies to latest versions by @lukekim in #136
• chore(deps): bump the python-dependencies group with 6 updates by @dependabot[bot] in #154
• chore(deps): bump idna from 3.11 to 3.15 in the uv group across 1 directory by @dependabot[bot] in #159
• chore(deps): bump the actions group across 1 directory with 3 updates by @dependabot[bot] in #171
• chore(deps): bump the python-dependencies group across 1 directory with 15 updates by @dependabot[bot] in #182

Full Changelog: v3.1.0...v4.0.0