Skip to content

v0.55.0

Choose a tag to compare

@kmoneil kmoneil released this 21 Oct 18:39
· 19 commits to main since this release
Immutable release. Only release title and notes can be modified.

BigQuery, as a new module, bigquery (#270): the datasets a project holds, their tables with their schemas, and the job history, from any project; and queries as a console's query editor runs them: dry-run for what they would read, run under a bytes limit, waited for, read a page at a time with every type decoded exactly, and cancelled, each request built first so it can be confirmed. Experimental: new in this release. No breaking change: the module is new, and core gains a hook and a public function. The default user agents changed.

// What it would read, for nothing; then run it under a limit and read it.
var dry = try client.dryRun(.{ .query = text });
defer dry.deinit();
std.debug.print("reads {d} bytes\n", .{dry.value.total_bytes_processed orelse 0});

var page = try client.query(.{ .query = text, .maximum_bytes_billed = 100_000_000 });
defer page.deinit();
while (!page.value.complete) {
    const next = try client.getQueryResults(page.value.job.?, .{});
    page.deinit();
    page = next;
}
for (page.value.rows) |row| std.debug.print("{f}: {f}\n", .{ row.cells[0].value, row.cells[1].value });
Call What it does
Client.listDatasets(options), getDataset(ref) A project's datasets, the client's own or another's (bigquery-public-data); hidden ones with all, a label filter
Client.listTables(ref, options), getTable(ref) A dataset's tables; a get carries the schema with nested records, row count, bytes, partitioning, clustering, and a view's or materialized view's query
Client.listJobs(options), getJob(ref) The job history, newest first, with each query's text; all users', by state and creation time
Client.dryRun(options) What a query would read and bill, its columns and statement type, without running it: free, never in the history
Client.query(options) Runs a query: its first page when it ends within timeout_ms, or its job to read further
Client.getQueryResults(job, options) A page of a job's results, or a running job's state once the call has been held timeout_ms; by token, by size, or from any row
Client.cancelJob(job) Asks a job to stop
Client.dryRunRequest, queryRequest, cancelJobRequest The exact requests, after the same checks with the same errors
  • Every query is GoogleSQL: the client sends useLegacySql false, BigQuery's default being legacy SQL still. Timestamps are asked as microseconds, as Google's clients ask them, and decode exactly.
  • Every query carries a request id, a UUID per call unless you give one (Client.newRequestId, so a preview and its run send the same bytes). A resend within 15 minutes answers the first job, so query is retried as the reads are.
  • Every type decodes exactly: bigquery.Value holds INT64 as i64, NUMERIC and BIGNUMERIC as exact decimal text, BYTES decoded, DATE, TIME, DATETIME, TIMESTAMP to the microsecond, INTERVAL in BigQuery's three parts, RANGE with unbounded ends, arrays and records. A Cell keeps production's wire text beside its value. Value.format writes a value as the console shows it, and bigquery.errorPosition reads the at [line:column] of a query error.
  • Errors by reason: error.InvalidQuery (syntax, semantic and runtime errors), BytesBilledLimitExceeded, ResultTooLarge, QuotaExceeded; rate limits are retried even when BigQuery sends them as a 403. This needed one hook in core: core.rpc.Engine.classify_refusal.
  • One OAuth scope, Client.Options.scope: .bigquery by default. BigQuery has no read-only scope; .cloud_platform_read_only serves every read, dry run and query, and the client refuses a cancel under it before sending, as production does.

What production showed

Measured on 2026-10-21 in a test project:

  • A query that reads a table bills at least 10 MiB; one of literals alone, or one the cache answers, bills nothing. A maximum_bytes_billed below the bill refuses the run at once, billing nothing: "Query exceeded limit for bytes billed: 1000. 10485760 or higher required."
  • A syntax error and a runtime error (1 / 0) are refused at once, a dry run and a run alike, with "at [line:column]" where there is one.
  • A query that outlives timeout_ms answers its job and nothing else; getQueryResults held each call about as long as asked.
  • A repeated request id answers the first job, unfinished and without rows, even when that job had already ended: read its results.
  • A cancel answers the job still running; it then ends done with the error stopped. A cancel of a done job is no refusal.
  • DDL and DML (CREATE, INSERT) answer no row count. Schemas use the legacy type names (INTEGER, FLOAT, BOOLEAN, RECORD), which FieldType reads alongside the new ones.

Testing

bigquery.testing.FakeBigQuery is BigQuery in memory, with no SQL engine: a test gives each query text its answer (schema and rows, the tables it reads, bytes, how many reads it takes to finish, the cache, or a refusal in BigQuery's words) and puts in datasets, tables and jobs. Dry runs, billing and the limit, request ids, polls, pages, cancels, job locations and refusals follow production; what it does not model it answers 501, naming it. A slow fuzz property holds it to a model, in a new nightly job.

The gated production test (BIGQUERY_TEST=1, about two minutes, inside the free tier) reads the public samples, makes a zigps_ dataset through the client's own DDL, runs, pages, polls, refuses and cancels queries, and replays the same script on a fake seeded from production's answers: all 39 outcomes agree, word for word.

The guides: BigQuery and Testing. examples/bigquery_query.zig --fake --yes runs the whole dry-run, confirm, run, wait and read loop offline.

The changelog has each module's entry.

zig fetch --save git+https://github.com/kmoneil/zig-gcp#v0.55.0