Skip to content

v0.56.0

Choose a tag to compare

@kmoneil kmoneil released this 21 Oct 20:26
· 14 commits to main since this release
Immutable release. Only release title and notes can be modified.

BigQuery Data Transfer, as a new module, datatransfer (#271): BigQuery's scheduled queries and the other transfer configs, in every location, with their schedules, their runs and each run's messages; and one write a console confirms before sending: run a config now, its exact request built first and sent once. Experimental: new in this release. No breaking change: the module is new. The default user agents changed.

// Every location's scheduled queries: production lists one location at a time.
var locations = try client.listLocations(.{});
defer locations.deinit();
for (locations.value.locations) |location| {
    var configs = try client.listTransferConfigs(location.id, .{ .data_sources = &.{"scheduled_query"} });
    defer configs.deinit();
    for (configs.value.configs) |config| std.debug.print("{s}: {t}\n", .{ config.display_name, config.state });
}

// Run one now, the request shown first.
const options: datatransfer.StartManualRunOptions = .{ .run_time = std.Io.Clock.real.now(io) };
var request = try client.startManualRunRequest(config_name, options);
defer request.deinit();
if (try confirm(request.value.method, request.value.url, request.value.body)) {
    var started = try client.startManualRun(config_name, options);
    defer started.deinit();
}
Call What it does
Client.listLocations(options) The locations configs can live in: us, europe, the regions and BigQuery Omni's
Client.listTransferConfigs(location, options), getTransferConfig(name) A location's configs, by data source (scheduled_query alone); a get adds the owner
Client.listRuns(config_name, options), getRun(name) A config's runs, newest run time first, by state, every attempt or the latest a run date
Client.listMessages(run_name, options) A run's messages, newest first, by severity
Client.startManualRun(config_name, options) Runs a config now, or for a time given
Client.startManualRunRequest(config_name, options) The run now's exact request, after the same checks with the same errors
Client.configName(arena, location, id) A config's name in this project
  • Names as production gives them: production names the project by number, and every call takes a config's or run's name as a list or get gave it.
  • A TransferConfig carries one Schedule for both of the API's schedule shapes (time-based, manual, event-driven, default), its next run time, its latest run's state, disabled, its destination dataset and region, its owner, its params as JSON text per key, and a typed ScheduledQuery view: query, destination table template, write disposition.
  • A run now is sent once. It has no request id, production started a second run for the same time sent twice, and Google's clients never retry it; only a refused connection or a failed name lookup is tried again.
  • One OAuth scope, Client.Options.scope: .bigquery by default, as bigquery.Client's, so one token serves both modules. A .cloud_platform_read_only client reads everything and refuses a run now before sending, as production refuses it.

What production showed

Measured on 2026-10-21 in a test project, with scheduled queries made for the measurement and deleted after:

  • There is no "every location": - is refused ("BigQuery Data Transfer Service does not yet support location: unknown"), so the client refuses it before sending, and a console walks listLocations: 54 locations on one page, which refuses a page size.
  • A run now answered at once, PENDING; the run read RUNNING within 3 s, and a scheduled query of literals ended SUCCEEDED 75 s after it started. A time 60 s ahead of production's clock was taken, though the docs say it must be past.
  • A disabled config's run now is refused: "Transfer is disabled. Please enable the transfer config before starting transfer runs."
  • A failing run carries its error ("division by zero: 1 / 0; JobID: ..."), a succeeded one an empty error; a run's messages come newest first; runs list newest run time first, configs by id.

Testing

datatransfer.testing.FakeDataTransfer holds locations, configs, runs and messages a test puts in, and runs a config now as the test says it ends: pending until moved on, or succeeded or failed at once with production's lines. Every behaviour above follows production; what it does not model it answers 501, naming it. A slow fuzz property holds it to a model, in a new nightly job.

The gated production test (DATATRANSFER_TEST=1, about three minutes, nothing billed) makes scheduled queries, runs two now through the client with the request shown first, follows them to their end, and replays the same script on a fake seeded from production's answers: all 18 outcomes agree, word for word.

The guides: Data Transfer and Testing. examples/datatransfer_runs.zig --fake --run --yes runs the list, confirm, run and follow loop offline.

The changelog has each module's entry.

zig fetch --save git+https://github.com/kmoneil/zig-gcp#v0.56.0