Repository navigation
v0.56.0
BigQuery Data Transfer, as a new module, datatransfer (#271): BigQuery's scheduled queries and the other transfer configs, in every location, with their schedules, their runs and each run's messages; and one write a console confirms before sending: run a config now, its exact request built first and sent once. Experimental: new in this release. No breaking change: the module is new. The default user agents changed.
// Every location's scheduled queries: production lists one location at a time.
var locations = try client.listLocations(.{});
defer locations.deinit();
for (locations.value.locations) |location| {
var configs = try client.listTransferConfigs(location.id, .{ .data_sources = &.{"scheduled_query"} });
defer configs.deinit();
for (configs.value.configs) |config| std.debug.print("{s}: {t}\n", .{ config.display_name, config.state });
}
// Run one now, the request shown first.
const options: datatransfer.StartManualRunOptions = .{ .run_time = std.Io.Clock.real.now(io) };
var request = try client.startManualRunRequest(config_name, options);
defer request.deinit();
if (try confirm(request.value.method, request.value.url, request.value.body)) {
var started = try client.startManualRun(config_name, options);
defer started.deinit();
}| Call | What it does |
|---|---|
Client.listLocations(options) |
The locations configs can live in: us, europe, the regions and BigQuery Omni's |
Client.listTransferConfigs(location, options), getTransferConfig(name) |
A location's configs, by data source (scheduled_query alone); a get adds the owner |
Client.listRuns(config_name, options), getRun(name) |
A config's runs, newest run time first, by state, every attempt or the latest a run date |
Client.listMessages(run_name, options) |
A run's messages, newest first, by severity |
Client.startManualRun(config_name, options) |
Runs a config now, or for a time given |
Client.startManualRunRequest(config_name, options) |
The run now's exact request, after the same checks with the same errors |
Client.configName(arena, location, id) |
A config's name in this project |
- Names as production gives them: production names the project by number, and every call takes a config's or run's name as a list or get gave it.
- A
TransferConfigcarries oneSchedulefor both of the API's schedule shapes (time-based, manual, event-driven, default), its next run time, its latest run's state, disabled, its destination dataset and region, its owner, its params as JSON text per key, and a typedScheduledQueryview: query, destination table template, write disposition. - A run now is sent once. It has no request id, production started a second run for the same time sent twice, and Google's clients never retry it; only a refused connection or a failed name lookup is tried again.
- One OAuth scope,
Client.Options.scope:.bigqueryby default, asbigquery.Client's, so one token serves both modules. A.cloud_platform_read_onlyclient reads everything and refuses a run now before sending, as production refuses it.
What production showed
Measured on 2026-10-21 in a test project, with scheduled queries made for the measurement and deleted after:
- There is no "every location":
-is refused ("BigQuery Data Transfer Service does not yet support location: unknown"), so the client refuses it before sending, and a console walkslistLocations: 54 locations on one page, which refuses a page size. - A run now answered at once,
PENDING; the run readRUNNINGwithin 3 s, and a scheduled query of literals endedSUCCEEDED75 s after it started. A time 60 s ahead of production's clock was taken, though the docs say it must be past. - A disabled config's run now is refused: "Transfer is disabled. Please enable the transfer config before starting transfer runs."
- A failing run carries its error ("division by zero: 1 / 0; JobID: ..."), a succeeded one an empty error; a run's messages come newest first; runs list newest run time first, configs by id.
Testing
datatransfer.testing.FakeDataTransfer holds locations, configs, runs and messages a test puts in, and runs a config now as the test says it ends: pending until moved on, or succeeded or failed at once with production's lines. Every behaviour above follows production; what it does not model it answers 501, naming it. A slow fuzz property holds it to a model, in a new nightly job.
The gated production test (DATATRANSFER_TEST=1, about three minutes, nothing billed) makes scheduled queries, runs two now through the client with the request shown first, follows them to their end, and replays the same script on a fake seeded from production's answers: all 18 outcomes agree, word for word.
The guides: Data Transfer and Testing. examples/datatransfer_runs.zig --fake --run --yes runs the list, confirm, run and follow loop offline.
The changelog has each module's entry.
zig fetch --save git+https://github.com/kmoneil/zig-gcp#v0.56.0