Skip to content

v0.2.0 — API fixes from the pilot, plus a launcher

Choose a tag to compare

View all tags
@sshaplygin sshaplygin released this 04 Aug 02:20
· 122 commits to main since this release
v0.2.0
603cc10

Everything in this release came from using 0.1.0 rather than reviewing it: a
production-shaped pilot (sessionize) and a
new launcher were written against the API, and every place it forced a
workaround was filed as an issue. All four are closed here.

New: ytsaurus-client

A thin HTTP API v4 client — run a job with no Python installation:

let client = Client::from_env()?;
client.upload_worker("target/…/my_job", "//tmp/my_job")?;
let id = client.start_map(&spec)?;
client.wait_for_operation(&id)?;

Verified against a local cluster with nothing Python on PATH: creates tables,
uploads the worker as executable, writes rows, runs a map, polls to completion,
reads both tables back and compares them byte for byte.

Parameters and specs are encoded with this project's own YSON codec rather than
JSON, which keeps the dependency list short and exercises ytsaurus-yson
against a real cluster on every request.

Two limits are documented rather than hidden: heavy commands are not routed via
/hosts, and ureq 3.3 exposes no trailers, so a failure the proxy reports
mid-stream cannot be seen. read_table compensates by rejecting a response that
is not a complete YSON list fragment.

ytsaurus-job

Reduce keys (#2). A reducer no longer re-derives the key from its first
row:

let mut groups = reader.groups_by(["user_id"]);
while let Some(mut group) = groups.next_group()? {
    let user = group.key().bytes("user_id").unwrap_or_default();
    // ...
}

YTsaurus does not transmit the key — key_switch carries no payload — so this
reads it from the group's first row: the same work, done once instead of in
every reducer. Accessors are byte-first, because reduce keys routinely are not
UTF-8.

Error classification (#1). JobError::kind() gives a stable,
allocation-free identifier for a reason column; is_row_local() separates
"quarantine this row" from "the stream is broken, stop". On the pilot's cluster
run the rejects table collapsed to 3x invalid_yson instead of three distinct
formatted messages.

Named output tables (#4):

let (mut writer, [events, rejects]) = JobWriter::named(["events", "rejects"])?;
writer.write(rejects, &row)?;

Two output tables of different meaning is exactly where transposing 0 and 1
gives a job that runs happily and fills each table with the other's rows.

Documentation

The guide now covers the output side (#3): an output row may borrow from
the input row, so a rejects table needs no to_vec(). Filed as an API gap, it
turned out to be a documentation gap — verified by compiling the borrowing form
against 0.1.0 before writing the issue up.

Breaking

JobError::UnknownTable gained a names field, so an out-of-range write
reports output table 9 does not exist; this job has 2 output table(s): events, rejects rather than a bare index. Add .. to any exhaustive pattern.

JobWriter::write and write_raw now take impl Into<TableId>. This is
source-compatible: write(0, &row) still resolves.

Verified

202 tests and 19 doctests, clippy clean under -D warnings. On a live cluster:
the pilot's 888 input rows give 877 kept / 11 quarantined and 143 sessions across
60 users, matching an independent Python recomputation of the whole expectation;
cat still reproduces its input byte for byte; wordcount still matches a
hand-computed result.

Assets 6
Loading