Skip to content

v0.1.0 — YSON codec and MapReduce job runtime

Choose a tag to compare

@sshaplygin sshaplygin released this 04 Aug 00:49
· 128 commits to main since this release

First release. Write YTsaurus MapReduce workers in Rust instead of C++.

[dependencies]
ytsaurus-job = "0.1"
Crate Version
ytsaurus-job 0.1.0 job runtime — docs
ytsaurus-yson 0.1.0 YSON codec — docs

A YTsaurus job is an ordinary executable: it reads rows from fd 0 and writes
output tables to fds 1, 4, 7… in binary YSON. There is no official Rust SDK, so
this provides the stack — and the guide takes you from
an empty file to a running operation.

What it does

  • Streams. The reader holds one buffer no matter how large the input is —
    2 GB flows through at 1.9 MiB peak RSS.
  • Zero-copy rows. Decode into &str / &[u8] straight out of the read
    buffer; borrowed decoding measures ~15 % faster than owned.
  • Byte-exact pass-through. Row::raw() returns the original bytes, so an
    identity job reproduces its input exactly.
  • Control records. table_index, row_index and range_index are applied
    and surfaced per row; key_switch becomes per-key iterators for reduce.
  • Multi-table output. One descriptor per table (3k + 1), or a single
    stream with <table_index=N># switch records.
  • Static workers. scripts/build-worker.sh produces fully static
    x86_64-unknown-linux-musl binaries, and works on macOS without a
    cross-toolchain.

Downloads

The attached binaries are the example workers from this release, built exactly
as scripts/build-worker.sh builds them — fully static
x86_64-unknown-linux-musl, stripped, ready to hand to yt map --local-file.
Verify with shasum -a 256 -c SHA256SUMS.

  • cat-x86_64-unknown-linux-musl — identity map; copies input to output unchanged
  • wordcount-x86_64-unknown-linux-muslwordcount map / wordcount reduce

They are examples, not the library. To use the library, depend on the crates above.

Verified against a real cluster

Not just against a reading of the specification — a local YTsaurus in Docker ran:

  • the identity map, output table byte-identical to the input (309 688 bytes);
  • table switching across two input and two output tables;
  • a wordcount map-reduce matching a hand-computed result.

The offline end-to-end fixtures are captured from that cluster, so CI keeps a
meaningful signal without Docker.

158 tests. Fuzzing ran 6.5 M iterations across both YSON formats without a crash.

Upstream

ytsaurus-yson is a fork of ss123she/yson-rs
@ ba2044c, taken under Apache-2.0 (upstream offers MIT OR Apache-2.0). Nearly
all of the codec is the original author's work.

Vendoring it surfaced three defects, all fixed here and listed in the
changelog:

  1. An infinite loop on a stray / in text input. Input as short as /a
    never returned. It allocates nothing, so no memory watchdog catches it.
  2. Non-UTF-8 map keys were rejected, though the DOM stores keys as Vec<u8>.
  3. Non-UTF-8 attribute names were silently replaced with "".

The last two matter because YTsaurus strings and attribute names are arbitrary
byte strings, not text.

Known limitations

Documented in the codec README. The two that
bite most: maps round-trip as values rather than bytes (BTreeMap sorts keys),
and decoding into String fails on non-UTF-8 columns — use serde_bytes.

The public API has not had an independent review yet; expect it to move before
1.0.