v0.1.0 — YSON codec and MapReduce job runtime
First release. Write YTsaurus MapReduce workers in Rust instead of C++.
[dependencies]
ytsaurus-job = "0.1"| Crate | Version | |
|---|---|---|
ytsaurus-job |
0.1.0 | job runtime — docs |
ytsaurus-yson |
0.1.0 | YSON codec — docs |
A YTsaurus job is an ordinary executable: it reads rows from fd 0 and writes
output tables to fds 1, 4, 7… in binary YSON. There is no official Rust SDK, so
this provides the stack — and the guide takes you from
an empty file to a running operation.
What it does
- Streams. The reader holds one buffer no matter how large the input is —
2 GB flows through at 1.9 MiB peak RSS. - Zero-copy rows. Decode into
&str/&[u8]straight out of the read
buffer; borrowed decoding measures ~15 % faster than owned. - Byte-exact pass-through.
Row::raw()returns the original bytes, so an
identity job reproduces its input exactly. - Control records.
table_index,row_indexandrange_indexare applied
and surfaced per row;key_switchbecomes per-key iterators for reduce. - Multi-table output. One descriptor per table (
3k + 1), or a single
stream with<table_index=N>#switch records. - Static workers.
scripts/build-worker.shproduces fully static
x86_64-unknown-linux-muslbinaries, and works on macOS without a
cross-toolchain.
Downloads
The attached binaries are the example workers from this release, built exactly
as scripts/build-worker.sh builds them — fully static
x86_64-unknown-linux-musl, stripped, ready to hand to yt map --local-file.
Verify with shasum -a 256 -c SHA256SUMS.
cat-x86_64-unknown-linux-musl— identity map; copies input to output unchangedwordcount-x86_64-unknown-linux-musl—wordcount map/wordcount reduce
They are examples, not the library. To use the library, depend on the crates above.
Verified against a real cluster
Not just against a reading of the specification — a local YTsaurus in Docker ran:
- the identity map, output table byte-identical to the input (309 688 bytes);
- table switching across two input and two output tables;
- a
wordcountmap-reduce matching a hand-computed result.
The offline end-to-end fixtures are captured from that cluster, so CI keeps a
meaningful signal without Docker.
158 tests. Fuzzing ran 6.5 M iterations across both YSON formats without a crash.
Upstream
ytsaurus-yson is a fork of ss123she/yson-rs
@ ba2044c, taken under Apache-2.0 (upstream offers MIT OR Apache-2.0). Nearly
all of the codec is the original author's work.
Vendoring it surfaced three defects, all fixed here and listed in the
changelog:
- An infinite loop on a stray
/in text input. Input as short as/a
never returned. It allocates nothing, so no memory watchdog catches it. - Non-UTF-8 map keys were rejected, though the DOM stores keys as
Vec<u8>. - Non-UTF-8 attribute names were silently replaced with
"".
The last two matter because YTsaurus strings and attribute names are arbitrary
byte strings, not text.
Known limitations
Documented in the codec README. The two that
bite most: maps round-trip as values rather than bytes (BTreeMap sorts keys),
and decoding into String fails on non-UTF-8 columns — use serde_bytes.
The public API has not had an independent review yet; expect it to move before
1.0.