Releases: surya-koritala/Glyd
Releases · surya-koritala/Glyd
Release list
v0.19.0
- A second GPU layout,
mma12(gpu/):
a weight's exponent a 4-bit code into the tensor's 15 commonest, a
step's exceptions in a list; four weights are three byte permutes, so
the decode keeps up with an H100's HBM3 where the tiered one is bound by
arithmetic. 12.04 bits a weight, 25% under bf16, bit for bit. On an H100
SXM, Qwen3-32B in 49.23 GB at 26.39 ms of GPU time a token against
bf16's 65.52 GB and 28.22 ms (the tiered layout: 44.45 GB, 40.58 ms);
Qwen2.5-7B 7.35 ms against 7.47; the products at 1-16 tokens 1.1-1.2x
faster than cuBLAS's (Qwen3-32B's MLP at one token: 75-79 us against
86-90); MMLU 78.1% (bf16 78.3%). On an RTX 4080 SUPER, where memory is
the limit, the tiered layout stays the faster at 1-32 sequences and
mma12leads at 64 (2,455.9 tokens/s against 2,244.7; bf16 2,160.0).
pack_mma12;mma_gemm,mma_gemm_bigandmma_unpacktake either;
e2e.py --format mma12; the kernels are written once over both layouts.
v0.18.0
- The KV cache compressed in GPU memory (gpu/kv.py), bit
for bit:GlydKVCache(config)for a Hugging Face model keeps each
layer's newest tokens as they are and packs every full page of 64 in
the mma layout's tiered code (keys by token, values transposed); with
fused=Truea step of one new token a sequence runsattn_decode,
attention straight from the packed pages (keys and values decoded in
registers, both products on the tensor cores, an online softmax, a
fixed order: the same result every run). Qwen2.5-7B-Instruct on an RTX
4080 SUPER: the cache 31% smaller (16K tokens: 651 MB for 947),
peak memory below the plain cache's, a step as fast (21.7 ms against
21.6 at 16K; 19.2 against 18.3 at 1K); decoded, the cache is the plain
one's bit for bit (the same tokens); throughattn_decode, 256 tokens
fed one at a time give perplexity 2.4778 against 2.4809 (16K). - Larger models on rented GPUs, bf16 and Glyd in the same runs, MMLU on
1,000 questions (gpu/README.md):
Qwen3-32B on one 48 GB RTX A6000 (44.45 GB; bf16 65.52 GB across
two) at 1.24-1.28x bf16's tokens/s for 1-8 sequences, MMLU 78.0%
(bf16 78.5%); Qwen2.5-72B on three (97.80 GB; bf16 145.41 GB
across four) at 1.40-1.42x, MMLU 81.8% (81.9%); on an H100 SXM
Qwen3-32B in 44.45 GB with MMLU 78.2% as bf16's, its products slower
than cuBLAS's (40.6 ms of GPU time a token against 28.2). - The README leads with the AI work:
nvidia-smifrom the runs
(e2e.py --smi, drawn byscripts/term_svg.py), the measured limits
(about 34% off bf16 weights or KV cache for any lossless code, 18% off
FP8, 7% off NVFP4). e2e.py --mmlu N,--kv LENGTHS,--smi PREFIX;pack_mmatakes
given tiers and a chunk size,mma_catappends packs;
gpu_lambda.shtakes models as[org/]name[:B[:G]](side by side on
B GPUs, then Glyd alone on G) and follows the run as it goes.
v0.17.0
- Model weights on the GPU (gpu/): the
mmalayout's
exponents in tiers of 2-bit digits — the tensor's 3 commonest
exponents, digit 3 going on to the next 3, then the next 3, then the
exponent itself: 10.80 bits a weight over Qwen2.5-7B's matrices,
32.5% under bf16 (was 11.25), every tensor bit for bit. A step's
escapes are decoded by the whole warp (each lane 16 of its tier-2
digits, placed by warp scans, through shared memory), so no lane
waits on another's. Qwen2.5-7B-Instruct on an RTX 4080 SUPER: 10.61
GB where bf16 takes 15.25 (was 11.05); 1.25-1.32x bf16's tokens/s
at 1 to 32 sequences (one: 55.7, bf16 43.4; 32: 1,518.9, bf16
1,153.7), 1.13x at 48, 1.04x at 64; prompts of 16 to 128 tokens
19-29 ms (bf16 24-29), 256 to 4096 within 5-10%. Perplexity as
before (Wikipedia, 64-token windows 17.0052 vs bf16's 17.0015,
512-token 7.5660 vs 7.5677). - The GPU extension builds for the GPU it runs on (
sm_90aon Hopper);
on Hoppere2e.pymultiplies prompts past 64 tokens by decoding then
cuBLAS;e2e.py --profile Nsplits GPU time by kernel.
scripts/gpu_lambda.shrunsgemm.pyande2e.pyon one Lambda
Cloud GPU instance launched for the run (a time cap, terminated and
checked on exit). - gpu/ is under the Business Source License 1.1 from this release (the
store's terms); the codec stays BSD-3-Clause OR GPL-2.0.
v0.16.0
- Model weights on the GPU, several tokens at once
(gpu/): themmalayout (pack_mma), the fast
format's 3-bit codes into the tensor's densest run of 7 exponents,
each step of 1024 weights one run in the order the tensor cores take
their operand.mma_gemm(1 to 64 tokens) decodes it in registers
straight intomma.syncfragments;mma_gemm_big(prompts) is a
tiled GEMM whose producer warps decode the weights into shared memory
while its consumer warps multiply. Qwen2.5-7B-Instruct on an RTX 4080
SUPER, in 11.05 GB where bf16 takes 15.25: 1.23-1.33x bf16's
tokens/s at 1 to 48 sequences at once (one: 55.2 tokens/s, bf16
43.3; 32: 1,528.6, bf16 1,149.0; the batched fast-format product it
replaces ran 0.72-0.90x); prompts of up to 128 tokens faster than
bf16 (128: 27 ms, bf16 29; was 60), 256 to 4096 within 5-9% (4096:
696 ms, bf16 645). The same result every run (sums in a fixed order);
perplexity as bf16's (Wikipedia, 64-token windows 17.0052 vs 17.0015,
512-token 7.5660 vs 7.5677). gpu/e2e.py:--format mma,--batch,--gpus N(layers spread
over GPUs by their bytes; bf16 by accelerate's device map),--ppl;
every kernel runs on its tensors' device and PyTorch's current stream.
gpu/setup_env.shbuilds the environment without root (nvcc pinned
to PyTorch's CUDA);scripts/gpu_aws.shruns Qwen2.5-32B and 72B on
4x L40S.
v0.15.0
- Model weights on the GPU (gpu/, Python and CUDA beside
the library): a bf16 model's weights held compressed in GPU memory and
decoded there bit for bit, the sign-and-mantissa byte as it is and the
exponent coded, in two formats: dense (a prefix code read by counting
leading zeros, as short as Huffman's; 10.9 bits a weight) and fast
(3-bit codes into the 7 most common exponents, an escape to the rest;
11.25). Generation multiplies straight from the packed weights, never
writing bf16 out. Qwen2.5-7B-Instruct on an RTX 4080 SUPER (16 GB):
55.1 tokens/s in 11.05 GB (fast) and 52.0 in 10.60 GB (dense),
against bf16's 43.2 in 15.25 GB; the fast format's 128 tokens as
bf16's. Prompts of up to 64 tokens multiply on the tensor cores from
the fast format (35-40 ms, bf16 24-27); longer prompts decode each
matrix and use PyTorch's matmul (2048 tokens: 337 ms, bf16 301). - PyTorch checkpoints (
torch.save): the zip's tensor storages go in as
byte planes, their element widths read from the checkpoint's pickle
(a reader for the opcodestorch.savewrites, no dependency); against
a base checkpoint (--base, the store) each storage the base holds
under the same name and size goes in as XOR that storage, where that
is the cheaper (weights move little between checkpoints, Adam's first
moment as much as it holds). Qwen2.5-0.5B fine-tuned with AdamW
(fp32 weights and both moments, 5.93 GB a checkpoint): 83.2% of its
size alone, 77.3% against the checkpoint 50 steps before; zstd -19
92.2% (its--patch-fromstops at 2 GB). The store finds a
checkpoint's predecessor by its storages' names and sizes.
v0.14.9
- The store: a family's first object sits at the depth cap only until
its family first needs it; then it is lifted, stored again against a
shallower base, in place of every new family starting shallow
(v0.14.8's rule). The terabyte gate, same corpus and instance
(report): 43.61 GB,
3.52× fewer bytes than zstd -3 (44.35 GB, 3.46× in v0.14.8), 27.2×
against raw; the English Wikipedia tables 8.67 → 7.94 GB, the kernels
as in v0.14.8 or smaller. Put 374 MB/s, read-back 486 MB/s (zstd -3's
346), restore 559 MB/s; all 1,192 objects byte-exact, both reads. - The store finds a model checkpoint's predecessor: a safetensors
object's fingerprints are its tensors' names, widths and sizes (it
shares no bytes with the checkpoint before it). Nine Pythia-410M
checkpoints (14.6 GB): 5.88 GB in the store, where zstd -19 stores
7.28 GB and zstd -3 8.63 GB; the last two as deltas of 501 MB, every
object verified (report). - Reads: the decoder no longer spins while a unit waits for the ones
ahead of it; the unit that completes the run writes it out. Decompress
CPU at the plain levels fell by up to 65% on 32 threads (logs--max
3.08 → 1.09 s), wall time unchanged. - The same bytes on every machine: an input is cut into sixteen units
whatever the core count (it followed the thread count before, so a
32-thread machine cut twice as finely as a 16-core one). GitHub
events at--ultra5.4% smaller on 32 threads; unchanged on 16. - Noise-level bytes of a small alphabet (a model weight's exponent or
mantissa plane) are coded as literals alone where the parse's short
matches would cost more:--maxon Pythia's exponent plane 171.7 →
140.1 MB (zstd -19: 143.8), on Qwen2.5's 208.3 → 169.8 MB (171.6).
Text, logs, SQL and kernel tars come out byte-identical in size. The
store's dense level (stripes on all cores) too: a stored checkpoint
752 → 697 MB. - Model weights: a safetensors file is opened, each tensor of 2-, 4- or
8-byte elements as byte planes (exponents together, mantissas
together), the header kept; closed byte for byte. Pythia-410M (fp32,
1,621 MB) at-9: 701 MB in 3.5 s, against zstd -19's 809 MB in 92 s
and zstd -3's 959 MB; Qwen2.5-0.5B (bf16, 988 MB): 663 MB against
750 and 769. Reads at 1.1 GB/s. - Checkpoints against checkpoints: in base mode, a safetensors file
against one holds each tensor the base has under the same name and
size as that tensor XOR the base's, in planes. Pythia-410M step 71000
against step 70000: 612 MB in 4.7 s, where zstd -19--patch-from
stores 805 MB (the raw delta finds nothing to match); step 143000
against 142000: 501 MB. Qwen2.5-0.5B-Instruct against its base model:
558 MB. - The command line asks for
--basefor a container opened against a
base (a gzip against a gzip, weights against weights): such a file
started with the container envelope, so decoding it without that
check failed. - The savings calculator re-measured at this code (LZ4, gzip, zstd -3
and -19, four Glyd levels, decompress CPU per row; the bucket row
from the v0.14.8 gate): report.
v0.14.8
- At a terabyte: 3.46× fewer bytes than zstd -3 (3.32× in
v0.14.7). The gate's 1,192 objects, 1.18 TB, put through the store
into S3 from one 16-vCPU instance next to the bucket
(report): 44.35 GB
stored against zstd -3's 153.5 GB, 26.7× against raw; put at 386
MB/s (372 in v0.14.7, zstd -3's own put 535); every object read back
by its own process at 464 MB/s (zstd -3's read-back 348), all 1,192
byte-exact; the whole bucket restored by one process at
571 MB/s (603 in v0.14.7), all 1,192 byte-exact. Kernel releases 5.15 at 286×,
6.1 at 406× (155× in v0.14.7), 6.6 at 377× against raw; hourly
GitHub events 14.1×. One cost in the release's own rule (below):
the English Wikipedia tables 19.1× against raw, where v0.14.7 kept
them at 20.9×. - Record mode writes 1.6–1.9× faster, the same bytes. On the
Ryzen box, output byte-identical to v0.14.7: the NASA access log on
one core 151 -> 265 MB/s, on all cores 749 -> 1,186 MB/s; a
Wikipedia table dump 126 -> 198 and 407 -> 790 MB/s. A dictionary
column's values are hashed once, and its recency list is kept in
place, searched eight entries at a time and not at all for a value
it cannot hold; a time value on the last exact value's date is not
printed back to be checked; field ranges are 32-bit (a log unit's
ranges had outweighed its text); a delimited line is split in one
pass; a SQL dump's rows no longer allocate, and its text is crossed
a word at a time to the next special byte. - Parquet files with snappy or zstd pages are opened
(src/parquet.rs,src/resnappy.rs,src/rezstd/): the footer's
column chunks and page headers are read (thrift's compact protocol,
no dependency), and every page is written back byte for byte by a
port of the compressor that wrote it, so the page's raw bytes are
compressed instead of its LZ tokens: google/snappy 1.2 level 1
(builds differ in their hash, a multiply or the CRC32C instruction,
and their table, 2^14 entries up to 1.1.10, 2^15 since 1.2.0), and
zstd 1.5.2 through 1.5.7 at levels 1 and 3 (the fast and
double-fast finders and their variants past the window's wrap,
Huffman literals with the previous block's table, FSE sequence
tables, the capacity rules, the checksum; 1.5.7's pre-block-splitter
and its two double-fast rules, 1.5.2's three fast-finder rules;
1.5.4 and 1.5.6 write what 1.5.5 does) as its library's one-shot
call writes them and as its command line does: the
--single-threadstream of 128 KB chunks, and the default of 2 MB
jobs from fresh contexts seeded with the 64 KB before each, so a
filezstdwrote opens too. Checked against zstd's own output on
the fixtures, on a sweep of 3,500 inputs under every version, level
and writer, and on 200 MB logs and dumps. The opener finds the
build that wrote a page and keeps a page no build made (zstd 1.4
and older, other levels, gzip pages). A container under a frame
opens in turn: the snappy taxi file inside azstd -3frame,
52.3 MB, comes to 34.8 MB. A whole zstd frame (a.zstobject) opens the
same way, a container under it opened in turn: the NASA access log
as zstd 1.5.5 wrote it at level 1, 22.3 MB, comes to 8.0 MB at
--max(its records modeled), 1.4 s to write and 0.6 s to read
back, byte-exact.
A page's plain values are then modeled so the LZ and entropy stages
see their structure: fixed-width values as byte planes, integers in
their unit (microseconds that are whole seconds divided down) and
as deltas, doubles that are decimals as scaled integers, byte
arrays as lengths then bytes; dictionary-index pages have their
runs decoded, written again by a port of Arrow's run-length encoder
and compared, and the indices laid out as planes of the bytes that
hold them. Each page takes the cheapest of its candidates or stays
as it is, judged by the max level's own output, and the level
blocks stay ahead. A NYC taxi month written by pyarrow 21 with
snappy, 61.7 MB: 174 of 174 pages reproduced;--max34.8 MB in
0.5 s on ten cores (1.8 s on one), read back in 94 ms (zstd -3 on
the file 52.3 MB, zstd -19 49.8; the same table's zstd-page file
50.3; record mode on the table as CSV 37.0),--ultra32.3 MB,
every decode byte-exact; the same table's zstd-page file, 50.3 MB
(pyarrow 14, zstd level 1):--max34.8 MB in 0.5 s, read back in
0.14 s. A month of for-hire trips, 519 MB with snappy, 1,291 pages,
more of them ids:--max376.5 MB in 3.4 s (zstd -3 on the file
473.2), read back in 1.0 s; with zstd pages, 472.8 MB: 376.5 MB in
2.7 s. Pages compressed with gzip, lz4 and brotli are left as they
are.
Files polars and DuckDB write open fully too: polars' snappy pages
are the Rustsnapcrate's (the multiply's older hash, shifted by
the table's size, which differs on blocks under 8 KB), and each
writer's run-length encoder for dictionary indices is ported beside
Arrow's (polars: repeats of more than eight, literal runs of up to
8192 values packed in blocks of 32; DuckDB: repeats of four or more,
bit-packed blocks of 256 written whole), the one that writes a
page's runs again named in its recipe. A bit-packed run's padding
(a block's earlier values) is taken as the writers leave it. The
same taxi month at--max: polars' snappy file, 86.9 MB, 51.4 ->
48.1 MB, its zstd file, 57.8 MB, 50.8 -> 48.1 MB; DuckDB's snappy
file, 61.1 MB, 36.6 -> 35.0 MB, its zstd file, 45.7 MB, 36.6 ->
35.0 MB; every decode byte-exact. - A new family starts shallow. An object that starts a family (a
new major release) takes an ancestor at depth 1 or the chain's root
as its base, so its versions come back to it at the depth cap; one
that had landed at the cap itself sent them to the chain's root.
At the gate, 6.1.1 sat at depth 4 on a 5.15 release and every fifth
6.1 release was a 26 MB delta of 5.15.1. On the Ryzen box, 5.15.1-100
then the 150 releases of 6.1 in the gate's order: 6.1 1,316 -> 502
MB, all 250 releases 1,714 -> 899 MB. At the gate the rule also
moved the monthly Wikipedia page tables, each month a family of its
own, onto bases two months back and the September one to alone:
0.73 GB more there, most of the kernels' 0.81 GB gain. The next
release lifts only a family's first object that sits at the cap,
when its family first needs it (measured on the box: the kernels
as here, the Wikipedia tables as in v0.14.7). - The store keeps a version's base among its own kind. An object
whose base holds under 98% of its fingerprints starts a family (a
new kernel major holds 0.85–0.94 of the old one's releases; point
releases hold 0.99–1.00 of the last, a 16-day Ubuntu image 0.97,
monthly Wikipedia tables 0.69–0.99), and past the depth cap a version's base is its
family's first object, not the chain's root. At the terabyte gate
every kernel release sat in one chain rooted at 5.15.1, and every
fifth 6.1 and 6.6 release was a delta of 5.15.1 at 42 MB against
2–4 MB within its series: 1.55 of the 6.6 series' 1.84 GB. Measured
on the Ryzen 9 box, 5.15.1 then the 150 releases of 6.6 in the
gate's order: 2,024 MB stored before, 643 MB now, one 42 MB
delta (6.6.1 itself against 5.15.1). The index line carries the
family (an eighth field; older lines read as before, their family
the chain's root). A star-shaped chain tree was tried and dropped:
6% fewer bytes on kernels, 30% more on monthly tables, where a base
two months back costs half again the neighbour's. - Whether a delta pays is judged on four 8 MB windows spread over
the object, each against its own base region, at the same 80% bar
the whole must meet (the head's 32 MB at a 50% bar before). Across
the English Wikipediapagedump the ratio of delta to alone runs
56–110% by window, 68% whole; the head's verdict had stored the
2026-09 dump alone, 1,776 MB where its delta against 2026-08 is
1,205 MB (v0.12.0 had that delta; v0.13.0's sample lost it). An
object holding a fingerprint several times now counts once among
its holders. The four windows run on threads of their own: an hour
of GitHub events put in 0.50 s on the box against 0.79 s with them
one after another. - The gate's read-back compares each object with the corpus file of
its name, the index line's last field (it read the seventh, which
the family field made the family's id, and counted every object
failed); each gate run syncs into a directory of its own (two runs at
once shared one, and the first to finish stopped the other's wait).
v0.14.7
- The store's put is 1.6–4.0× faster, its get 1.1–1.5×. On a
Ryzen 9 7950X3D (16 cores), best of three, every object read back
byte-exact: Linux 6.10.1 as a version of 6.10 (1.5 GB) put at
1,224 MB/s against 308 before, alone at 2,545 against 668, read
back at 1,590 against 1,051; an Ubuntu 24.04 cloud root filesystem
(1.1 GB, gzip inside) as a version 215 against 127, read 287
against 258; a Wikipedia table a month on (108 MB) as a version 383
against 233. What changed:put_filemaps the file and keeps the
mapping as the cached copy (put_vectakes the bytes over; no
second copy of the object in memory); the container path's deflate
emulation returns at once when nothing in the object opened; an
opened base is decoded once and cached; the base region a unit
searches is cut to what its fingerprints reach (97% of the hits
kept, never under the unit and 16 MB); each thread keeps one
region-and-unit buffer; a delta under a thirty-second of the object
is taken without also compressing the object alone;get_towrites
into the caller's buffer. The Ubuntu root's version is bounded by
the deflate emulator, which runs at zlib's own search speed per
thread. At a terabyte, one im4gn.4xlarge next to S3
(report): put at 372
MB/s (243 in the 2026-09-22 run), every object read back by its own
process at 461 MB/s (166 before; zstd -3's own read-back on that
instance 341), the bucket restored by one process at 603 MB/s, all
1,192 objects byte-exact both times; 46.3 GB stored against zstd
-3's 153.5 GB, 3.32× fewer bytes (3.10× then: the hourly events
store 9% smaller; one English Wikipedia table of fifteen went alone
that was a delta before, 0.6 GB, to be looked at). - CLI reads: the decoded batch lives on huge pages. The output
batch is an anonymous 2 MB-aligned mapping withMADV_HUGEPAGE,
the input is populated on a thread while the first units decode,
and-d -sstreams on one thread. One core againstzstd -d -T1:
Ryzen 9 7950X3D 1.00–1.30× its speed (0.65–0.81× before), Graviton3
1.15–1.32× (1.03×), Sapphire Rapids 0.79–0.97× (0.80×). All cores:
3.9–13.3 GB/s on the Ryzen, 3.6–10.0 GB/s on eight Graviton3 cores,
1.9–5.1 GB/s on eight Sapphire Rapids cores. - CRC-32C in three lanes: the block checksum runs three CRC
streams over 1 KB lanes and joins them by table, so the CRC
instruction's latency overlaps; it was a tenth of a one-core read on
Sapphire Rapids. Bytes unchanged: the checksum's value is the same. - A corrupted unit fails a stream's parallel decode instead of
hanging it: the units after a failed one waited for their turn
for ever; the first error now stops the rest
(tests/fuzz_safety.rs). - Library:
Store::put_vec,Store::put_file,Store::get_to. CI
keeps the corpus between runs and fetches enwik8 from a second host
when the first answers with a page.
v0.14.6: blocks cut where the bytes' statistics change
- Blocks are cut where the bytes' statistics change (
src/split.rs,
the idea of zstd 1.5.7's pre-splitter): before each block's parse,
sixteen bytes of every 256 in the 256 KB ahead are counted per 16 KB
segment, and the block ends at the segment boundary where the two
parts coded on their own statistics beat the whole by more than a few
blocks' overhead, each part charged for describing its table; never
under 32 KB, and the search rests after 32 windows without a cut. On
binaries with sections of different content it pays: Silesia mozilla
0.9% smaller (now 1.0% under zstd 1.5.5, 0.2% over zstd 1.5.7) for
5% more write time on that file; JSON events, the NASA log, a table
dump and enwik8 are unchanged in bytes and time. Every stream decodes
as before (blocks were always any length up to 256 KB).
v0.14.5: --max's parse is zstd -3's double-fast, no lazy step, one rule stricter
--max's parse is zstd -3's double-fast, no lazy step, with one
of zstd's rules made stricter. The lazy compare one byte on cost
4–8% of the write time for 0.1% (a table dump) to 4% (a log) fewer
bytes; it is gone. In its place: when only the short table matched,
the long table's entry one byte on (loaded already) is tried and its
match taken when it is at least two bytes longer; zstd's
unconditional form loses 0.7% on the table dump, this one gains on
every file. The literal Huffman lengths meet their 11-bit limit by
package-merge (optimal) instead of halving the counts. Bytes against
zstd -3: GitHub events −7.8%, a Wikipedia table dump −1.2%, the NASA
log −0.6%, enwik8 −0.5%, Silesia mozilla −0.1% (zstd 1.5.5; against
1.5.7, whose new block splitter gains 1.2% on mozilla, that file is
+1.1%). One core on Graviton3 againstzstd -3as installed: events
1.05× its speed, mozilla 1.05×, enwik8 1.01×, the log and the dump
0.89×; againstzstd -3 --single-thread1.03–1.26× on all five.
Eight cores againstzstd -3 -T8: 1.09×, 1.21×, 0.97×, 1.08×,
0.83×. On a Ryzen 9 7950X3D (one core, zstd 1.5.7) 1.07–1.26×
faster on all five. The CLI's one-core path writes on a second
thread, as zstd's does; the parse loop keeps fewer values live.
Every stream decodes as before.