Library Guide

Library guide

zipir's module is named zipir. Every decompressor is a std.Io.Reader over any input reader, and every compressor a std.Io.Writer into any output writer. You own each workspace, set it up in place with init, and reuse it; zipir allocates nothing and starts no threads.

Add zipir as a dependency

Fetch the release's source package into your project. It holds only the build files, the source, the tests, and the license:

zig fetch --save=zipir https://github.com/eneskemalergin/zipir/releases/download/v0.3.0/zipir-0.3.0-source.tar.gz

Zig records the package and its hash in your build.zig.zon. Fetch the archive, never a live checkout or .: a fetch copies what it is given into Zig's package cache.

In your build.zig, with its existing b, target, optimize, and exe:

const zipir = b.dependency("zipir", .{ .target = target, .optimize = optimize });
exe.root_module.addImport("zipir", zipir.module("zipir"));

Your code can now @import("zipir"). Importing the module builds neither zipir's command nor its tests. To force every CPU kernel onto its portable path, pass .@"kernel-backend" = .portable beside .target; Platforms and acceleration explains when that helps.

To build against a local checkout without editing build.zig.zon, run zig build --fork=../zipir: wherever your dependency tree names zipir (matched by package name and fingerprint, whatever the version), Zig uses the checkout, and it fetches nothing for it.

Workspaces

A workspace is large, so allocate it once and reuse it:

• about 172 KB for any decompressor;
• 7.7 KB for a zlib or raw DEFLATE SliceDecoder, and 73 KB for a bgzf.BlockDecoder;
• 429 KB for a gzip, zlib, or raw DEFLATE compressor;
• 658 KB for a BGZF compressor, which stages two blocks of input to cut blocks as bgzip does.

init sets a workspace up in place and resets it, also after an error. The reader or writer lives inside it, so the workspace must not move while it is in use.

Read a stream

const zipir = @import("zipir");

// Decoded bytes, borrowed a chunk at a time.
const decoder = try allocator.create(zipir.gzip.Decompressor);
defer allocator.destroy(decoder);
decoder.init(&file_reader.interface, .{});
while (decoder.reader.peekGreedy(1)) |chunk| {
    use(chunk);
    decoder.reader.toss(chunk.len);
} else |err| switch (err) {
    error.EndOfStream => {},
    error.ReadFailed => return decoder.err.?, // CrcMismatch, Truncated, ..., or the input's ReadFailed
}

A failed read is always error.ReadFailed; decoder.err then holds the reason, such as CrcMismatch, Truncated, TrailingData, or the input reader's own ReadFailed. Bytes read before that error came before the failed check, so treat them as unverified until the end of the stream (when decoded bytes become readable). To decode a whole stream into a writer, use one pump: decoder.reader.streamRemaining(writer). The input reader's buffer can be any size, except that trailing_data = .leave needs 16 bytes (28 for BGZF) to leave the following bytes in it.

Options bound what untrusted input can cost:

• max_output_bytes fails the stream with OutputLimitExceeded once decoded output would pass it.
• trailing_data = .leave stops at the end of the stream and leaves any following bytes in the input reader; the default, .reject, fails with TrailingData.
• gzip's max_header_bytes (1 MiB by default) bounds each member's optional header fields.

Write a stream

// Plain bytes in, one gzip member out; the output does not depend on the write sizes.
const encoder = try allocator.create(zipir.gzip.Compressor);
defer allocator.destroy(encoder);
try encoder.init(&out.interface, .{ .preset = .even }); // .fast, .even, or .dense
try encoder.writer.writeAll(record);
_ = try encoder.finish();
try out.interface.flush();

finish writes the final block and the trailer and returns the number of plain bytes; the writer then fails until the next init. encoder.writer.flush() is a full flush: everything written so far can be decoded, and the next bytes start a new history. Neither flushes your output writer; flush it yourself. A contiguous request of up to 32 KiB (writableSlice, writeInt, print) always fits.

zlib and raw DEFLATE

Every format namespace (zipir.gzip, zipir.zlib, zipir.deflate, zipir.bgzf) has the same Decompressor, DecompressOptions, DecompressError, Compressor, CompressOptions, and CompressError, so zlib and raw DEFLATE read and write exactly as above. zipir.Decompressor(.zlib) and zipir.Compressor(.zlib) name the same types when the format is a comptime value.

For many independent streams already in memory, such as the binary arrays of an mzML file, SliceDecoder decodes one complete zlib or raw DEFLATE stream straight into a buffer you own, with an 11 KB workspace and up to about 16.3 KiB of stack:

// One whole zlib stream in memory, decoded into a buffer of the size you expect.
const slicer = try allocator.create(zipir.zlib.SliceDecoder);
defer allocator.destroy(slicer);
const n = try slicer.decode(compressed, values);
if (n != values.len) return error.ShortArray;

For zlib, decode checks the header and the Adler-32; for either format it rejects bytes after the stream, and OutputLimitExceeded means the stream needs more room than the buffer. Bytes of the buffer past the returned length are unspecified. Use the Decompressor reader for files and pipes, SliceDecoder for streams held whole in memory, and bgzf.BlockDecoder for single BGZF blocks.

BGZF, indexes, and seeking

The BGZF compressor can fill a .gzi index as it writes, and the decompressor seeks by virtual offset (seek) or by uncompressed offset through the index (seekUncompressed). Both seeks need the positional std.Io.File.Reader the decompressor was given.

var entries: [4096]zipir.bgzf.IndexEntry = undefined;
var index: zipir.bgzf.IndexBuilder = .init(&entries);
const writer = try allocator.create(zipir.bgzf.Compressor);
defer allocator.destroy(writer);
try writer.init(&out.interface, .{ .preset = .fast, .split = .lines, .index = &index });
try writer.writer.writeAll(reads);
_ = try writer.finish();
try out.interface.flush();
try zipir.bgzf.writeIndex(&gzi.interface, index.slice());

const reader = try allocator.create(zipir.bgzf.Decompressor);
defer allocator.destroy(reader);
reader.init(&file_reader.interface, .{});
try reader.seekUncompressed(&file_reader, index.slice(), 1_000_000);
const line = try reader.reader.takeDelimiterExclusive('\n');

The index storage is yours; when it is full, the compressor's writer fails and writer.err is IndexFull. .split = .lines gives exactly the block boundaries of bgzip 1.24 on text; .fill makes full blocks. zipir.bgzf.IndexReader reads a .gzi file back and rejects entries that do not increase strictly.

reader.tell() gives the virtual offset of the next byte a read returns, the offset a BAM index records for each record: call it before reading a record, and seek to it later to come back. Between blocks it names the start of the next block with data, as htslib's bgzf_tell does. It answers from the last 16 blocks with data the decompressor decoded, so PositionUnavailable means a single peek held more than 16 of them.

For work on single blocks, zipir.bgzf.scan walks block headers without decoding, BlockDecoder decodes one whole block, and BlockSplitter with BlockEncoder write blocks one at a time.

Blocks you compress yourself

BlockSplitter decides where blocks end, and each BlockEncoder.compressBlock call encodes one block from nothing but its input. Written in order and followed by the EOF marker, the blocks are exactly what bgzf.Compressor writes with the same preset and split:

var splitter: zipir.bgzf.BlockSplitter = .init(.fill);
var block: [zipir.bgzf.MAX_BLOCK]u8 = undefined;
var rest = plain;
while (splitter.next(rest, true)) |n| {
    const size = encoder.compressBlock(rest[0..n], &block, .even);
    try out.writeAll(block[0..size]);
    rest = rest[n..];
}
try out.writeAll(&zipir.bgzf.EOF_MARKER);

next(available, at_end) returns null until it has BlockSplitter.LOOKAHEAD bytes or the input has ended, so a caller that streams keeps that much input ahead. Because blocks are independent, a program that wants several cores can give each thread its own BlockEncoder, compress blocks there, and write them in order on one thread; zipir itself starts no threads.

Many outputs through one encoder

bgzf.Compressor holds its own BlockEncoder (428,584 bytes). A program writing many BGZF outputs at once, such as one file per group of records, can share one encoder: each output keeps only its pending input and its own BlockSplitter, and calls compressBlock on the shared encoder whenever a block is ready.

Copy unchanged blocks

To change the start of a BGZF file, such as a BAM header, without recompressing the rest: scan gives each block's offset and size, BlockDecoder decodes the block that changes, BlockEncoder encodes its new contents (at most BLOCK_INPUT bytes), and the remaining blocks are copied byte for byte.

var input: std.Io.Reader = .fixed(original);
var scanner = zipir.bgzf.scan(&input, .{});
const first = (try scanner.next()).?;
const n = try block_decoder.decodeBlock(original[@intCast(first.coffset)..][0..first.size], &plain);
// ... edit plain[0..n] ...
try out.writeAll(block[0..encoder.compressBlock(plain[0..n], &block, .even)]);
try out.writeAll(original[first.size..]);

decodeBlock writes plain only once the block's checks pass. When the caller discards its buffer on any error anyway, as a block cache does, decodeBlockInto decodes straight into it and saves the copy of every byte.

tar through any format

zipir.tar.Writer(Source) is a reader of the archive it builds, so a compressor pulls from it, and zipir.tar.Reader(Visitor) is a writer that calls the visitor per entry, so a decompressor pushes into it. A .tar.gz round trip:

// Source declares `next(*Source) !?zipir.tar.Entry` and `data(*Source) *std.Io.Reader`.
var tar_buffer: [4096]u8 = undefined;
var archive: zipir.tar.Writer(Source) = .init(&source, &tar_buffer, .{});
try gz.init(&out.interface, .{});
_ = try archive.reader.streamRemaining(&gz.writer);
_ = try gz.finish();
try out.interface.flush();

// Visitor declares `entry(*Visitor, Entry) !Action`, `data(*Visitor, []const u8) !void`, and `entryEnd(*Visitor) !void`.
var name: [zipir.tar.MAX_NAME + 1]u8 = undefined;
var link: [zipir.tar.MAX_NAME + 1]u8 = undefined;
var tar_reader: zipir.tar.Reader(Visitor) = .init(&visitor, .{ .name = &name, .link = &link });
gunzip.init(&in.interface, .{});
_ = try gunzip.reader.streamRemaining(&tar_reader.writer);
const summary = try tar_reader.finish(); // entries, and whether the end blocks were there

Source.data must hand out exactly Entry.size bytes; fewer is SourceTooShort. The visitor's entry returns .read to receive the entry's bytes through data, or .skip. Entry names borrow the Names buffers until the next entry. After a failed write or read, finish returns the cause.

The API reference lists every name, option, and error.