syc archive format

syc archive format (.syc)

Format 2, frozen at tag v0.4.0. syc writes it from v0.4.0 on; any later change is a new format number, and format 2 stays as described here. Test vectors and a reference reader written from this page: syc-vectors.zip in the v0.4.0 release and tools/syc_ref.py. Format 1 (syc 0.3.1–0.3.4) is the same except for a 16-byte vhead without the codec; syc reads it, and appends to it in format 1.

Little-endian throughout; varint is unsigned LEB128 (7 bits a byte, low first, bit 7 = more follows, at most 10 bytes, value below 2^64).

An archive is append-only: syc a on an existing archive adds a version at the end and never rewrites what is before it. A version holds only what changed since the previous one: new and modified files, their new data, and deletions. The state at version N is versions 1..N applied in order. Data is deduplicated: files are cut into content-defined fragments, and a fragment whose hash is already in the archive (in any version, or earlier in the same one) is not stored again.

archive  := header  version*
header   := "SYCA"  fmt:u8 (=2)  cdcMinLog:u8  cdcAvgLog:u8  cdcMaxLog:u8        8 bytes
version  := vhead  block*  end  index  commit
vhead    := "SYCV"  vnum:u32  time:i64  codec:u8  reserved:u8[3]               20 bytes
block    := rawLen:u32  packed:u32  crc32:u32  payload                          as a .sylz stream's block
end      := rawLen = 0                                                          4 zero bytes
index    := "SYCI"  rawLen:u32  packed:u32  crc32:u32  payload                  the index, one block
commit   := "SYCE"  vstart:u64  istart:u64  crc32:u32                           24 bytes

• header: 6 ≤ cdcMinLog < cdcAvgLog < cdcMaxLog ≤ 24 (syc writes 14, 16, 19).
• vhead: vnum is the version's number, 1 for the first and one more for each; time is when it was written, in seconds since 1970 UTC (information only); codec says what the version's blocks and index are: 1 = sylz (the only one so far; a reader refuses another); reserved is zero.

Blocks

Exactly the block of a .sylz stream: packed bit 31 set = payload stored raw (then the payload length equals rawLen), otherwise one block of the version's codec (sylz: format v2, or v3 when written with -ec, see sylz format); the low 31 bits are the payload length; crc32 (IEEE, zlib's) of the raw data. 1 ≤ rawLen ≤ 2^29 (512 MiB). A block's raw data is the concatenation of the version's new fragments, in order, and its length is the sum of their sizes; a fragment never spans two blocks. syc closes a block when the next fragment would take it past the block size (-b, default 16 MiB).

Fragments (content-defined chunking)

Each file is cut on its own (a fragment never spans two files) with a gear rolling hash, FastCDC-style with normalization, parameters from the header (defaults: min 2^14 = 16 KiB, average 2^16 = 64 KiB, max 2^19 = 512 KiB):

gear[0..255]: splitmix64 from state 0x5359434143444331 ("SYCACDC1"):
    state += 0x9E3779B97F4A7C15; z = state
    z = (z xor (z shr 30)) * 0xBF58476D1CE4E5B9
    z = (z xor (z shr 27)) * 0x94D049BB133111EB
    gear[i] = z xor (z shr 31)

from the start s of a fragment, with n bytes left in the file:
    if n <= min: the fragment is the rest of the file
    h = 0; i = min
    while i < min(n, max):
        h = (h shl 1) + gear[byte[s + i]]          (mod 2^64)
        i += 1
        bits = avgLog + 2 if i < avg else avgLog - 2
        if h shr (64 - bits) = 0: cut here (the fragment is i bytes)
    no cut: the fragment is min(n, max) bytes

A reference implementation of this section, in Python: tools/syc_cdc_ref.py (prints each fragment's offset, length and hash).

A fragment's identity is its BLAKE2s-256 hash (RFC 7693, unkeyed, 32 bytes). Fragments are numbered from 1 across the whole archive in the order versions store them; fragment 0 does not exist.

The index

The index record is a block with a magic in front: stored or compressed with the version's codec, rawLen ≤ 2^29, its crc32 checked like a block's. Its raw data is:

indexdata := nblocks:varint  blockinfo[nblocks]
             nfrags:varint   fraginfo[nfrags]
             nentries:varint entry[nentries]
blockinfo := bytes:varint (the block's whole size, header included)  frags:varint
fraginfo  := size:varint  hash[32]
entry     := path  type:u8  [fields by type]
path      := len:varint  utf8[len]     relative, "/" separators, no "", ".", ".." components
type      := 0 deleted   (nothing follows)
             1 file      meta  size:varint  nrefs:varint  first:varint  delta:varint[nrefs-1]
             2 directory meta
             3 symlink   meta  target:len:varint utf8[len]
meta      := mtime:i64 (ns since 1970 UTC)  attrKind:u8  attrs:varint  [uid:varint gid:varint]
attrKind  := 0 none   1 Unix st_mode (permission bits, 12 bits)  + uid/gid follow
             2 Windows file attributes (readonly, hidden, system, archive; and
               for a symlink, directory: whether it points to one)

• The version's new fragments are numbered from (fragments in earlier versions) + 1, in the order of fraginfo; the blocks hold them in that order, blockinfo.frags of them each (nfrags = the sum). Block k starts at vstart + 20 + Σ bytes of the blocks before it, so a reader finds any fragment without scanning the data; after the last block come the 4 zero bytes of end, then the index at istart.
• Ranges: 12 ≤ blockinfo.bytes ≤ 12 + 2^29, 1 ≤ blockinfo.frags, 1 ≤ fraginfo.size ≤ 2^cdcMaxLog; a path is 1 to 65535 bytes, a symlink's target up to 65535; attrKind ≤ 2, attrs, uid, gid < 2^32; a file's references are fragments 1 .. the last one of this version (a version may refer to fragments of earlier versions, never of later ones). Nothing follows the last entry.
• attrs: with attrKind 1, the low 12 bits of st_mode (permissions, setuid, setgid, sticky); with attrKind 2, Windows attributes: 0x1 read-only, 0x2 hidden, 0x4 system, 0x20 archive, and 0x10 (directory) only on a symlink that points to a directory.
• A file's references: first is a fragment number; each delta is the next one as zigzag(next − (previous + 1)), so a run of consecutive fragments codes as zeros. size must equal the sum of the referenced fragments' sizes. An empty file has nrefs = 0 (then first is absent).
• Entries are sorted by path (bytewise); a path appears at most once per version. An entry replaces the path's previous state; deleted removes it (and in a later version the path may come back).
• Hard links are stored as separate files: dedup makes them cost a few bytes.

Commit and recovery

vstart is the offset of the version's vhead, istart that of its index; crc32 covers the 20 bytes before it. A version counts only once its commit is complete and valid: the first version's vstart is 8 and each next one starts right after the commit before it; the index record ends where the commit starts.

• Paths are UTF-8 on every system (on Windows syc converts from and to UTF-16).
• Opening a file (a, x, l, t, i): read the commit at the end; its vstart − 24 is the previous version's commit, down to vstart = 8. Every index is read through istart: no data is read to list or plan an extraction.
• If the end is not a valid commit (power cut, full disk during a): scan forward from offset 8, version by version (block headers carry their lengths), to the last valid commit. l/x/t warn and use the versions up to it; the next a truncates the archive there and appends.
• A non-seekable input (l -, i -, t -, x -): syc copies it to a temporary file and reads that. The index of each version comes after its data, and only the index says where each fragment is in a block, so a single forward pass could check the blocks' CRCs but neither the fragments' hashes nor which file a fragment belongs to.
• syc a - writes a new archive to stdout (one version, written front to back); appending needs a file.

What syc a puts in a version

Not part of the format either: an entry whose type, size, mtime, attributes, owner and link target match the last version is left out (its file is not read); a path that was under one of the PATHs given and is gone gets a deleted entry, unless it is excluded, could not be read this time, or --no-delete is given. With no change, no version is written. Before the commit, the version's bytes are flushed to disk; after it, the commit.

Extraction rules (syc x)

Not part of the format, but what a reader should do with it: never write through a symlink (an archive's symlinks are made after every file, and an existing directory on the way that is a symlink skips the entry); files go to a temporary name and are renamed when complete; directories get their times last, deepest first.

Integrity (what a reader checks)

Three levels, all checked by syc t and by syc x on what it extracts: each block's crc32 (and the sylz decoder's own bounds checks), the index's crc32 and structure (every number in range, sizes adding up), and each fragment's BLAKE2s-256 against the data it decodes to. Any failure exits with status 3.

Decisions (2026-10-24)

• Fragment hashes: 32 bytes in the archive, 16 in memory. The index stores the full BLAKE2s-256 (~0.05% of the data at 64 KiB fragments) and every extraction checks it; the dedup table that syc a keeps in RAM is keyed by the first 16 bytes (~24 bytes a fragment: 1 TB of unique data ≈ 384 MB, 100 GB ≈ 38 MB). An accidental 128-bit match is out of reach (~2^64 fragments).
• Average fragment 64 KiB (zpaq's): it finds duplicates inside large files that change little (VM images, databases, regenerated tars). The block size, not the fragment size, sets the compression context, so the ratio does not depend on it. The parameters live in the header, so a later option can choose others per archive.
• Not stored: on Windows, uid/gid, ACLs and alternate streams; anywhere, extended attributes and hard links as links (they are stored as separate files, and dedup makes the copies cost a few bytes).