syc stream format

syc stream format (.sylz)

What syc c writes and syc d reads: one file (or a pipe) as a sequence of compressed blocks, gzip/zstd style. Version 4, frozen at tag v0.4.0: a later change is a new version number. Little-endian throughout. Reference reader: tools/syc_ref.py; test vectors in syc-vectors.zip (v0.4.0 release).

stream := "SYZ0"  version:u8 (=4)  codec:u8  level:u8  block*  end
block  := rawLen:u32  packed:u32  crc32:u32  payload
end    := rawLen = 0                      (4 zero bytes)

• codec: 1 = sylz (the only one so far; a reader refuses another). level is the level the writer used, for information only.
• A block: 1 ≤ rawLen ≤ 2^29 (512 MiB); packed bit 31 set means the payload is the raw data (its length is then rawLen), clear means one block of the codec (sylz: format v2 or v3, see sylz format); the low 31 bits are the payload length; crc32 (IEEE, as zlib's crc32) of the raw data, checked after decoding. The decoded stream is the blocks' raw data in order.
• syc's blocks are 4 MiB by default (-b); the last one is shorter. An empty input is the header and end.
• Nothing follows end; a stream that ends before it is truncated.

Older versions, read by syc and not written: 3 (syc 0.2.14–0.3.4; sylz v2 or v3 blocks), 2 (0.2.5–0.2.13; v3 literal chunks of mode 1), 1 (0.2.2–; v2 only). They have no codec byte: the header is "SYZ0" version level, 6 bytes, and the codec is sylz.