0.4.6

Fixes encrypted archives with more than one backup version. If hbkit failed on part of an encrypted archive with decompress failed, try again on this version.

Hyper Backup gives every backup version its own data key: Pool/vkey.db holds one sealed key per version, and each version encrypts the chunks it writes with its own. hbkit only ever loaded the first one, so every chunk written by version 2 or later failed to decompress. Nothing came out wrong, but those files could not be restored at all.

Diagnosed by @jby in #4, who scanned all 399,676 chunk records of a 7-version archive, showed each decodes under exactly one key, and sent the patch this release is built on. With it, 6,839 of their 6,840 files verify; the last is the known Pool/file_pool whole-file dedup gap.

hbkit now unseals every version's key and, per bucket, tries the key that last worked before the others. A key is accepted only when the chunk's MD5 matches, even with verification off, because a wrong key can occasionally still decompress to the right length.

Also from #4:

• doctor passed 11/11 on the affected archive because its random sample only hit first-version files. It now takes one file from each slice of the archive, so the newest files are always checked, and reports how many data keys the sample used.
• get and verify now reject options they don't take. verify '*' --version 3 used to verify the newest version without a word. hbkit reads the newest version, and doctor no longer suggests otherwise.

Install: brew install YordiLorenzo/tap/hbkit · pip install hbkit

v0.4.5

Fixes a silent recovery failure. If you have an archive hbkit could not fully restore, try again on this version.

Shard files inside a chunk_index or file_chunk family can have gaps in their numbering: a real archive in #1 was missing 7.idx from one family and 27.idx from another. hbkit compacted the shards it found into a contiguous list and derived each one's position from its place in that list, so every shard after a gap was addressed 8 MiB too early. The bytes at the wrong offset still parse as plausible records, so any file whose chunks lived past a gap failed to rebuild.

Diagnosed and fixed by @arjungopisetty in #3, who also confirmed the repaired restore: 74,396,840 bytes rebuilt exactly across 9,072 verified chunks, straight from Backblaze B2.

A shard's logical position now comes from its filename rather than its index in the list, and reading inside a gap returns nothing instead of the neighbouring shard's bytes, so a hole fails loudly rather than producing wrong data.

Also fixed in #3: hbk <archive> doctor -s N silently ignored the sample count and always rebuilt 10 files per share. -s, -n and --sample all work now.

Install: brew install YordiLorenzo/tap/hbkit · pip install hbkit

v0.4.4

A small fix release. Nothing here changes extraction: if 0.4.3 restores your archive correctly, so does this.

doctor reported file_chunk shards as empty on every archive. The shard families are opened lazily, only when a file actually references one, and doctor describes an archive without extracting from it, so nothing had opened them by the time it printed. A healthy archive looked malformed. Found via #1, where it sent the investigation off in the wrong direction before a probe found the shard sitting there all along.

Also landed, though not shipped inside the package: contrib/hbk_probe.py, a read-only diagnostic that walks one file's chunk list and reports where and why a rebuild fails. Written for #1 and useful for any future report of the same shape.

Install: brew install YordiLorenzo/tap/hbkit · pip install hbkit

v0.4.3

Two bugs that only exist in packaged installs, both found while packaging for Nix.

Extraction failed entirely under wrapper-based installs. Workers are spawned as sys.executable -m hbkit.runner, and sys.executable is the unwrapped interpreter. Wherever hbkit reaches sys.path through a wrapper script instead of site-packages — a Nix build, a PYTHONPATH-based distro package — the child could not import hbkit and every worker died before extracting a byte. The parent's sys.path is now passed down.

hbk <archive> doctor ignored HBK_PASSWORD. It forwarded only -p, so a perfectly readable encrypted archive was reported as unrecoverable. Both paths now share one password resolver.

Neither was reachable from pip install hbkit, which is why CI never saw them. Both have regression tests.

Install: brew install YordiLorenzo/tap/hbkit · pip install hbkit

v0.4.2

Adds hbk --version, plus packaging recipes so hbkit can be installed outside pip.

• Homebrew (macOS/Linux): brew install YordiLorenzo/tap/hbkit
• AUR (Arch): packaging/aur/PKGBUILD
• nixpkgs: packaging/nix/package.nix

hbkit 0.4.1 — docs sync

Documentation catch-up plus one real fix.

• README command reference now includes mount, warm, unmount, remotes and --strict
• TUI key list documents ctrl+q (plain q types into whichever field has focus)
• Cache path unified on ~/.cache/hbkit. The code previously used ~/.cache/hbk-recovery
  while the docs claimed otherwise. The old directory is still honoured if it exists, so
  upgrading will not orphan an index that took minutes to build over a network mount.
• Scope note now reflects that both an unencrypted 3 TB archive and an encrypted one have
  been verified byte-exact.

hbkit 0.4.0 — durable resume, live progress

Resume is now verifiable

Previously resume was inferred from the destination alone: a file of the right size was
assumed done. That silently accepts a file that is the right length and the wrong
bytes — which contradicts the whole point of a tool that checksums every chunk.

Each restored file now records its size and MD5 in <dest>/.hbkit-restore.jsonl, hashed
as it is written (no extra I/O). Plain resume still skips on size — fast, and correct for
an interrupted run of hbkit's own. --strict re-checks the recorded hashes and repairs
mismatches:

hbk /path/to/backup get "/Photos/*" ~/restore --strict

Leftover .part files from an interrupted run are swept automatically.

Progress that actually moves

Progress was per-file, so a single large file showed a frozen screen — a 674 MB restore
over R2 sat at zero for 73 seconds. Workers now report bytes as they go.

• rates below 1 MB/s used to render as 0 MB/s; now adaptive KB/s
• the TUI shows a rolling recent-files panel (bounded ring buffer — same cost for 9 files
  or 500,000)
• indexing has a real progress bar with ETA instead of a spinner
• ctrl+q quits from anywhere. Plain q was being typed into the focused path field
  instead of quitting.

Indexing over a network mount: 48 seconds

Share databases are staged to local disk before SQLite opens them. SQLite issues thousands
of small random page reads, and on a mounted bucket each becomes its own range request.
Indexing 1.1M files over R2 now takes 48s instead of many minutes.

Benchmarks

Profiled against a 3 TB archive in R2 (150 MB, 18,104 chunks):

phase	share of wall time
reading chunk data	53%
reading bucket indexes	20%
chunk_index lookups	15%
LZ4 + MD5 + CRC32	0.7%

Verification is free over a network — never disable it to go faster. Throughput is
bound by round-trips, so concurrency is the only knob: 4.0 MB/s at -j 1, 8.5 at -j 8,
9.5 at -j 16, then flat. Use -j 16 on a network mount.

Measured and rejected, so you needn't try them: larger read buffers (no effect,
latency-bound) and targeted 32-byte bucket-index reads instead of caching the whole 200 KB
file (18% slower — a file's chunks cluster in one bucket). Read amplification is 0.82×.

hbkit 0.3.0 — object storage that actually works

Adds hbk mount / unmount / warm / remotes, and makes archives in S3/R2 genuinely usable.

rclone config                                    # your credentials stay with rclone
hbk remotes
hbk mount r2:mybucket ~/mnt/backup --for browse
hbk-tui ~/mnt/backup/target.hbk
hbk unmount ~/mnt/backup

hbkit stores no credentials and calls no cloud API. It just picks the rclone flags —
which turns out to decide whether a mounted bucket is usable at all.

Measured on a 3 TB archive in Cloudflare R2:

	before	now
cold mount + directory pre-warm	~12 min	16 s
opening the archive	9 min 21 s	1.8 s
extracting a 10 KB file	584 s	11 s

Three things get you there, and you can apply them by hand if you prefer:

• --no-modtime — the big one. rclone otherwise issues a HEAD per object just to fill
  in modification times for a directory listing (~0.27 s per entry), so the 2,021-shard
  chunk_index directory took 546.8 s. With it, 4.07 s — 134× faster. hbkit never
  uses the modtimes of an archive's internal files, so this costs nothing.
• --dir-cache-time 72h — pay the expensive first listing once.
• Pre-warming — hbk mount walks the index directories up front so the wait is visible
  at mount time rather than looking like a hang on your first command. Also hbk warm <dir>.

--for browse (default) uses range requests, best for opening an archive and cherry-picking
files. --for restore uses whole-file fetching as read-ahead for bulk restores, with the
cache capped at 50G since it holds whole ~50 MB bucket files.

Also: 6 new tests covering the mount helpers, all network-free so they run in CI.

hbkit 0.2.1 — network mounts

Makes archives on network mounts (rclone / S3 / R2) practical to open.

Cat used to stat() every index shard to build its offset table — about 2,600
round-trips on a 3 TB archive. Negligible on local disk; on a mounted bucket it meant
minutes before a single byte of backup data was read.

Shards are a fixed 8 MiB except the last, so offsets are now computed: one directory
listing plus a single stat of the final shard per index family. read() indexes straight
to the right shard instead of scanning all of them, and file_chunk<N>.index families are
opened only when a file actually references one.

If that fixed-size assumption were ever violated, a short read surfaces as a chunk MD5
failure — loudly — never as silent corruption.

If you mount a bucket, the flags matter more than hbkit does:

rclone mount r2:bucket ~/mnt/r2 --read-only \
  --vfs-cache-mode off \      # range requests
  --dir-cache-time 72h

Do not use --vfs-cache-mode full. hbkit reads a 32-byte index record and a ~5 KB
chunk at a time; in full mode each pulls a whole file, so a measured 10 KB extraction
downloaded ~82 MB and took ten minutes.

Even configured well, a network mount is far slower than local storage — thousands of
small scattered reads. Copy the archive locally if you can. Treat mounted-bucket recovery
as workable for pulling out a few files, not for restoring terabytes.

Also adds synthetic-shard tests covering the multi-shard read path (boundary spanning,
short final shard, single-shard case) — they need no archive, so CI exercises them.

hbkit 0.2.0 — encrypted archives

Encrypted Hyper Backup archives are now supported.

hbk /path/to/backup doctor -p yourpassword
hbk-tui /path/to/backup          # shows a password field when it detects encryption

Both chunk payloads and filenames are decrypted, so the file tree, search, globs and
output paths all work on real names — browsing an encrypted archive needs the password
because the directory tree itself is ciphertext.

A wrong password is rejected instantly, before any backup data is read: the public
key derived from your password is compared against the one stored in the archive.

Scheme (see FORMAT.md for the full spec, now including section 7):

salt      = MD5(unikey)
seed      = argon2i13(password, salt, ops=4, mem=16MiB, 32B)
pub, priv = crypto_box_seed_keypair(seed)      # pub == Config/public.pem
key, iv   = crypto_box_seal_open(vkey.rsa_vkey / rsa_vkey_iv)
content   = LZ4(unpad(AES-256-CBC(ct, key, iv)))
fnKey     = SHA256(priv || unikey)
fnIV      = MD5(unikey || secret_saltFnKeyIV)

Verified byte-exact — names and contents — against a real encrypted archive, checked
against checksums taken from the originals beforehand.

Not covered: older RSA-wrapped archives, and key-file (rather than password) unlock.
doctor will tell you which kind you have.

Note: the index cache in ~/.cache/hbkit stores decrypted filenames (not contents)
for encrypted archives. Delete it if that matters to you.

Also in this release: pynacl and cryptography are now required dependencies.