Skip to content

v0.5.0 — incremental transfer

Latest

Choose a tag to compare

@phaus phaus released this 12 Nov 20:08
· 5 commits to main since this release

Incremental transfer

Unchanged data is no longer re-transferred. After a successful push the
client stores a cache file (/var/db/kopia-darwin-ppc, override with
KOPIA_CACHE_DIR) recording per directory: file name/size/mtime/object-id
plus each directory object id and the SHA-256 of its JSON.

Next run:

  • files with unchanged size + mtime are not read, chunked or uploaded —
    the previous content-addressed object id is reused
  • directories whose rebuilt JSON hashes identically reuse the previous
    directory object id — unchanged subtrees are skipped entirely

A run therefore costs walk time + changed data. Measured on the G4 for the
production source (96,459 files / 586 MB): v0.4 full push 1 h 14–29 min;
with the v0.5 cache an unchanged tree transfers nothing but the walk.
No local index database; the cache is
written atomically after success. Missing, corrupt or foreign-identity
caches (different server/user/hostname/source) fall back to a full push.

After deleting all snapshots of a source or recreating the repository,
delete the client's cache directory once — otherwise the next push may
reference objects that no longer exist.

Fixed along the way

  • sha256() in kopia_sha256.c was an empty stub since the server does
    all content hashing — the cache is its first real user. Implemented
    SHA-256 + HMAC-SHA256, verified against FIPS 180-4 / RFC 4231 vectors.
  • HPACK: huffman-decoded header strings were not NUL-terminated; the
    dynamic-table bookkeeping read past the allocation (found by ASan).
  • The walker now walks the resolved source path, so ignore rules and cache
    keys stay correct when the source path contains symlinks
    (/tmp → /private/tmp on macOS).

Binaries

  • kopia-darwin-ppc — Mach-O ppc_7400 (G4), statically linked mbedTLS 2.28
  • kopia-darwin-ppc64 — Mach-O ppc64 (G5), untested

Verified in production on the G4: second push of an unchanged tree
produces the identical object id with diag: reused 2 files, 2 dirs;
a changed tree reuses exactly the untouched entries.