Summary
Follow-up to #118. The v1 dataless handling (detect SF_DATALESS, propagate from parent snapshot on (size, mtime, inode) match, never open) works well: warm-at-any-point files stay in every snapshot, no bandwidth is burned, and the warning noise is gone.
What remains is the gap called out as deferred in the #118 resolution: a dataless file that is new, or changed remotely since it was last warm, is never captured. For iCloud-heavy homes ("Desktop & Documents" is one checkbox), that can be tens of GB of user documents that exist in exactly one place — Apple's cloud. Accidental-edit protection, the main reason to back these files up at all, is exactly the case where the local copy is dataless and the remote copy is the one that changed.
Key observation: no download needed to detect change
There is no public API to get a content checksum of a cloud-only file without hydrating it. But none is needed: fileproviderd keeps the placeholder's stat metadata current — when the remote content changes, the dataless inode's size/mtime are updated without materialization.
The existing parent-match identity (size, mtime, inode) is therefore already a remote-change detector:
| Dataless file state |
Action |
Bandwidth |
| identity matches parent snapshot |
propagate ChunkRefs (shipped in #118) |
none |
| identity differs / no parent entry |
hydrate this one file, chunk it, optionally evict |
that file only |
Steady-state cost per backup is the remote delta only. The first run in a hydrate mode pays a one-time cost for files never previously captured.
Proposed configuration
Per-source setting with a global default, as sketched in #118:
sources:
- path: /Users/jbb
dataless: skip # current behavior, stays the default
| Mode |
Behavior |
skip (default) |
current v1 behavior: parent-propagate or omit |
hydrate |
on identity mismatch, read the file normally (open triggers materialization); file stays warm afterwards |
hydrate-evict |
as hydrate, then return the file to dataless after the snapshot commits, via NSFileProviderManager.evictItem |
Implementation notes
Hydration read path. Opening a dataless file starts async materialization; size/blocks change mid-read, which today trips the drift guard. Cleanest is materialize-then-read: after open(), poll fstat until SF_DATALESS clears (with backoff and a per-file timeout), then chunk through the existing unchanged path so the drift guard keeps its semantics. This also sidesteps the EDEADLK class of problems (#183) on the file read path.
Eviction safety (hydrate-evict).
- Only evict files vykar itself hydrated this run — never files that were already warm (the user may be working on them).
- Re-stat immediately before evicting; if mtime advanced since hydration, the user touched it — skip eviction.
- Eviction is best-effort: log failures (file in use, provider busy) and continue; never fail the backup.
- Needs a small Swift/Obj-C bridge for
NSFileProviderManager.evictItem(identifier:) (macOS 11+), behind #[cfg(target_os = "macos")].
Guards.
- Pre-flight: sum the sizes of files that will hydrate this run and refuse with a clear error if it exceeds free disk space, rather than ENOSPC mid-backup.
hydrate_parallel: N knob to bound concurrent hydrations (head-of-line blocking on one multi-GB file is otherwise easy to hit).- End-of-source summary line: hydrated N files / M bytes, evicted K.
Interaction with the file cache. The ctime-ignoring dataless lookup from #118 already covers the evict-rehydrate ctime churn, so hydrate-evict does not re-download unchanged files on subsequent runs — that was the prerequisite for this mode being viable at all.
Restore semantics are unchanged from #118: restored files are warm, and FileProvider xattrs should be stripped on restore (still open as its own deferred item).
Sequencing
Two PRs would keep review small:
dataless: skip|hydrate — config plumbing, materialize-then-read, guards, tests. Pure Rust, no bridge.hydrate-evict — the Swift/Obj-C bridge and eviction tracking.
Happy to build both, pending direction on the approach.
Summary
Follow-up to #118. The v1 dataless handling (detect
SF_DATALESS, propagate from parent snapshot on(size, mtime, inode)match, never open) works well: warm-at-any-point files stay in every snapshot, no bandwidth is burned, and the warning noise is gone.What remains is the gap called out as deferred in the #118 resolution: a dataless file that is new, or changed remotely since it was last warm, is never captured. For iCloud-heavy homes ("Desktop & Documents" is one checkbox), that can be tens of GB of user documents that exist in exactly one place — Apple's cloud. Accidental-edit protection, the main reason to back these files up at all, is exactly the case where the local copy is dataless and the remote copy is the one that changed.
Key observation: no download needed to detect change
There is no public API to get a content checksum of a cloud-only file without hydrating it. But none is needed:
fileproviderdkeeps the placeholder's stat metadata current — when the remote content changes, the dataless inode'ssize/mtimeare updated without materialization.The existing parent-match identity
(size, mtime, inode)is therefore already a remote-change detector:Steady-state cost per backup is the remote delta only. The first run in a hydrate mode pays a one-time cost for files never previously captured.
Proposed configuration
Per-source setting with a global default, as sketched in #118:
skip(default)hydratehydrate-evicthydrate, then return the file to dataless after the snapshot commits, viaNSFileProviderManager.evictItemImplementation notes
Hydration read path. Opening a dataless file starts async materialization; size/blocks change mid-read, which today trips the drift guard. Cleanest is materialize-then-read: after
open(), pollfstatuntilSF_DATALESSclears (with backoff and a per-file timeout), then chunk through the existing unchanged path so the drift guard keeps its semantics. This also sidesteps the EDEADLK class of problems (#183) on the file read path.Eviction safety (
hydrate-evict).NSFileProviderManager.evictItem(identifier:)(macOS 11+), behind#[cfg(target_os = "macos")].Guards.
hydrate_parallel: Nknob to bound concurrent hydrations (head-of-line blocking on one multi-GB file is otherwise easy to hit).Interaction with the file cache. The ctime-ignoring dataless lookup from #118 already covers the evict-rehydrate ctime churn, so
hydrate-evictdoes not re-download unchanged files on subsequent runs — that was the prerequisite for this mode being viable at all.Restore semantics are unchanged from #118: restored files are warm, and FileProvider xattrs should be stripped on restore (still open as its own deferred item).
Sequencing
Two PRs would keep review small:
dataless: skip|hydrate— config plumbing, materialize-then-read, guards, tests. Pure Rust, no bridge.hydrate-evict— the Swift/Obj-C bridge and eviction tracking.Happy to build both, pending direction on the approach.