Silently downloads iCloud placeholder files, making scans appear to hang

[fahadsiddiqui]

Summary

On macOS with iCloud Drive's "Desktop & Documents" plus Optimize Mac Storage, files can be dataless placeholders: the directory entry and size are local, but the contents live in iCloud. Reading such a file forces macOS to download it.

calculateChecksum opens and reads every file, so dedup silently pulls gigabytes over the network with no indication of why it has stalled.

Reproduction

Measured on a real iCloud-synced Desktop, 456 files / 362 MB:

272/456 files are dataless (st_blocks == 0)
200.0 MB would be downloaded from iCloud to hash them

operation	time
stat-only pass over all 456 files	0.115 s
dedup --quiet --no-script ~/Desktop	>120 s — killed before it finished

Three orders of magnitude, none of it CPU. Every Processing: line blocks on a network fetch.

Detection

A dataless file reports st_blocks == 0 with a non-zero st_size:

    180871 bytes  blocks=       0   <- placeholder
   1123364 bytes  blocks=    2200   <- local

In Go this is syscall.Stat_t.Blocks via os.Stat().Sys(). macOS also exposes the SF_DATALESS flag (st_flags), and NSURLUbiquitousItemDownloadingStatusKey at the Foundation level. st_blocks == 0 && st_size > 0 is the cheapest portable-ish check and covers APFS placeholders; note it can also be true for sparse files, which is acceptable here since a sparse file is equally cheap to skip.

Impact

• Silent, unbounded network transfer. Nothing in the output says "downloading"; the tool just appears hung.
• Defeats Optimize Mac Storage. Scanning re-materialises evicted files, re-filling the disk the setting exists to free.
• Metered/offline connections. Potentially expensive or simply impossible.
• Affects the tool's single most likely target directories — Desktop, Documents, Downloads.

Expected behaviour

1. Detect dataless files during the stat pass.
2. Skip them by default, and report the count and total size in the report's warnings, e.g.
   Skipped 272 file(s) (200.0 MB) not downloaded from iCloud — pass --include-cloud to hash them.
3. Add --include-cloud for users who genuinely want them materialised.
4. Never hash a placeholder without saying so first.

Interaction with #12

The size-first funnel largely mitigates this by accident: on the measured Desktop, 450 of 456 files have a size no other file shares, so hashing — and therefore downloading — is unnecessary for 98% of them. os.Stat does not materialise a placeholder.

But #12 alone is not sufficient: a size collision between two dataless files would still trigger a silent download. #12 should land first, then this check on top of it.

Tests to write first

• A file with st_blocks == 0 and non-zero size is skipped and recorded as a warning.
• --include-cloud hashes it instead.
• A local file with the same size is unaffected.
• Detection is behind a runtime.GOOS == "darwin" guard, or written so other platforms simply never see a placeholder.

Fixture note: creating a genuine dataless file in a test is impractical, so the stat call needs a seam the test can substitute.