-
-
Notifications
You must be signed in to change notification settings - Fork 5
Download Pipeline
icloudpd-rs uses a streaming, concurrent download architecture designed for large libraries.
Assets flow from the iCloud API directly into the download pipeline. The API is paginated, and while one page of results is being downloaded, the next page is being fetched. This eliminates the multi-minute startup delay that the Python version experiences on large libraries.
For a library with 100k+ photos, downloads begin within seconds of authentication completing.
Multiple files can be downloaded simultaneously using --threads-num. Downloads use buffer_unordered — files complete in whatever order they finish, not the order they were started.
If a download is interrupted, the partially downloaded .part file is kept. On the next run, the existing bytes are verified and the download resumes from where it left off using HTTP Range requests. The final SHA256 checksum covers the entire file (existing + new bytes).
After the main download pass, any failed downloads get a second attempt. The cleanup pass re-fetches CDN URLs from iCloud before retrying, which fixes failures caused by expired download URLs on large files.
Every download is verified against the SHA256 checksum provided by iCloud. Both Apple's 32-byte raw format and 33-byte prefixed format are handled.
Downloads write to a temporary .part file, then atomically rename to the final path. This prevents partial files from appearing in the download directory.
A progress bar tracks downloads in real time, showing the number of assets processed out of the total. It auto-hides when stdout is not a TTY (e.g., cron jobs or piped output) or when --no-progress-bar is set.
The total is based on photo count. Each photo can produce multiple files (live photo MOVs, RAW alternates), so the counter may slightly overshoot the total..