-
-
Notifications
You must be signed in to change notification settings - Fork 5
Download Pipeline
icloudpd-rs uses a streaming, concurrent download architecture designed for large libraries.
Assets flow from the iCloud API directly into the download pipeline. The API is paginated, and while one page of results is being downloaded, the next page is being fetched. This eliminates the multi-minute startup delay that the Python version experiences on large libraries.
For a library with 100k+ photos, downloads begin within seconds of authentication completing.
Multiple files can be downloaded simultaneously using --threads-num (default: 10). Downloads use buffer_unordered — files complete in whatever order they finish, not the order they were started.
File downloads use a dedicated HTTP client with no total request timeout. This prevents large files (e.g., 500MB MOV videos) from being killed mid-transfer when sharing bandwidth with many concurrent downloads. The download client uses:
- 30s connect timeout — fast failure for unreachable hosts
- 120s read timeout — detects stalled connections (no bytes received for 120s) without killing slow-but-progressing transfers
API calls (authentication, album enumeration) continue to use a 30s total timeout for fast failure on errors.
If a download is interrupted, the partially downloaded .part file is kept. On the next run, the existing bytes are verified and the download resumes from where it left off using HTTP Range requests. The final SHA256 checksum covers the entire file (existing + new bytes).
Resume hashing uses a 256KB buffer for fast re-verification of large partial files.
After the main download pass, any failed downloads get a second attempt. The cleanup pass re-fetches CDN URLs from iCloud before retrying, which fixes failures caused by expired download URLs on large files.
If Apple invalidates the session mid-sync (common on very large libraries that run for hours), downloads start failing with HTTP 401/403 errors. The pipeline detects these auth errors and, after a threshold is reached, pauses downloads and triggers automatic re-authentication.
Once re-authenticated, the download pass resumes. Already-downloaded files are naturally skipped, so progress isn't lost. See Authentication for details.
Every download is verified against the SHA256 checksum provided by iCloud. Both Apple's 32-byte raw format and 33-byte prefixed format are handled.
When a shutdown signal is received (Ctrl+C, SIGTERM, SIGHUP), the pipeline finishes any downloads already in flight, then stops processing new ones. Partial .part files from interrupted downloads are preserved and resumed on the next run via HTTP Range requests.
A second signal force-exits immediately.
When multiple iCloud assets share the same filename (common with generic names like IMG_0001.jpg), the default --file-match-policy (name-size-dedup-with-suffix) ensures both files are preserved:
- If a file exists with the same size, it's considered already downloaded and skipped
- If a file exists with a different size, the new file is saved with a size suffix (e.g.,
photo-12345.jpg)
This matches Python icloudpd's behavior and ensures no photos are silently skipped due to filename collisions.
Downloads write to a temporary .part file, then atomically rename to the final path. This prevents partial files from appearing in the download directory.
A progress bar tracks downloads in real time, showing the number of assets processed out of the total. It auto-hides when stdout is not a TTY (e.g., cron jobs or piped output) or when --no-progress-bar is set.
Already-downloaded files advance the progress counter on resume, so the bar reflects true progress rather than starting from zero.
The total is based on photo count. Each photo can produce multiple files (live photo MOVs, RAW alternates), so the counter may slightly overshoot the total.