Skip to content

v1.4.0 – Parallel Downloads, Upload Concurrency & Transfer Benchmarks

Choose a tag to compare

@xSAVIKx xSAVIKx released this 21 Sep 21:35
· 4 commits to main since this release

No breaking changes. Caches saved by v1.1, v1.2 and v1.3 stay valid.

Added

  • Parallel downloads. A restore now fetches any archive larger than download-chunk-size
    (default 8388608, 8 MiB) as concurrent Range requests, download-concurrency (default 8,
    1–32) at a time, in both file and streaming mode, the same fan-out actions/cache uses with a
    larger block (it uses 4 MiB; 8 MiB measured faster on every provider). Each part is retried on
    its own. Archives no larger than one chunk, and every restore with download-concurrency: 1,
    use a single request as before. A provider that answers a ranged
    request with the whole object logs
    s3://<bucket>/<key> does not support ranged GET requests; downloading it in one request. and
    gets the single request. The restore's metrics line reports the part count as downloadParts.
  • upload-concurrency input (default 8, 1–32) on the main and save actions: how many
    multipart parts a save sends at once, in both file and streaming mode. It was fixed at 4.
  • Transfer benchmark workflow (benchmark.yml, manual) that measures save and restore speed
    for several transfer settings against the live providers, and a "Transfer Performance" guide
    with the measured tables and the settings that measured best.

Changed

  • Upload defaults follow actions/cache. The default upload-chunk-size is 64 MiB instead of
    10 MiB and a save sends 8 parts at once instead of 4, so a save may hold up to 512 MiB of parts
    in memory. Set upload-concurrency: 4 and upload-chunk-size: 10485760 to keep the v1.3
    footprint. An upload-chunk-size below 5 MiB or above 128 MiB now warns and uses the default;
    before, a small value was silently replaced by 10 MiB.