Skip to content

Provide zstd downloads #5353

@konstin

Description

@konstin

Currently, PyPy only provides .tar.bz2 downloads at https://downloads.python.org/pypy/. Would it be possible to provide .tar.zst downloads too? The are smaller and much faster to unpack.

I noticed in uv that installing PyPy is a lot slower than zstandard compressed CPython:

Benchmark 1: tar -xjf pypy3.11-v7.3.20-linux64.tar.bz2 -C tmp
  Time (mean ± σ):      2.246 s ±  0.339 s    [User: 2.197 s, System: 0.140 s]
  Range (min … max):    2.081 s …  2.851 s    5 runs
 
Benchmark 2: tar -xf pypy3.11-v7.3.20-linux64.tar.zst -C tmp
  Time (mean ± σ):     115.1 ms ±   2.2 ms    [User: 110.2 ms, System: 104.2 ms]
  Range (min … max):   112.6 ms … 117.3 ms    5 runs
 
Summary
  tar -xf pypy3.11-v7.3.20-linux64.tar.zst -C tmp ran
   19.50 ± 2.97 times faster than tar -xjf pypy3.11-v7.3.20-linux64.tar.bz2 -C tmp

File sizes:

  • pypy3.11-v7.3.20-linux64.tar.bz2: 33M
  • pypy3.11-v7.3.20-linux64.tar.zst: 29M

Reproducer:

BZ2="pypy3.11-v7.3.20-linux64.tar.bz2"
TAR="pypy3.11-v7.3.20-linux64.tar"
ZST="pypy3.11-v7.3.20-linux64.tar.zst"

wget -q --show-progress "https://downloads.python.org/pypy/$BZ2"
bzcat "$BZ2" > "$TAR"
zstd -19 -q -k "$TAR" -o "$ZST"

ls -lh "$BZ2" "$ZST" | awk '{printf "  %s: %s\n", $9, $5}'

hyperfine --warmup 1 --runs 5 \
  --prepare 'rm -rf tmp && mkdir tmp' \
  "tar -xjf $BZ2 -C tmp" \
  "tar -xf $ZST -C tmp"

rm -rf tmp

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions