Skip to content

Compile compression.zstd as added in CPython 3.14#6240

Merged
agriyakhetarpal merged 10 commits into
pyodide:mainfrom
agriyakhetarpal:zstd-py314
May 18, 2026
Merged

Compile compression.zstd as added in CPython 3.14#6240
agriyakhetarpal merged 10 commits into
pyodide:mainfrom
agriyakhetarpal:zstd-py314

Conversation

@agriyakhetarpal
Copy link
Copy Markdown
Member

@agriyakhetarpal agriyakhetarpal commented May 15, 2026

Description

xref #6233, #6033, #5995. This PR adds the compression.zstd module which is now available in Python 3.14, but we weren't including it.

I caught this while working on zarr-developers/numcodecs#529 locally as part of a dependency that used the module, but I couldn't import it.

Checklist

  • Add a CHANGELOG entry
  • Add / update tests
  • Add new / update outdated documentation

@agriyakhetarpal agriyakhetarpal added this to the 314.0.0 milestone May 15, 2026
@agriyakhetarpal
Copy link
Copy Markdown
Member Author

Okay, this is passing in the test-core- tests for the trio of Chrome, Firefox, and Node.

I guess I just need to update the CHANGELOG, but maybe we should also document that we should periodically look at https://github.com/python/cpython-source-deps and update it too, to stay in sync with CPython. I'll do in my evening later today, but till that time, this is ready for review. Thanks!

@ryanking13
Copy link
Copy Markdown
Member

So it looks like this adds ~10KB to asm.mjs and ~200KB to asm.wasm (compressed). It is not a blocker for this PR, but maybe worth remembering.

- pyodide.asm.mjs:
    Original size: 1170 KB
    Gzip compressed size (level 1): 292 KB
    Gzip compressed size (level 6): 245 KB
    Gzip compressed size (level 9): 245 KB
    Brotli compressed size: 195 KB
- pyodide.asm.wasm:
    Original size: 8934 KB
    Gzip compressed size (level 1): 3750 KB
    Gzip compressed size (level 6): 3331 KB
    Gzip compressed size (level 9): 3322 KB
    Brotli compressed size: 2571 KB
- pyodide.asm.mjs:
    Original size: 1241 KB
    Gzip compressed size (level 1): 304 KB
    Gzip compressed size (level 6): 255 KB
    Gzip compressed size (level 9): 255 KB
    Brotli compressed size: 202 KB
- pyodide.asm.wasm:
    Original size: 9640 KB
    Gzip compressed size (level 1): 3986 KB
    Gzip compressed size (level 6): 3530 KB
    Gzip compressed size (level 9): 3517 KB
    Brotli compressed size: 2708 KB

Copy link
Copy Markdown
Member

@ryanking13 ryanking13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Could you check the gha failure? I don't think it is related to this PR, but it seems like GHA or conda changed something recently, making the build fail.

@agriyakhetarpal
Copy link
Copy Markdown
Member Author

Thanks! Fixed in #6241.

I think we can go further and reduce the size a bit. Let me try!

@agriyakhetarpal
Copy link
Copy Markdown
Member Author

Our size-checking script reports no size increase because we report it in KB and it rounds off the increase, effectively hiding it. So I checked: I downloaded pyodide-core.tar.gz from main and the one from here and updated the script to check with higher granularity (will update in a separate PR), and we get:

on main (061a9d0) at the time of writing

- pyodide.asm.mjs:
    Original size: 1,198,477 bytes (1170.39 KB)
    Gzip compressed size (level 1): 299,914 bytes (292.88 KB)
    Gzip compressed size (level 6): 251,707 bytes (245.81 KB)
    Gzip compressed size (level 9): 251,526 bytes (245.63 KB)
    Brotli compressed size: 200,678 bytes (195.97 KB)
- pyodide.asm.wasm:
    Original size: 9,149,350 bytes (8934.91 KB)
    Gzip compressed size (level 1): 3,840,709 bytes (3750.69 KB)
    Gzip compressed size (level 6): 3,411,795 bytes (3331.83 KB)
    Gzip compressed size (level 9): 3,401,934 bytes (3322.20 KB)
    Brotli compressed size: 2,630,679 bytes (2569.02 KB)
- python_stdlib.zip:
    Original size: 2,552,031 bytes (2492.22 KB)
    Gzip compressed size (level 1): 2,512,287 bytes (2453.41 KB)
    Gzip compressed size (level 6): 2,509,781 bytes (2450.96 KB)
    Gzip compressed size (level 9): 2,509,727 bytes (2450.91 KB)
    Brotli compressed size: 2,493,672 bytes (2435.23 KB)

here, after adding the zstd build and after enabling the MinSizeRel build + removing legacy zstd support

- pyodide.asm.mjs:
    Original size: 1,246,605 bytes (1217.39 KB)
    Gzip compressed size (level 1): 308,489 bytes (301.26 KB)
    Gzip compressed size (level 6): 259,000 bytes (252.93 KB)
    Gzip compressed size (level 9): 258,860 bytes (252.79 KB)
    Brotli compressed size: 206,235 bytes (201.40 KB)
- pyodide.asm.wasm:
    Original size: 9,610,091 bytes (9384.85 KB)
    Gzip compressed size (level 1): 3,998,661 bytes (3904.94 KB)
    Gzip compressed size (level 6): 3,549,976 bytes (3466.77 KB)
    Gzip compressed size (level 9): 3,536,690 bytes (3453.80 KB)
    Brotli compressed size: 2,741,612 bytes (2677.36 KB)
- python_stdlib.zip:
    Original size: 2,552,062 bytes (2492.25 KB)
    Gzip compressed size (level 1): 2,512,337 bytes (2453.45 KB)
    Gzip compressed size (level 6): 2,509,843 bytes (2451.02 KB)
    Gzip compressed size (level 9): 2,509,798 bytes (2450.97 KB)
    Brotli compressed size: 2,493,736 bytes (2435.29 KB)

So we now add 8.38 KB (was ~11 KB) to asm.js and 108 KB (was ~195 KB) to asm.wasm compressed, which is much better.

Copy link
Copy Markdown
Member

@ryanking13 ryanking13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome. Thanks!

@agriyakhetarpal agriyakhetarpal merged commit b495c51 into pyodide:main May 18, 2026
55 of 57 checks passed
@agriyakhetarpal agriyakhetarpal deleted the zstd-py314 branch May 18, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants