PYTHON-5522: Support std lib zstandard in 3.14

[Rogdham]

Issue Key

Summary

Python 3.14 introduces support for Zstandard in the standard library via the compression.zstd module (PEP-784).

This PR removes the zstdandard dependency in favor of the native support in Python 3.14+. For older Python versions, the backport is used.

It also has the advantage of supporting free threading.

Full disclosure: I'm the author and maintainer of backports.zstd, and the maintainer of pyzstd (which code was used as a base for the integration into Python). I also helped with PEP-784 and its integration into CPython.

Changes in this PR

• Remove zstandard dependency (in the extra group zstd)
• Add backports.zstd dependency for Python before 3.14 (in the extra group zstd)
• Adapt code accordingly

Testing Plan

I tried to run the tests but fail to launch just setup-tests, so all tests were not run.

I let reviewers perform tests on both Python versions before 3.14 and 3.14.

Checklist

Checklist for Author

• Did you update the changelog (if necessary)?
• Is the intention of the code captured in relevant tests?
• If there are new TODOs, has a related JIRA ticket been created?

Checklist for Reviewer {@primary_reviewer}

• Does the title of the PR reference a JIRA Ticket?
• Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
• Have you checked for spelling & grammar errors?
• Is all relevant documentation (README or docstring) updated?

Focus Areas for Reviewer (optional)

[blink1073]

Hi @Rogdham, thanks for this! Unfortunately the version of CPython we have in our CI does not yet include zstandard support, so we'll have to block this PR on getting that added to our infrastructure. I will circle back once that is complete.

[blink1073]

I opened PYTHON-5625 and blocked this ticket on that work.

[ShaneHarvey]

I would prefer we keep using zstandard to limit the scope of these changes.

[ShaneHarvey]

@Rogdham what are the benefits to using backports.zstd vs zstandard? I'd prefer to avoid changing our dependency unless there is a strong motivation.

[Rogdham]

The main benefit is to be sure to have the same exact API as compressor.zstd, so switching one for the other based on Python version will always work. This is guarantied because it is the exact same code as in CPython, with only the needed modifications to support older versions.

Other benefits are less clear and depends on the use cases:

• Free threading support on Python 3.13
• Wheels are smaller (e.g. for Linux 64bits on 3.13 490.8kB vs 5.5 MB)
• Adds support for Zstandard in other stdlib modules such as tarfile or zipfile

[blink1073]

After working on PYTHON-5625, we actually do support compression.zstd on Ubuntu 22. We don't test on Ubuntu 22 by default because it doesn't support all server versions. If you add the following change it will add tests for the combination of Linux and compression.zstd support, and keep a set of tests that do not have compression.zstd on RHEL 8:

diff --git a/.evergreen/scripts/generate_config.py b/.evergreen/scripts/generate_config.py
index daec0841d..116aea77d 100644
--- a/.evergreen/scripts/generate_config.py
+++ b/.evergreen/scripts/generate_config.py
@@ -194,6 +194,22 @@ def create_compression_variants():
                 expansions=expansions,
             )
         )
+    # Add explicit tests with compression.zstd support on linux.
+    host = HOSTS["ubuntu22"]
+    expansions = dict(COMPRESSOR="ztsd")
+    tasks = [
+        ".test-standard !.server-4.2 !.server-4.4 !.server-5.0 .python-3.14",
+        ".test-standard !.server-4.2 !.server-4.4 !.server-5.0 .python-3.14t",
+    ]
+    display_name = get_variant_name(f"Compression {compressor}", host)
+    variants.append(
+        create_variant(
+            tasks,
+            display_name,
+            host=host,
+            expansions=expansions,
+        )
+    )
     return variants

You would then run just lint generate-config to update the generated CI files. I am happy to do this and push the change if you prefer.

[Rogdham]

Thank you for looking into this @blink1073!

I think I managed to apply your suggested changes bindly 🙈

[blink1073]

Thanks @Rogdham! Here's the result of the run: