Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WiP] Rewrite zstd decoder to use an API that supports multiple frames (fix issue #3008) #3021

Closed
wants to merge 1 commit into from
Closed

[WiP] Rewrite zstd decoder to use an API that supports multiple frames (fix issue #3008) #3021

wants to merge 1 commit into from

Commits on May 8, 2023

  1. Rewrite zstd decoder to use an API that supports multiple frames 

    Compressed zstd data is composed of one or more frames. python-zstandard has
several APIs that only support decoding one frame. This is not useful for
urllib3 because it will result in decompressed data that is truncated to
1048576 bytes.

This patch changes the zstd decoder to use an API that is known to be able to
decompress multiple frames at a time. It uses ZstdDecompressionReader, which
takes any object that fulfills the interface of io.RawBase. The approach here
is to have ZstdDecoder create an io.BytesIo object, feed data to it, and have
python-zstandard consume data when it can.
    @grossag
    grossag committed May 8, 2023
    Configuration menu
    Copy the full SHA
    76dc4d0 View commit details
    Browse the repository at this point in the history