Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 15 additions & 13 deletions Lib/compression/zstd/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,9 @@ def __setattr__(self, name, _):
def get_frame_info(frame_buffer):
"""Get Zstandard frame information from a frame header.

*frame_buffer* is a bytes-like object. It should start from the beginning
of a frame, and needs to include at least the frame header (6 to 18 bytes).
*frame_buffer* is a bytes-like object. It should start from the
beginning of a frame, and needs to include at least the frame header
(6 to 18 bytes).

The returned FrameInfo object has two attributes.
'decompressed_size' is the size in bytes of the data in the frame when
Expand Down Expand Up @@ -103,16 +104,17 @@ def finalize_dict(zstd_dict, /, samples, dict_size, level):
finalize *zstd_dict* by adding headers and statistics according to the
Zstandard dictionary format.

You may compose an effective dictionary content by hand, which is used as
basis dictionary, and use some samples to finalize a dictionary. The basis
dictionary may be a "raw content" dictionary. See *is_raw* in ZstdDict.
You may compose an effective dictionary content by hand, which is used
as basis dictionary, and use some samples to finalize a dictionary. The
basis dictionary may be a "raw content" dictionary. See *is_raw* in
ZstdDict.

*samples* is an iterable of samples, where a sample is a bytes-like object
representing a file.
*samples* is an iterable of samples, where a sample is a bytes-like
object representing a file.
*dict_size* is the dictionary's maximum size, in bytes.
*level* is the expected compression level. The statistics for each
compression level differ, so tuning the dictionary to the compression level
can provide improvements.
compression level differ, so tuning the dictionary to the compression
level can provide improvements.
"""

if not isinstance(zstd_dict, ZstdDict):
Expand Down Expand Up @@ -140,8 +142,8 @@ def compress(data, level=None, options=None, zstd_dict=None):
COMPRESSION_LEVEL_DEFAULT ('3').
*options* is a dict object that contains advanced compression
parameters. See CompressionParameter for more on options.
*zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary. See
the function train_dict for how to train a ZstdDict on sample data.
*zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary.
See the function train_dict for how to train a ZstdDict on sample data.

For incremental compression, use a ZstdCompressor instead.
"""
Expand All @@ -152,8 +154,8 @@ def compress(data, level=None, options=None, zstd_dict=None):
def decompress(data, zstd_dict=None, options=None):
"""Decompress one or more frames of Zstandard compressed *data*.

*zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary. See
the function train_dict for how to train a ZstdDict on sample data.
*zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary.
See the function train_dict for how to train a ZstdDict on sample data.
*options* is a dict object that contains advanced compression
parameters. See DecompressionParameter for more on options.

Expand Down
39 changes: 20 additions & 19 deletions Lib/compression/zstd/_zstdfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ def __init__(self, file, /, mode='r', *,

*file* can be either an file-like object, or a file name to open.

*mode* can be 'r' for reading (default), 'w' for (over)writing, 'x' for
creating exclusively, or 'a' for appending. These can equivalently be
given as 'rb', 'wb', 'xb' and 'ab' respectively.
*mode* can be 'r' for reading (default), 'w' for (over)writing, 'x'
for creating exclusively, or 'a' for appending. These can
equivalently be given as 'rb', 'wb', 'xb' and 'ab' respectively.

*level* is an optional int specifying the compression level to use,
or COMPRESSION_LEVEL_DEFAULT if not given.
Expand Down Expand Up @@ -296,26 +296,27 @@ def open(file, /, mode='rb', *, level=None, options=None, zstd_dict=None,
encoding=None, errors=None, newline=None):
"""Open a Zstandard compressed file in binary or text mode.

file can be either a file name (given as a str, bytes, or PathLike object),
in which case the named file is opened, or it can be an existing file object
to read from or write to.
file can be either a file name (given as a str, bytes, or PathLike
object), in which case the named file is opened, or it can be
an existing file object to read from or write to.

The mode parameter can be 'r', 'rb' (default), 'w', 'wb', 'x', 'xb', 'a',
'ab' for binary mode, or 'rt', 'wt', 'xt', 'at' for text mode.
The mode parameter can be 'r', 'rb' (default), 'w', 'wb', 'x', 'xb',
'a', 'ab' for binary mode, or 'rt', 'wt', 'xt', 'at' for text mode.

The level, options, and zstd_dict parameters specify the settings the same
as ZstdFile.
The level, options, and zstd_dict parameters specify the settings the
same as ZstdFile.

When using read mode (decompression), the options parameter is a dict
representing advanced decompression options. The level parameter is not
supported in this case. When using write mode (compression), only one of
level, an int representing the compression level, or options, a dict
representing advanced compression options, may be passed. In both modes,
zstd_dict is a ZstdDict instance containing a trained Zstandard dictionary.

For binary mode, this function is equivalent to the ZstdFile constructor:
ZstdFile(filename, mode, ...). In this case, the encoding, errors and
newline parameters must not be provided.
representing advanced decompression options. The level parameter is not
supported in this case. When using write mode (compression), only one
of level, an int representing the compression level, or options, a dict
representing advanced compression options, may be passed. In both
modes, zstd_dict is a ZstdDict instance containing a trained Zstandard
dictionary.

For binary mode, this function is equivalent to the ZstdFile
constructor: ZstdFile(filename, mode, ...). In this case, the encoding,
errors and newline parameters must not be provided.

For text mode, an ZstdFile object is created, and wrapped in an
io.TextIOWrapper instance with the specified encoding, error handling
Expand Down
31 changes: 16 additions & 15 deletions Modules/_zstd/clinic/compressor.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 16 additions & 14 deletions Modules/_zstd/clinic/decompressor.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 16 additions & 14 deletions Modules/_zstd/clinic/zstddict.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

37 changes: 19 additions & 18 deletions Modules/_zstd/compressor.c
Original file line number Diff line number Diff line change
Expand Up @@ -332,14 +332,14 @@ _zstd.ZstdCompressor.__new__ as _zstd_ZstdCompressor_new

Create a compressor object for compressing data incrementally.

Thread-safe at method level. For one-shot compression, use the compress()
function instead.
Thread-safe at method level. For one-shot compression, use the
compress() function instead.
[clinic start generated code]*/

static PyObject *
_zstd_ZstdCompressor_new_impl(PyTypeObject *type, PyObject *level,
PyObject *options, PyObject *zstd_dict)
/*[clinic end generated code: output=cdef61eafecac3d7 input=92de0211ae20ffdc]*/
/*[clinic end generated code: output=cdef61eafecac3d7 input=bbfeeaa06fd3bd4d]*/
{
ZstdCompressor* self = PyObject_GC_New(ZstdCompressor, type);
if (self == NULL) {
Expand Down Expand Up @@ -592,15 +592,15 @@ _zstd.ZstdCompressor.compress

Provide data to the compressor object.

Return a chunk of compressed data if possible, or b'' otherwise. When you have
finished providing data to the compressor, call the flush() method to finish
the compression process.
Return a chunk of compressed data if possible, or b'' otherwise.
When you have finished providing data to the compressor, call the
flush() method to finish the compression process.
[clinic start generated code]*/

static PyObject *
_zstd_ZstdCompressor_compress_impl(ZstdCompressor *self, Py_buffer *data,
int mode)
/*[clinic end generated code: output=ed7982d1cf7b4f98 input=ac2c21d180f579ea]*/
/*[clinic end generated code: output=ed7982d1cf7b4f98 input=11726dff64d7b2f9]*/
{
PyObject *ret;

Expand Down Expand Up @@ -650,14 +650,14 @@ _zstd.ZstdCompressor.flush

Finish the compression process.

Flush any remaining data left in internal buffers. Since Zstandard data
consists of one or more independent frames, the compressor object can still
be used after this method is called.
Flush any remaining data left in internal buffers. Since Zstandard
data consists of one or more independent frames, the compressor
object can still be used after this method is called.
[clinic start generated code]*/

static PyObject *
_zstd_ZstdCompressor_flush_impl(ZstdCompressor *self, int mode)
/*[clinic end generated code: output=b7cf2c8d64dcf2e3 input=0ab19627f323cdbc]*/
/*[clinic end generated code: output=b7cf2c8d64dcf2e3 input=130e0b1eddf0f498]*/
{
PyObject *ret;

Expand Down Expand Up @@ -699,19 +699,20 @@ _zstd.ZstdCompressor.set_pledged_input_size

Set the uncompressed content size to be written into the frame header.

This method can be used to ensure the header of the frame about to be written
includes the size of the data, unless the CompressionParameter.content_size_flag
is set to False. If last_mode != FLUSH_FRAME, then a RuntimeError is raised.
This method can be used to ensure the header of the frame about to
be written includes the size of the data, unless the
CompressionParameter.content_size_flag is set to False.
If last_mode != FLUSH_FRAME, then a RuntimeError is raised.

It is important to ensure that the pledged data size matches the actual data
size. If they do not match the compressed output data may be corrupted and the
final chunk written may be lost.
It is important to ensure that the pledged data size matches the
actual data size. If they do not match the compressed output data
may be corrupted and the final chunk written may be lost.
[clinic start generated code]*/

static PyObject *
_zstd_ZstdCompressor_set_pledged_input_size_impl(ZstdCompressor *self,
unsigned long long size)
/*[clinic end generated code: output=3a09e55cc0e3b4f9 input=afd8a7d78cff2eb5]*/
/*[clinic end generated code: output=3a09e55cc0e3b4f9 input=d6d1669171af3da4]*/
{
// Error occured while converting argument, should be unreachable
assert(size != ZSTD_CONTENTSIZE_ERROR);
Expand Down
Loading
Loading