bz2, lzma: add option to limit output size

[tiran]

BPO	15955
Nosy	@pitrou, @vstinner, @tiran, @merwok, @vadmium, @serhiy-storchaka
Files	issue15955.diff issue15955_r2.diff issue15955_r3.diff gzip-bomb.patch: Moved to Issue 23528 LZMAFile.patch: New version in Issue 23529 issue15955_lzma_r4.diff: Adds max_length support in LZMADecompressor issue15955_lzma_r5.diff: Adds max_length support in LZMADecompressor issue15955_bz2.diff: Adds max_length support in BZ2Decompressor issue15955_bz2_rev2.diff: Adds max_length support in BZ2Decompressor issue15955_bz2_rev3.diff issue15955_bz2_rev5.diff issue15955_bz2_rev6.diff issue15955_bz2_rev7.diff: Adds max_length support in BZ2Decompressor

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-02-26.12:12:10.383>
created_at = <Date 2012-09-17.12:22:42.212>
labels = ['type-feature', 'library']
title = 'bz2, lzma: add option to limit output size'
updated_at = <Date 2015-02-26.12:12:10.381>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2015-02-26.12:12:10.381>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2015-02-26.12:12:10.383>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2012-09-17.12:22:42.212>
creator = 'christian.heimes'
dependencies = []
files = ['34609', '34789', '34792', '37644', '37658', '37677', '37734', '38194', '38206', '38211', '38217', '38218', '38226']
hgrepos = []
issue_num = 15955
keywords = ['patch']
message_count = 56.0
messages = ['170598', '171059', '171248', '171257', '171304', '174860', '174912', '175028', '176811', '176883', '176896', '177213', '177228', '180271', '187534', '208522', '208763', '209151', '209228', '209999', '214773', '214774', '215211', '215657', '215661', '215959', '220580', '220675', '225219', '226685', '232953', '233661', '233799', '233865', '233948', '234115', '234160', '234172', '234173', '235084', '235123', '235124', '236248', '236351', '236356', '236416', '236450', '236463', '236468', '236473', '236474', '236514', '236647', '236664', '236665', '236666']
nosy_count = 10.0
nosy_names = ['pitrou', 'vstinner', 'christian.heimes', 'nadeem.vawda', 'eric.araujo', 'Arfrever', 'nikratio', 'python-dev', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue15955'
versions = ['Python 3.5']

[tiran]

The gzip, bz2 and lzma file reader have no method to get the actual size and compression ratio of a compressed file. In my opinion it's useful to know how much disk space a file will need before it's decompressed.

[nadeemvawda]

As far as I can tell, there is no way to find this out reliably without decompressing the entire file. With gzip, the file trailer contains the uncompressed size modulo 2^32, but this seems less than useful. It appears that the other two formats do not store the total uncompressed data size in any form.

For bz2 and lzma, one can get the uncompressed size by doing f.seek(0, 2) followed by f.tell(). However this approach is ugly and potentially very slow, so I would be reluctant to add a method based on it to the (BZ2|LZMA)File classes.

[tiran]

I've checked the implementation of the gzip command. It uses some tricks to get the result size of a compressed file. The bzip2 command doesn't have the ability. lzmainfo can print the actual size of the file but it says "Unknown" for my test file.

Also see bpo-16043 why this would be a useful feature.

[pitrou]

You don't need to know the final size. You just need to be able to pass an output size limit. This would be an easy feature to add to functions like zlib.decompress(), since they already handle sizing of the output buffer.

[serhiy-storchaka]

Agree with Antoine. This would be a desirable feature.

[nadeemvawda]

I agree that being able to limit output size is useful and desirable, but
I'm not keen on copying the max_length/unconsumed_tail approach used by
zlib's decompressor class. It feels awkward to use, and it complicates
the implementation of the existing decompress() method, which is already
unwieldy enough.

As an alternative, I propose a thin wrapper around the underlying C API:

    def decompress_into(self, src, dst, src_start=0, dst_start=0): ...

This would store decompressed data in a caller-provided bytearray, and
return a pair of integers indicating the end points of the consumed and
produced data in the respective buffers.

The implementation should be extremely simple - it does not need to do
any memory allocation or reference management.

I think it could also be useful for optimizing the implementation of
BZ2File and LZMAFile. I plan to write a prototype and run some benchmarks
some time in the next few weeks.

(Aside: if implemented for zlib, this could also be a nicer (I think)
solution for the problem raised in bpo-5804.)

[serhiy-storchaka]

unconsumed_tail should be private hidden attribute, which automatically prepends any consumed data. This should not be very complicated. But benchmarks needed to show what kind of approach is more efficient.

[nadeemvawda]

I suspect that it will be slower than the decompress_into() approach, but
as you say, we need to do benchmarks to see for sure.

[nadeemvawda]

I've tried reimplementing LZMAFile in terms of the decompress_into()
method, and it has ended up not being any faster than the existing
implementation. (It is _slightly_ faster for readinto() with a large
buffer size, but all other cases it was either of equal performance or
significantly slower.)

In addition, decompress_into() is more complicated to work with than I
had expected, so I withdraw my objection to the approach based on
max_length/unconsumed_tail.

unconsumed_tail should be private hidden attribute, which automatically prepends any consumed data.

I don't think this is a good idea. In order to have predictable memory
usage, the caller will need to ensure that the current input is fully
decompressed before passing in the next block of compressed data. This
can be done more simply with the interface used by zlib. Compare:

    while not d.eof:
        output = d.decompress(b'', 8192)
        if not output:
            compressed = f.read(8192)
            if not compressed:
                raise ValueError('End-of-stream marker not found')
            output = d.decompress(compressed, 8192)
        # <process output>

with:

    # Using zlib's interface
    while not d.eof:
        compressed = d.unconsumed_tail or f.read(8192)
        if not compressed:
            raise ValueError('End-of-stream marker not found')
        output = d.decompress(compressed, 8192)
        # <process output>

A related, but orthogonal proposal: We might want to make unconsumed_tail
a memoryview (provided the input data is know to be immutable), to avoid
creating an unnecessary copy of the data.

[serhiy-storchaka]

# Using zlib's interface
while not d.eof:
    compressed = d.unconsumed_tail or f.read(8192)
    if not compressed:
        raise ValueError('End-of-stream marker not found')
    output = d.decompress(compressed, 8192)
    # \<process output\>

This is not usable with bzip2. Bzip2 uses large block size and unconsumed_tail
can be non empty but decompress() will return b''. With zlib you possible can
see the same effect on some input when read by one byte.

A related, but orthogonal proposal: We might want to make unconsumed_tail
a memoryview (provided the input data is know to be immutable), to avoid
creating an unnecessary copy of the data.

It looks interesting. However the data should be copied anyway if the input
data is not a bytes object.

[serhiy-storchaka]

Actually it should be:

    # Using zlib's interface
    while not d.eof:
        output = d.decompress(d.unconsumed_tail, 8192)
        while not output and not d.eof:
            compressed = f.read(8192)
            if not compressed:
                raise ValueError('End-of-stream marker not found')
            output = d.decompress(d.unconsumed_tail + compressed, 8192)
        # <process output>

Note that you should use d.unconsumed_tail + compressed as input, and therefore do an unnecessary copy of the data.

Without explicit unconsumed_tail you can write input data in the internal mutable buffer, it will be more effective for large buffer (handreds of KB) and small input chunks (several KB).

[nadeemvawda]

> # Using zlib's interface
> while not d.eof:
> compressed = d.unconsumed_tail or f.read(8192)
> if not compressed:
> raise ValueError('End-of-stream marker not found')
> output = d.decompress(compressed, 8192)
> # <process output>

This is not usable with bzip2. Bzip2 uses large block size and unconsumed_tail
can be non empty but decompress() will return b''. With zlib you possible can
see the same effect on some input when read by one byte.

I don't see how this is a problem. If (for some strange reason) the
application-specific processing code can't handle empty blocks properly, you can
just stick "if not output: continue" before it.

Actually it should be:

# Using zlib's interface
while not d.eof:
    output = d.decompress(d.unconsumed_tail, 8192)
    while not output and not d.eof:
        compressed = f.read(8192)
        if not compressed:
            raise ValueError('End-of-stream marker not found')
        output = d.decompress(d.unconsumed_tail + compressed, 8192)
    # \<process output\>

Note that you should use d.unconsumed_tail + compressed as input, and therefore
do an unnecessary copy of the data.

Why is this necessary? If unconsumed_tail is b'', then there's no need to
prepend it (and the concatenation would be a no-op anyway). If unconsumed_tail
does contain data, then we don't need to read additional compressed data from
the file until we've finished decompressing the data we already have.

Without explicit unconsumed_tail you can write input data in the internal
mutable buffer, it will be more effective for large buffer (handreds of KB)
and small input chunks (several KB).

Are you proposing that the decompressor object maintain its own buffer, and
copy the input data into it before passing it to the decompression library?
Doesn't that just duplicate work that the library is already doing for us?