compression_wrapper compatibility issue

[piskvorky]

This used to work in 1.5.6, perhaps even later:

import smart_open

some_fp = io.BytesIO()  # any file-like object
…
wrapped = smart_open.compression_wrapper(some_fp, some_filename, 'rb')

It doesn't work now, smart_open.compression_wrapper no longer exists.

What's the reason? I cannot find anything in the CHANGELOG.

[piskvorky]

Upon investigation, it seems like:

from smart_open.smart_open_lib import _compression_wrapper
smart_open.compression_wrapper = _compression_wrapper

will restore this (desirable) functionality.

@mpenkov I'd strongly prefer to avoid such monkey-patching shenanigans. Do you see a better way?

If not, can we bring compression_wrapper back? (and perhaps other useful functions that suffered a similar fate)

[mpenkov]

What is the use case here? What are you attempting to achieve, and what are the parameters you're passing to the wrapper?

[piskvorky]

Parameter 1 = io.BufferedIOBase, parameter 2 = filename str, parameter 3 = mode = 'rb'.

We're wrapping an open file-like binary object so that it decompresses transparently.

[mpenkov]

I demoted compression_wrapper to be an internal function (it's called _compression_wrapper) when I redesigned the smart_open library over a year ago. My reasoning for making it internal was:

• The function wasn't publicly documented anywhere, unlike e.g. the obviously public smart_open and s3_iter
• There was no example code for using that function
• There were no unit tests exercising that function on its own

The main benefits of demoting such functions to internal is a cleaner API.

Here's a better way to satisfy your use case than the import-rename pattern you suggested:

>>> import io, gzip
>>>
>>> # prepare some gzipped data
>>> buf = io.BytesIO()
>>> with gzip.GzipFile(fileobj=buf, 'w') as fout: fout.write(b'this is a bytestring')
>>> buf.seek(0)
0
>>> # use case starts here
>>> buf.name = 'file.gz' # add a .name attribute so smart_open knows what compressor to use
>>> import smart_open
>>> smart_open.open(buf, 'rb').read() # will gzip-decompress transparently
b'this is a bytestring'

If your file object already has an appropriate .name, then you don't have to add one.