snappy.UncompressError: stream missing snappy identifier

[jkryanchou]

When I used snappy module to decompress a file it occurs the issue

snappy.UncompressError: stream missing snappy identifier

[andrix]

The implementation for file compression/decompression using streaming was done by @jtolds. He may have a better idea of what's going on.

[jtolio]

There's two forms of Snappy compression, the basic form and the streaming form. The basic form has the limitation that it all must fit in memory, so the streaming form exists to be able to compress larger amounts of data. The streaming format has a header and then subranges that are compressed. If the header is missing, it sounds like maybe you compressed using the basic form and are trying to uncompress with the streaming form.

[jkryanchou]

@jtolds so how to uncompressed with streaming form? any tutorial to help me figure it out?

[jtolio]

What did you do to uncompress the data? Can you paste the code you used? That way I can help with what needs to change.

[jkryanchou]

@jtolds The following were the snappy cli I used.

python -m snappy -d ./02_1473019352277439472_14838198.snappy
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/lib/python2.7/site-packages/snappy.py", line 305, in <module>
    cmdline_main()
  File "/usr/local/lib/python2.7/site-packages/snappy.py", line 301, in cmdline_main
    method(src, dst)
  File "/usr/local/lib/python2.7/site-packages/snappy.py", line 263, in stream_decompress
    buf = decompressor.decompress(buf)
  File "/usr/local/lib/python2.7/site-packages/snappy.py", line 192, in decompress
    raise UncompressError("stream missing snappy identifier")
snappy.UncompressError: stream missing snappy identifier

I dont know what's wrong with it. Please help me out with this issue. Thanks

[jtolio]

okay, the commandline tool only supports the streaming format. You will need to write a program that imports the snappy package and uses the plain uncompress method with the data from that file.

import snappy, sys; sys.stdout.write(snappy.uncompress(file("./02_1473019352277439472_14838198.snappy").read()))

[jkryanchou]

Oh. I got it. It works like a charm. Thanks a lot. It confused me so long, And why not provide an option to process the file?

[jtolio]

Because unframed snappy compression has a maximum size limit, so if you're dealing with files you almost certainly want the streaming format

[jkryanchou]

OK. I got it.

[rajeevanair]

Hi,
I installed snappy package, but the module uncompress is not present. Kindly, help me with this.

Thanks,
Rajeev A. Nair

[martindurant]

@rajeevanair , can you please be more specific about your situation, how you installed, what versions you got and what you did to see the error.

[rajeevanair]

Hi Martin,
I installed python-snappy-0.5.1, using python setup.py install command.
Then I imported snappy using :
import snappy

However snappy library does not have an uncompress function.

[martindurant]

$ conda create -n test python cffi
$ source activate test
$ pip install git+https://github.com/andrix/python-snappy  # this is version 0.5.2
$ python
>>> import snappy
>>> snappy.uncompress
<function uncompress at 0x108df7510>

[rajeevanair]

A thousand thanks, Martin! That worked.