Skip to content
This repository has been archived by the owner on Mar 24, 2021. It is now read-only.

pypy snappy compression seg fault on larger messages #508

Closed
johnistan opened this issue Mar 21, 2016 · 3 comments
Closed

pypy snappy compression seg fault on larger messages #508

johnistan opened this issue Mar 21, 2016 · 3 comments
Assignees
Labels

Comments

@johnistan
Copy link
Contributor

This is a carry over from #480. It is also an upstream python snappy issue but we should track it some how.

So any byte array above a certain size causes a seg fault.

(pypy) ➜ cat s.py
from uuid import uuid4
import snappy

payload = b''.join([uuid4().bytes for i in range(10)])

c = snappy.compress(payload)
assert snappy.decompress(c) == payload
(pypy) ➜  python s.py
[1]    4587 segmentation fault (core dumped)  python s.py
(pypy) ➜  python
Python 2.7.10 (bbd45126bc69, Mar 18 2016, 21:35:08)
[PyPy 5.0.1 with GCC 4.8.4] on linux2

while

from uuid import uuid4
import snappy

payload = b''.join([uuid4().bytes for i in range(5)])

c = snappy.compress(payload)
assert snappy.decompress(c) == payload

passes

Perhaps a warning doc for pypy users.

@emmettbutler
Copy link
Contributor

We could also use some tests for this case in test_compression.py.

@johnistan
Copy link
Contributor Author

woh python-snappy could use some love. last code commit was three years ago and there is no CI hooked up.

@johnistan
Copy link
Contributor Author

from googling.

https://github.com/rjpower/ctypes-snappy

@emmettbutler emmettbutler self-assigned this Mar 24, 2016
yungchin added a commit that referenced this issue Apr 13, 2016
* parsely/master: (130 commits)
  Import sys as _sys
  Handle intepreter shutdown in RequestHandler
  fix compare link, increment version
  changelog, increment version
  use a None check to enable tests that do not use a Producer
  enqueue popped message
  fix NoneType error in producer for large messages
  Fix then vs than typo
  handle IllegalGeneration in SimpleConsumer. fixes #515
  ensure consumer exists before producing
  remove self.consumer reference from tearDownClass
  fix protocol tests for python 3
  stop producer tests from sharing a consumer instance. fixes #484
  clearer message
  test and warn about pypy snappy segfaulting on some messages. fixes #508
  remove unused import
  clarify documentation around latest available offsets. fixes #494
  reformat protocol tests for readability
  clean up readme
  readme note about managed consumer
  ...

Conflicts:
	pykafka/__init__.py
	pykafka/broker.py
	pykafka/connection.py
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants