Skip to content

dogpile.cache 1.1.0 fails to serialize on pypy3.6 7.3.2 #195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RazerM opened this issue Nov 20, 2020 · 11 comments
Closed

dogpile.cache 1.1.0 fails to serialize on pypy3.6 7.3.2 #195

RazerM opened this issue Nov 20, 2020 · 11 comments
Labels
bug Something isn't working

Comments

@RazerM
Copy link

RazerM commented Nov 20, 2020

Full reproducer using docker:

$ docker run --rm -it pypy:3-7 bash
root@302cdcffd48a:/# pip install dogpile.cache
Collecting dogpile.cache
  Downloading dogpile.cache-1.1.0.tar.gz (365 kB)
     |████████████████████████████████| 365 kB 1.7 MB/s
Collecting decorator>=4.0.0
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting stevedore>=3.0.0
  Downloading stevedore-3.2.2-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 1.9 MB/s
Collecting pbr!=2.1.0,>=2.0.0
  Downloading pbr-5.5.1-py2.py3-none-any.whl (106 kB)
     |████████████████████████████████| 106 kB 21.7 MB/s
Collecting importlib-metadata>=1.7.0; python_version < "3.8"
  Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
Collecting zipp>=0.5
  Downloading zipp-3.4.0-py3-none-any.whl (5.2 kB)
Building wheels for collected packages: dogpile.cache
  Building wheel for dogpile.cache (setup.py) ... done
  Created wheel for dogpile.cache: filename=dogpile.cache-1.1.0-py3-none-any.whl size=49120 sha256=81530950d73e4632593c6f1fcfe313dbdc9574fce880f4023f1c50b9478e7719
  Stored in directory: /root/.cache/pip/wheels/d6/8a/7e/51ac2437518e978166dfc2c643ea90f94721b3b77146c84803
Successfully built dogpile.cache
Installing collected packages: decorator, pbr, zipp, importlib-metadata, stevedore, dogpile.cache
Successfully installed decorator-4.4.2 dogpile.cache-1.1.0 importlib-metadata-2.0.0 pbr-5.5.1 stevedore-3.2.2 zipp-3.4.0

root@302cdcffd48a:/# pypy
Python 3.6.9 (d38cd66c14b8, Sep 23 2020, 08:01:17)
[PyPy 7.3.2 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>>> from dogpile.cache.region import make_region
>>>> region = make_region().configure('dogpile.cache.dbm', arguments={'filename': 'cache.test'})
>>>> region.set('a', 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/pypy/site-packages/dogpile/cache/region.py", line 1305, in set
    self.backend.set_serialized(key, self._serialized_payload(value))
  File "/opt/pypy/site-packages/dogpile/cache/region.py", line 1233, in _serialized_payload
    return self._serialize_cached_value_elements(payload, metadata)
  File "/opt/pypy/site-packages/dogpile/cache/region.py", line 1218, in _serialize_cached_value_elements
    serializer(payload),
  File "/opt/pypy/lib-python/3/pickle.py", line 1564, in _dumps
    _Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
  File "/opt/pypy/lib-python/3/pickle.py", line 372, in __init__
    if protocol < 0:
TypeError: '<' not supported between instances of 'dict' and 'int'

I couldn't find where serializer gets assigned to find out what is happening

@zzzeek zzzeek added the bug Something isn't working label Nov 20, 2020
@zzzeek
Copy link
Member

zzzeek commented Nov 20, 2020

that's weird it's like the pickle.dumps() function is accepting two arguments ? not sure how all our tests pass if it's got a problem like that.

@zzzeek
Copy link
Member

zzzeek commented Nov 20, 2020

i dont see in your above test where "region" is coming from , can you show that please? backend etc.

@RazerM
Copy link
Author

RazerM commented Nov 20, 2020

Oops I missed those lines when copying, I updated the example

@zzzeek
Copy link
Member

zzzeek commented Nov 20, 2020

does not fail w/ cpython 3.8!! weeeeeird

@zzzeek
Copy link
Member

zzzeek commented Nov 20, 2020

i bet this has to do w/ cpickle vs pickle

@zzzeek
Copy link
Member

zzzeek commented Nov 20, 2020

in pypy 3.6.9 I get this:

TypeError: can't pickle _local objects

still looks like non-cpickle stuff

@zzzeek
Copy link
Member

zzzeek commented Nov 20, 2020

alrighty, here it is:

diff --git a/dogpile/cache/api.py b/dogpile/cache/api.py
index d89ac56..54d7c9a 100644
--- a/dogpile/cache/api.py
+++ b/dogpile/cache/api.py
@@ -435,8 +435,8 @@ class CacheBackend:
 
 
 class DefaultSerialization:
-    serializer: Optional[Serializer] = pickle.dumps
-    deserializer: Optional[Deserializer] = pickle.loads
+    serializer: Optional[Serializer] = staticmethod(pickle.dumps)
+    deserializer: Optional[Deserializer] = staticmethod(pickle.loads)
 
 
 class BytesBackend(DefaultSerialization, CacheBackend):

pypy turning it into a method, likely because it's pure python pickle.

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the master branch:

apply staticmethod() to pickle.dumps/loads as class variables https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/2378

@zzzeek
Copy link
Member

zzzeek commented Nov 20, 2020

let me know if you can run dogpile.cache master through your use case testing or not, in case there are more regressions we can find. otherwise I can just release with what we have.

@RazerM
Copy link
Author

RazerM commented Nov 23, 2020

master passes the PyPy CI testing I have, thanks

@zzzeek
Copy link
Member

zzzeek commented Nov 23, 2020

ok let's release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants