dogpile.cache 1.1.0 fails to serialize on pypy3.6 7.3.2

[RazerM]

Full reproducer using docker:

$ docker run --rm -it pypy:3-7 bash
root@302cdcffd48a:/# pip install dogpile.cache
Collecting dogpile.cache
  Downloading dogpile.cache-1.1.0.tar.gz (365 kB)
     |████████████████████████████████| 365 kB 1.7 MB/s
Collecting decorator>=4.0.0
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting stevedore>=3.0.0
  Downloading stevedore-3.2.2-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 1.9 MB/s
Collecting pbr!=2.1.0,>=2.0.0
  Downloading pbr-5.5.1-py2.py3-none-any.whl (106 kB)
     |████████████████████████████████| 106 kB 21.7 MB/s
Collecting importlib-metadata>=1.7.0; python_version < "3.8"
  Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
Collecting zipp>=0.5
  Downloading zipp-3.4.0-py3-none-any.whl (5.2 kB)
Building wheels for collected packages: dogpile.cache
  Building wheel for dogpile.cache (setup.py) ... done
  Created wheel for dogpile.cache: filename=dogpile.cache-1.1.0-py3-none-any.whl size=49120 sha256=81530950d73e4632593c6f1fcfe313dbdc9574fce880f4023f1c50b9478e7719
  Stored in directory: /root/.cache/pip/wheels/d6/8a/7e/51ac2437518e978166dfc2c643ea90f94721b3b77146c84803
Successfully built dogpile.cache
Installing collected packages: decorator, pbr, zipp, importlib-metadata, stevedore, dogpile.cache
Successfully installed decorator-4.4.2 dogpile.cache-1.1.0 importlib-metadata-2.0.0 pbr-5.5.1 stevedore-3.2.2 zipp-3.4.0

root@302cdcffd48a:/# pypy
Python 3.6.9 (d38cd66c14b8, Sep 23 2020, 08:01:17)
[PyPy 7.3.2 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>>> from dogpile.cache.region import make_region
>>>> region = make_region().configure('dogpile.cache.dbm', arguments={'filename': 'cache.test'})
>>>> region.set('a', 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/pypy/site-packages/dogpile/cache/region.py", line 1305, in set
    self.backend.set_serialized(key, self._serialized_payload(value))
  File "/opt/pypy/site-packages/dogpile/cache/region.py", line 1233, in _serialized_payload
    return self._serialize_cached_value_elements(payload, metadata)
  File "/opt/pypy/site-packages/dogpile/cache/region.py", line 1218, in _serialize_cached_value_elements
    serializer(payload),
  File "/opt/pypy/lib-python/3/pickle.py", line 1564, in _dumps
    _Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
  File "/opt/pypy/lib-python/3/pickle.py", line 372, in __init__
    if protocol < 0:
TypeError: '<' not supported between instances of 'dict' and 'int'

I couldn't find where serializer gets assigned to find out what is happening

[zzzeek]

that's weird it's like the pickle.dumps() function is accepting two arguments ? not sure how all our tests pass if it's got a problem like that.

[zzzeek]

i dont see in your above test where "region" is coming from , can you show that please? backend etc.

[RazerM]

Oops I missed those lines when copying, I updated the example

[zzzeek]

does not fail w/ cpython 3.8!! weeeeeird

[zzzeek]

i bet this has to do w/ cpickle vs pickle

[zzzeek]

in pypy 3.6.9 I get this:

TypeError: can't pickle _local objects

still looks like non-cpickle stuff

[zzzeek]

alrighty, here it is:

diff --git a/dogpile/cache/api.py b/dogpile/cache/api.py
index d89ac56..54d7c9a 100644
--- a/dogpile/cache/api.py
+++ b/dogpile/cache/api.py
@@ -435,8 +435,8 @@ class CacheBackend:
 
 
 class DefaultSerialization:
-    serializer: Optional[Serializer] = pickle.dumps
-    deserializer: Optional[Deserializer] = pickle.loads
+    serializer: Optional[Serializer] = staticmethod(pickle.dumps)
+    deserializer: Optional[Deserializer] = staticmethod(pickle.loads)
 
 
 class BytesBackend(DefaultSerialization, CacheBackend):

pypy turning it into a method, likely because it's pure python pickle.

[sqla-tester]

Mike Bayer has proposed a fix for this issue in the master branch:

apply staticmethod() to pickle.dumps/loads as class variables https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/2378

[zzzeek]

let me know if you can run dogpile.cache master through your use case testing or not, in case there are more regressions we can find. otherwise I can just release with what we have.

[RazerM]

master passes the PyPy CI testing I have, thanks

[zzzeek]

ok let's release