Use highest available pickle protocol when serializing (#737)

Use pickle 5 in IPC
uber · Mar 24, 2022 · 26e03c7 · 26e03c7
1 parent 76a36fe
commit 26e03c7
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 1 deletion.
diff --git a/docs/release-notes.rst b/docs/release-notes.rst
@@ -14,6 +14,7 @@ Release 0.11.4
 - `PR 731 <https://github.com/uber/petastorm/pull/731>`_ (resolves `PR 728 <https://github.com/uber/petastorm/issues/728>`_): Support passing multiple parquet dataset urls to make_reader.
 - `PR 732 <https://github.com/uber/petastorm/pull/732>`_ (resolves `PR 585 <https://github.com/uber/petastorm/issues/585>`_): Restructure process_pool implementation code in a way that resolves ``RuntimeWarning: 'petastorm.workers_pool.exec_in_new_process' found in sys.modules after
   import of package 'petastorm.workers_pool', but prior to execution of 'petastorm.workers_pool.exec_in_new_process'; this may result in unpredictable behaviou when using process pool`` warning.
+- `PR 737 <https://github.com/uber/petastorm/pull/737>`_: Use highest available pickle protocol for internal serialization.
 
 
 Release 0.11.3

diff --git a/petastorm/reader_impl/pickle_serializer.py b/petastorm/reader_impl/pickle_serializer.py
@@ -17,7 +17,7 @@
 class PickleSerializer(object):
 
     def serialize(self, rows):
-        return pickle.dumps(rows)
+        return pickle.dumps(rows, protocol=pickle.HIGHEST_PROTOCOL)
 
     def deserialize(self, serialized_rows):
         return pickle.loads(serialized_rows)