hashlib object cannot be pickled

[vstinner]

BPO	11771
Nosy	@rhettinger, @gpshead, @pitrou, @vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-04-07.08:17:54.635>
created_at = <Date 2011-04-05.11:49:41.254>
labels = ['type-feature', 'library']
title = 'hashlib object cannot be pickled'
updated_at = <Date 2017-07-14.12:14:45.388>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-07-14.12:14:45.388>
actor = 'Andrey.Kislyuk'
assignee = 'none'
closed = True
closed_date = <Date 2011-04-07.08:17:54.635>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2011-04-05.11:49:41.254>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 11771
keywords = []
message_count = 10.0
messages = ['133021', '133022', '133030', '133192', '133193', '222036', '222042', '222044', '222050', '298350']
nosy_count = 6.0
nosy_names = ['rhettinger', 'gregory.p.smith', 'pitrou', 'vstinner', 'Andrey.Kislyuk', 'approximately']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11771'
versions = ['Python 3.3']

[vstinner]

$ ./python 
Python 3.3a0 (default:76ed6a061ebe, Apr  5 2011, 12:25:00) 
>>> import hashlib, pickle
>>> hash=hashlib.new('md5')
>>> pickle.dumps(hash)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class '_hashlib.HASH'>: attribute lookup _hashlib.HASH failed

The problem is that _hashlib.HASH is not accessible at Python level. There is a C define to make it accessible, but it is disabled by default: "#if HASH_OBJ_CONSTRUCTOR". This test is as old as the _hashlib module (bpo-1121611, 624918e1c1b2).

[vstinner]

Oh, I don't know if it is possible to serialize a OpenSSL hash object (EVP_MD_CTX)...

[pitrou]

Why on Earth would you want to serialize a hashlib object?
It makes as much sense as serializing, say, a JSONEncoder.

[gpshead]

heh yeah. while all hash functions do have internal state and someone
could conceivably want to store such a state (it basically amounts to
queued up partial block of input data if any and the current starting
IV) there are not consistent APIs to expose that and I really don't
see why it'd be worth trying to find them.

remember, hashlib doesn't have to be openssl. there are non openssl
libtomcrypt based versions and someone nice should write a libnss
based version someday.

i'd mark this "won't fix." :)

-Greg

On Tue, Apr 5, 2011 at 7:02 AM, Antoine Pitrou <report@bugs.python.org> wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

Why on Earth would you want to serialize a hashlib object?
It makes as much sense as serializing, say, a JSONEncoder.

----------
nosy: +gregory.p.smith, pitrou

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11771\>

---

[rhettinger]

I also recommend closing this one.

[approximately]

Please reopen this bug. To answer the question: "Why on Earth would you want to serialize a hashlib object?" : multiprocessing.connection.ForkingPickler wants. I.e. if you want to parallelize your hash calculations, this will obstruct your efforts.

[gpshead]

Do you honestly have a situation where you need to share a computationally
significant amount of hashing state only to want to finish the computation
N different times with alternate computationally significant ending data
that multiprocessing would actually help with where you cannot use
threads? Hashlib releases the GIL during nontrivial hash computations.

[approximately]

You want to say: It doesn't work, but it is somehow intentional because you never used id, correct?

[gpshead]

Please be constructive.

There is no way to implement generic pickling for hash objects that would work across all implementations.

The underlying code implementing each function is free to store its internal state however it wants and does not provide an API to get at it or any standard representation of it.

Sure, you could hack things up and allow a specific version and build of openssl's EVP hashes to dump their state and restore it for use in another process running that same specific version and build of openssl (as would likely be the case for multiprocessing use) just as you could for any other implementation of a hash function such as the builtin libtomcrypt versions. But this is not portable between compilations using different implementations of the hash algorithm. That is not what someone using pickle would ever expect.

Public APIs to access the internal state of hash functions do not exist because it is not a common thing for people to do.

hashlib isn't going to support this unless someone contributes a very solid patch with tests that handles all of the compatibility issues in a friendly maintainable manner.

[AndreyKislyuk]

For anyone else looking for a solution to this, I wrote a library: https://github.com/kislyuk/rehash