WIP: Refactor out deserialize_bytelist#4077
Conversation
As it can be quite useful to have a way of directly working with a `bytes`-like `list` as is provided by `serialize_bytelist`, particularly when thinking about in-memory and/or performance-focused operations (where additional copies can meaningfully impact both), refactor out a `deserialize_bytelist` from `deserialize_bytes`.
Ensure `deserialize_bytelist` is available from `distributed.protocol`.
Make sure we are also able to handle collections of `bytes`-like objects in `deserialize_bytes`. This is useful if we may be deserializing collections of objects in-memory in some cases and deserializing data from disk in other cases.
| def deserialize_bytes(b): | ||
| try: | ||
| b = memoryview(b) | ||
| except TypeError: |
There was a problem hiding this comment.
I'm sorry, I think I've asked this before, can you remind me when you think we would hit a TypeError?
There was a problem hiding this comment.
If the object is not bytes-like, this will raise. We end up performing this coercion in unpack_frames later. So this merely performs this coercion a bit sooner.
The non-exception case is for things like spilling to disk (analogous to deserialize_bytes(serialize_bytes(obj))) and the exception case is for things like compressed in-memory spilling (analogous to deserialize_bytes(serialize_bytelist(obj))). IOW both cases can rely on using deserialize_bytes.
jakirkham
changed the title
deserialize_bytelistdeserialize_bytelist
|
There are errors in test_spill_to_disk which seem related
|
|
Yeah getting this right is a bit fiddly based on how the code works. I'll give it another look. Should add this is why I marked this as WIP for the moment. |
2 participants
Extracts the
deserialize_bytelistfunction fromdeserialize_bytes. Under-the-hooddeserialize_bytesdoes a minimal amount of prep work before calling intodeserialize_bytelist. Thus maintaining the same code path.This can be a useful place to hook into this part of serialization pipeline for things like in-memory compressed spilling in combination with
serialize_bytelist. Also this avoids unnecessary copying in this case (unlikedeserialize_bytes).